k8s Load Reconciliation

Every k8s CronJob run that uploads data to Spaces now writes a _run.json
manifest alongside the data objects. The VM-side reconciliation script reads
those manifests, compares them against the DuckDB stage status, and reports
what's pending, staged, failed, or empty.

Object layout

raw/nrl/dataset=odds_ticks/
  collector=k8s/
    season=2026/
      round=20/
        snapshot_label=refresh/
          run_id=20260714_060000/
            _run.json          ← manifest written by the CronJob
            ticks.ndjson.gz    ← raw tick data

snapshot_label is part of the path (open / refresh / close).

Manifest fields (`_run.json`)

Field	Description
`run_id`	`YYYYMMDD_HHMMSS` timestamp of the pod run
`task`	`odds_snapshot`
`collector`	`k8s` / `vm` / `manual`
`dataset`	`odds_ticks`
`season` / `round` / `snapshot_label`	routing keys
`started_at_utc` / `finished_at_utc`	ISO-8601 wall-clock times
`status`	`running` → `success` / `failure` / `zero_rows`
`record_count`	Number of tick rows in the NDJSON
`object_keys`	List of S3 keys written (ticks.ndjson.gz)
`error_message`	Set on `failure`
`pod_name` / `job_name` / `image_tag`	k8s metadata (from env vars)
`duckdb_staged_at_utc` / `duckdb_stage_status`	Set after VM staging

Reconciliation workflow

1. List Spaces manifests and report

python scripts/reconcile_k8s_raw_runs.py --season 2026 --round 20

Output:

Listing _run.json manifests under raw/nrl/dataset=odds_ticks/ …
Found 3 manifest(s)

Reconciliation summary (3 manifests found in Spaces)
  staged:    2
  pending:   1
  failed:    0
  zero_rows: 0

PENDING (not yet staged):
  run_id=20260714_120000 round=20 label=refresh records=84

Exit code: 0 if all staged, 1 if any pending.

2. Stage pending runs

python scripts/reconcile_k8s_raw_runs.py --season 2026 --round 20 --stage-pending

This calls stage_odds_ticks.py --ticks-key <key> for each pending run and
marks it as staged in ops_nrl_raw_run_reconciliation.

3. Stage a single known run

python scripts/stage_odds_ticks.py \
  --ticks-key "raw/nrl/dataset=odds_ticks/collector=k8s/season=2026/round=20/snapshot_label=refresh/run_id=20260714_060000/ticks.ndjson.gz"

DuckDB table

ops_nrl_raw_run_reconciliation tracks all known runs:

SELECT * FROM ops_nrl_raw_run_reconciliation
WHERE season=2026 AND round=20
ORDER BY reconciled_at_utc DESC;

Columns: run_id, collector, task, dataset, season, round,
snapshot_label, manifest_key, record_count, raw_status,
stage_status, staged_at_utc, staged_row_count, error_message,
reconciled_at_utc.

Gate integration

check_nrl_recommendation_gate.py includes a k8s_raw_data_staged check
(WARNING, non-blocking) that fires when pending runs exist:

python scripts/check_nrl_recommendation_gate.py --season 2026 --round 20 --market all

The check is non-blocking — it won't stop recommendations, but it will show
up as a warning on /ops.

/ops visibility

The /ops page shows a "Raw Run Reconciliation" section. It's populated from
ops_nrl_raw_run_reconciliation and updates whenever the reconciliation
script runs. A fresh cluster with no reconciliation runs yet shows an empty
section with a prompt to run the script.

Collect with manifest (--upload-to-spaces)

The collect_nrl_odds_snapshot.py script writes a manifest when
--upload-to-spaces is passed:

python scripts/collect_nrl_odds_snapshot.py \
  --season 2026 --round 20 --snapshot-label refresh \
  --upload-to-spaces --collector k8s

The k8s CronJob's command does not pass --upload-to-spaces by default —
add it when the cluster is the primary odds collector and reconciliation is
needed. The VM can also use this flag for its own runs.

Deploy test — verifying a tlwindow deploy (two-phase)

After redeploying the scrapers image (e.g. after a registry revert or a new
image tag), run the two-phase deploy test to confirm the tlwindow CronJobs
are healthy end-to-end.

Phase 1 — kubectl (from a machine with cluster access)

# Confirm 4 CronJobs registered with correct image:
kubectl -n doks-ci-cd get cronjobs \
  -l 'app.kubernetes.io/component=teamlist-window' -o wide

# Trigger a smoke-test job manually (pre-Tuesday check):
kubectl -n doks-ci-cd create job \
  --from=cronjob/scrapers-tlwindow-t0 tlwindow-smoketest
kubectl -n doks-ci-cd logs -l job-name=tlwindow-smoketest -f

# Check for ImagePullBackOff:
kubectl -n doks-ci-cd describe pod -l job-name=tlwindow-smoketest \
  | grep -A5 Events:

# After Tuesday 18:00 UTC — confirm all 4 fired:
kubectl -n doks-ci-cd get jobs \
  -l 'app.kubernetes.io/component=teamlist-window'

Phase 2 — VM (Spaces → DuckDB audit)

Run after the Tuesday jobs have had time to fire and upload to Spaces:

source nrl-bet-advisor/.venv/bin/activate
bash scripts/verify_k8s_tlwindow.sh --season 2026 --round <round>
# Expects: "✓ Phase 2 PASS" and tlwindow_clusters=4/4

The script reconciles Spaces _run.json manifests, stages any pending rows
into DuckDB, and runs audit_snapshot_coverage.py — exits 0 on pass, 1 on
fail. Phase 1 (kubectl) commands are also printed so you can copy-paste them.

One-writer rule

Staging (writing to movement_odds_snapshots or DuckDB) always happens on the
VM. The k8s CronJob only uploads raw artifacts to Spaces. Never write DuckDB
from a k8s pod.