Pipeline operations
Ops
cluster
· build
5fbefe95e6f1e92c21bbb3da97477b5ed59af873
2026-09-07 12:27 UTC
5 item(s) need attention
Recommendation readiness · 2026 R27
BLOCKED — paper recommendations are not allowed
- open odds missing
- team list missing
- close odds missing
Warnings: odds refresh not recorded; DQ gate has not been confirmed; fan picks not loaded for round
Gate command: bash scripts/run_cluster_task.sh dq-gate --season 2026 --round <round> --market all · live evidence report: /app/reports/live_evidence.md (present)
| Feed | Last delivered | Expected | Detail | |
|---|---|---|---|---|
| Monday open odds | — | Monday before round | no opening snapshot for selected round | |
| Six-hour odds refresh | — | ~6 hours through round week | no refresh recorded yet | |
| Tuesday team lists | — | within 30 minutes of Tuesday drop | required lineup signal has not arrived | |
| Per-kickoff close odds | — | before each kickoff | no close snapshot for selected round | |
| Draw | present | round refresh | table available | |
| Player stats | present | round refresh | table available | |
| Weather | present | round refresh | table available | |
| DQ gate | — | pass before seeding | run check_nrl_recommendation_gate.py | |
| Settlement / CLV | — | settle and populate CLV within 24h | 0 settled trade(s) missing CLV | |
| Fan picks (ESPN) | — | weekly before round | not yet loaded — run fan-pick-loader skill | |
| Player roster | present | monthly players spider | 779 canonical player entries |
Player markets: Player markets: not active, raw collection planned · recommendation impact: none.
Instance diagnostics
What THIS instance can see. On the cluster, a red row here is the answer to
"why is the data not coming through" — fix the row's hint before debugging
anything else. DB file: /tmp/nrl_bucket.duckdb
· last synced 2026-09-07 12:24:45 UTC (0.0h ago)
| Check | Detail | Fix hint | |
|---|---|---|---|
| DuckDB | /tmp/nrl_bucket.duckdb (via NRL_DUCKDB_PATH) — 185 MB, 72 tables | ||
| reports/ | 50 item(s) | ||
| docs/ | 34 item(s) | ||
| dbt schemas | 4 item(s) | ||
| model metrics | 1 item(s) | ||
| Basic auth | disabled (public read-only dashboard policy) | ||
| Build provenance | commit=5fbefe95e6f1e92c21bbb3da97477b5ed59af873, built_at=unknown | Build with --build-arg GIT_COMMIT=$(git rev-parse --short HEAD) and IMAGE_BUILT_AT=$(date -u +%Y-%m-%dT%H:%MZ). |
Data contracts — expected vs delivered
Every feed's contract: cadence, who delivers it (origin), when it last arrived, and when the next delivery is due. Read-only — no rerun triggers from this page by design; commands are listed for the operator.
| Feed | Origin | Cadence | Last delivered | Next expected | Detail | Command | |
|---|---|---|---|---|---|---|---|
| Odds collect — open | vm | Monday each round | 2026-07-13 01:57:45.903261+00:00 | 2026-07-20 01:57 UTC (overdue) | age 1354.5h | collect_nrl_odds_snapshot.py --snapshot-label open |
|
| Odds collect — refresh | vm | ~6h through round week | 2026-08-13 20:00:16.755854+00:00 | 2026-08-14 08:00 UTC (overdue) | age 592.4h | collect_nrl_odds_snapshot.py --snapshot-label refresh |
|
| Odds collect — close | — | pre-kickoff, per game | — | — | no snapshots recorded | collect_nrl_odds_snapshot.py --snapshot-label close |
|
| Odds collect — teamlist_window | vm | Tue T+5m/T+35m/T+2h/T+12h UTC | 2026-07-20 10:51:49.677495+00:00 | 2026-07-27 10:51 UTC (overdue) | 0/4 Tuesday captures last 7d | k8s CronJobs scrapers-tlwindow-* (go-cicd-ops) |
|
| TheOddsAPI — historical backfill | cluster/vm | one-time 2026 grab | 2026-07-21 03:50:42.234314+00:00 | 2026-07-28 03:50 UTC (overdue) | 5578 rows | scripts/backfill_odds_api_history.py |
|
| TheOddsAPI — match archive ticks | cluster/vm | open / Tue window / 5-10min final hour | 2026-07-24 08:00:18.829373+00:00 | 2026-07-31 08:00 UTC (overdue) | 9901 rows | match-observation archive |
|
| TheOddsAPI — intelligence/edges | vm | per decision snapshot | — | — | stg_odds_intelligence not created yet — new-vendor feed pending | /intelligence/ + /edges/ (collect+validate) |
|
| Draw & results | vm | weekly (post-round) | 2026 R23 | post-round (Sun/Mon) | results coverage | weekly_2026_round.sh |
|
| Team lists | vm | weekly (Tue ~4pm) | 2026 R24 | within 1 round of the draw | through 2026 R24 | run_spiders.py --spider team_lists |
|
| Player match stats | vm | weekly (post-round) | 2026 R23 | within 1 round of the draw | through 2026 R23 | run_spiders.py --spider match_stats + dbt gold_nrl_player_round_stats |
|
| Lineup features (dbt) | vm | weekly | 2026 R27 | within 1 round of the draw | through 2026 R27 | dbt build --select ml_nrl_match_lineup_features |
|
| Fan picks | vm | weekly (pre-round) | 2026 R19 | within 1 round of the draw | through 2026 R19 — 4 round(s) behind the draw | python scripts/load_fan_picks.py <csv> |
|
| Player roster refresh | vm | monthly | 2026 | — | latest season 2026 | monthly_2026_players.sh |
|
| CLV update | vm | post-round | 2026-07-05 23:43:02.196951+00:00 | after each round settles | 0 settled trade(s) missing CLV | update_nrl_movement_clv.py --settle |
|
| Paper trade seeding | vm | weekly (pre-round) | 2026 R20 | before each round's first kickoff | 229 trades in latest season | setup_paper_trading.py |
|
| open_fav_v1 evidence | vm | every round | 2026 R20 | before each round's first kickoff; CLV within 24h | 2 trade(s); 0 settled trade(s) missing CLV | setup_paper_trading.py (verify registered strategy inputs) |
|
| rapm_timing_v1 evidence | vm | every round | — | before each round's first kickoff; CLV within 24h | no registered strategy trades — quota not armed | setup_paper_trading.py (verify registered strategy inputs) |
|
| line_move_v1 evidence | vm | every round | — | before each round's first kickoff; CLV within 24h | no registered strategy trades — quota not armed | manage_line_move_paper.py |
|
| LightGBM retrain | vm | ≤ 45 days | — | — | lgbm_v1.pkl missing | cd nrl && python3 src/train.py |
ok · due · stale · unavailable. Origin vm = working machine cron, cluster = k8s CronJob, manual = human-triggered, repo = committed artifact.
Coverage audit — 2026 R27 FAIL
Per-game contract check: open captured first, a close inside [−120m, kickoff],
no capture gap >8h, h2h+line+total present at close. Same checks as
scripts/audit_snapshot_coverage.py.
| Match | Kickoff (UTC) | Rows | Notes | |
|---|---|---|---|---|
| Dragons v Eels | 2026-09-06 04:00 | 0 | no open snapshot; no close capture in [-120m, kickoff]; fewer than 2 pre-kickoff captures; h2h missing at close | |
| Bulldogs v Broncos | 2026-09-03 09:50 | 0 | no open snapshot; no close capture in [-120m, kickoff]; fewer than 2 pre-kickoff captures; h2h missing at close | |
| Titans v Dolphins | 2026-09-04 08:00 | 0 | no open snapshot; no close capture in [-120m, kickoff]; fewer than 2 pre-kickoff captures; h2h missing at close | |
| Cowboys v Raiders | 2026-09-05 07:30 | 0 | no open snapshot; no close capture in [-120m, kickoff]; fewer than 2 pre-kickoff captures; h2h missing at close | |
| Rabbitohs v Roosters | 2026-09-04 10:00 | 0 | no open snapshot; no close capture in [-120m, kickoff]; fewer than 2 pre-kickoff captures; h2h missing at close | |
| Warriors v Sea Eagles | 2026-09-05 05:00 | 0 | no open snapshot; no close capture in [-120m, kickoff]; fewer than 2 pre-kickoff captures; h2h missing at close | |
| Panthers v Wests Tigers | 2026-09-06 06:05 | 0 | no open snapshot; no close capture in [-120m, kickoff]; fewer than 2 pre-kickoff captures; h2h missing at close | |
| Sharks v Storm | 2026-09-05 09:35 | 0 | no open snapshot; no close capture in [-120m, kickoff]; fewer than 2 pre-kickoff captures; h2h missing at close |
Team-list window: 0/4 Tuesday capture clusters (FAIL) · collector mix:
Odds monitor
Movement Trade Candidates
No pending trade candidates for this round.
Snapshot Coverage
| Round | Label | Books | Matches | Markets | Latest snapshot |
|---|---|---|---|---|---|
| 19 | refresh | 3 | 15 | 3 | 2026-07-09T08:00 |
Raw run reconciliation — k8s CronJob history
Each k8s CronJob run writes a _run.json manifest to Spaces.
This table reflects the last reconciliation pass
(python scripts/reconcile_k8s_raw_runs.py).
Run with --stage-pending to push pending runs into DuckDB.
staged: 3
pending: 0
failed: 9
zero_rows: 3
| Run ID | Collector | Round | Label | Records | Raw | Stage | Staged at | Error | |
|---|---|---|---|---|---|---|---|---|---|
20260811_122341 |
vm | R24 | refresh | 192 | success | staged | 2026-08-11T12:24:07.453020+00:00 | ||
20260810_200006 |
vm | R24 | refresh | 176 | success | staged | — | ||
20260810_020019 |
vm | R24 | refresh | 176 | success | staged | — | ||
20260721_133930 |
vm | R21 | refresh | 192 | success | failed | — | Traceback (most recent call last): File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 318, in <module> sys.exit(main()) ^^^^^^ File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 258, in main con = duckdb.connect(str(db_path)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _duckdb.IOException: IO Error: Could not set lock on file "/home/nitro/nitro-repos/scrapers/duckdb/nrl_bucket.duckdb": Conflicting lock is held in /usr/bin/python3.12 (PID 257784). See also https://duckdb.org/docs/stable/connect/concurrency | |
20260721_020007 |
vm | R21 | refresh | 112 | success | failed | — | Traceback (most recent call last): File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 318, in <module> sys.exit(main()) ^^^^^^ File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 258, in main con = duckdb.connect(str(db_path)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _duckdb.IOException: IO Error: Could not set lock on file "/home/nitro/nitro-repos/scrapers/duckdb/nrl_bucket.duckdb": Conflicting lock is held in /usr/bin/python3.12 (PID 257784). See also https://duckdb.org/docs/stable/connect/concurrency | |
20260720_080014 |
vm | R21 | refresh | 112 | success | failed | — | Traceback (most recent call last): File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 318, in <module> sys.exit(main()) ^^^^^^ File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 258, in main con = duckdb.connect(str(db_path)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _duckdb.IOException: IO Error: Could not set lock on file "/home/nitro/nitro-repos/scrapers/duckdb/nrl_bucket.duckdb": Conflicting lock is held in /usr/bin/python3.12 (PID 257784). See also https://duckdb.org/docs/stable/connect/concurrency | |
20260720_020120 |
vm | R21 | refresh | 112 | success | failed | — | Traceback (most recent call last): File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 318, in <module> sys.exit(main()) ^^^^^^ File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 258, in main con = duckdb.connect(str(db_path)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _duckdb.IOException: IO Error: Could not set lock on file "/home/nitro/nitro-repos/scrapers/duckdb/nrl_bucket.duckdb": Conflicting lock is held in /usr/bin/python3.12 (PID 257784). See also https://duckdb.org/docs/stable/connect/concurrency | |
20260719_200006 |
vm | R21 | refresh | 112 | success | failed | — | Traceback (most recent call last): File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 318, in <module> sys.exit(main()) ^^^^^^ File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 258, in main con = duckdb.connect(str(db_path)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _duckdb.IOException: IO Error: Could not set lock on file "/home/nitro/nitro-repos/scrapers/duckdb/nrl_bucket.duckdb": Conflicting lock is held in /usr/bin/python3.12 (PID 257784). See also https://duckdb.org/docs/stable/connect/concurrency | |
20260719_140006 |
vm | R21 | refresh | 112 | success | failed | — | Traceback (most recent call last): File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 318, in <module> sys.exit(main()) ^^^^^^ File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 258, in main con = duckdb.connect(str(db_path)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _duckdb.IOException: IO Error: Could not set lock on file "/home/nitro/nitro-repos/scrapers/duckdb/nrl_bucket.duckdb": Conflicting lock is held in /usr/bin/python3.12 (PID 257784). See also https://duckdb.org/docs/stable/connect/concurrency | |
20260719_080021 |
vm | R21 | refresh | 102 | success | failed | — | Traceback (most recent call last): File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 318, in <module> sys.exit(main()) ^^^^^^ File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 258, in main con = duckdb.connect(str(db_path)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _duckdb.IOException: IO Error: Could not set lock on file "/home/nitro/nitro-repos/scrapers/duckdb/nrl_bucket.duckdb": Conflicting lock is held in /usr/bin/python3.12 (PID 257784). See also https://duckdb.org/docs/stable/connect/concurrency | |
20260721_133905 |
vm | R21 | open | 192 | success | failed | — | Traceback (most recent call last): File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 318, in <module> sys.exit(main()) ^^^^^^ File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 258, in main con = duckdb.connect(str(db_path)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _duckdb.IOException: IO Error: Could not set lock on file "/home/nitro/nitro-repos/scrapers/duckdb/nrl_bucket.duckdb": Conflicting lock is held in /usr/bin/python3.12 (PID 257784). See also https://duckdb.org/docs/stable/connect/concurrency | |
20260721_133537 |
vm | R21 | open | 192 | success | failed | — | Traceback (most recent call last): File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 318, in <module> sys.exit(main()) ^^^^^^ File "/home/nitro/nitro-repos/scrapers/scripts/stage_odds_ticks.py", line 258, in main con = duckdb.connect(str(db_path)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _duckdb.IOException: IO Error: Could not set lock on file "/home/nitro/nitro-repos/scrapers/duckdb/nrl_bucket.duckdb": Conflicting lock is held in /usr/bin/python3.12 (PID 257784). See also https://duckdb.org/docs/stable/connect/concurrency | |
20260721_044343 |
vm | R21 | close | 0 | zero_rows | zero_rows | — | ||
20260721_044217 |
vm | R21 | close | 0 | zero_rows | zero_rows | — | ||
20260720_071134 |
vm | R21 | close | 0 | zero_rows | zero_rows | — |