Data Ingest Strategy
Scope: every byte that enters the system — source, cadence, owner,
landing zone, and target state. Companion to docs/STRATEGY.md (why) and
docs/nrl-bucket-target-state.md (bucket layout detail).
Principles (unchanged, restated once)
- Bucket-first, immutable raw. Every ingest lands raw artifacts +
_run.jsonmanifest in DO Spaces before anything transforms it. Raw is
append-only forever. - The edge decides the priority. Ingest exists to feed the promotion
gates. The two feeds that starve the program if late: opening odds
(Monday) and pre-kickoff close snapshots — they are the CLV
instrument. Everything else can slip a day; those two cannot. - One writer. The VM writes DuckDB; k8s pods read. No pod ever writes
the analytics DB.
Feed inventory (current state)
| Feed | Source | Cadence | Trigger | Lands in | Feeds |
|---|---|---|---|---|---|
| Odds — open snapshot | The Odds API | Mon each round | VM cron | movement_odds_snapshots |
CLV baseline, paper-trade pricing, open-softness edge |
| Odds — refresh | The Odds API | ~6h in round week | k8s CronJob or VM cron | same | movement strategies |
| Odds — close snapshot | The Odds API | per kickoff −40min | scripts/schedule_close_snapshots.py cron lines |
same | CLV — the promotion currency |
| Draw / results | nrl.com (draw spider) | weekly post-round | weekly_2026_round.sh |
raw bucket → gold_nrl_draw_latest |
settlement, Elo/SS updates |
| Ladder | nrl.com | weekly | weekly script | gold ladder | UI, form features |
| Team lists | nrl.com | Tue ~4pm weekly | team_lists spider | stg_team_lists |
RAPM, lineup features, timing channel |
| Player match stats | NRL match-centre API | weekly post-round | weekly script (match_stats) + dbt gold_nrl_player_round_stats |
gold player stats | RAPM stats-priors, player value |
| Player roster/profiles | nrl.com players | monthly | monthly_2026_players.sh |
silver players → gold_nrl_player_name_map |
identity resolution |
| Weather | Open-Meteo | weekly + pre-round | ingest_nrl_weather.py |
ml_nrl_match_weather_features |
totals models, filters |
| Fan picks | ESPN | weekly | fan-pick-loader skill |
stg_fan_picks |
crowd feature (observation) |
| Historical odds | nrl_historical.csv (static) | one-off | manual | repo file | backtests |
| Odds — team-list window | The Odds API | Tue T+5m/T+35m/T+1h/T+2h/T+4h/T+12h UTC | k8s CronJobs (scrapers-tlwindow-*, go-cicd-ops) |
Spaces → reconcile → snapshots | line-v2 P3 repricing-speed measurement |
| Odds — HISTORICAL intraweek snapshots | The Odds API /v4/historical (BUSINESS key) |
ONE-TIME grab: 2026 season-to-date only (coverage starts ~May 2026) | scripts/backfill_odds_api_history.py (VM) |
raw/nrl/dataset=odds_api_history/ → staging TBD |
2026 intraweek paths (NOT multi-season — see below) |
| Match-observation archive (FORWARD) | live capture, all books + Pinnacle + Betfair Exchange | open + Tue window + 5–10min final hour | VM/k8s (spec: docs/match-observation-archive.md) |
stg_match_price_ticks → gold_match_observation |
the real ceiling-break — proprietary history from now |
Historical coverage correction (2026-07-21): rugbyleague_nrl on The Odds
API only has historical snapshots from ~May 2026 — there is no 2023–2025
intraweek history to buy. The backfill is a one-time 2026-season-to-date
grab; the durable edge comes from the forward archive
(docs/match-observation-archive.md), which captures every book, Pinnacle
(sharp anchor), Betfair Exchange (matched money), de-vigged fair/consensus
prices, team lists, weather, availability, and final score — the dataset
nobody else has.
Vendor + provider policy (2026-07-21): the program converted to
theoddsapi.com (Business) as PRIMARY (data/odds_provider.py,
ODDS_PROVIDER=theoddsapi); the-odds-api.com (Basic/free) is kept intact
as the season-dependent fallback — flip with the ODDS_PROVIDER env, no
code rewrite. Keys never cross (ODDS_API_BUSINESS vs ODDS_API_KEY).
Odds format: theoddsapi defaults to AMERICAN — every request forces
oddsFormat=decimal and every stored price passes odds_format.coerce_decimal
defensively (endpoints like /edges//intelligence/ may ignore the param).
Confirmed sports on the plan: NRL, AFL, UFC (rugby union + WNRL are NOT on
the 26-sport plan). Multi-sport is capture-only; NRL keeps modelling priority.
API key separation (standing rule, 2026-07-21): ODDS_API_BUSINESS is
the active live and historical provider key by default; ODDS_API_KEY remains
an explicit legacy fallback selected with ODDS_PROVIDER=the_odds_api.
Keys never cross. Business usage is measured from response headers; backfill
runs retain their --credit-budget and --min-remaining hard stops and
resumable ledger (reports/odds_api_history_ledger.json).
Candidate feeds (not built — build only on demonstrated need):
| Candidate | Source | Would feed | Status |
|---|---|---|---|
| Advanced off-ball stats (kick pressures, decoy runs, support runs) | NRL match-centre (exposed, not scraped) | delta_offball_effort v2 (line-v2 roadmap Feature 4) |
Composite from existing fields ships first; extend match_stats spider only if it shows signal |
| Injury / casualty ward | nrl.com weekly casualty ward (semi-structured) | explicit availability + return timelines | Implicit availability delta (team lists + minutes, already staged) ships first — see line-v2 roadmap |
Target state (season 2026 → 2027 pre-season)
- Close snapshots fully automated per kickoff (cron generated from the
staged draw). Alert (Jobs page → red) when a round'sclv_populated<
settled trades. Status: tooling shipped; verify weekly. - Team-list capture within 30min of the Tuesday drop, timestamped, so
the timing channel measures the announcement edge, not scrape lag.
captured_at_utcadded tostg_team_lists(populated from spider start
time viaannounced_at_utc). k8s CronJob fires at0 6 * * 2
(Tuesday 06:00 UTC = 4pm AEST)./opsfeed monitor tracks capture time. - Weekly player stats never lag the draw (Jobs page tracks round lag;
weekly script rebuilds the gold models — keep it green). - Betfair exchange feed (evaluate):
betfair_client.pyexists at repo
root as legacy scaffolding. Exchange prices would give (a) sharper closing
lines than bookmaker consensus and (b) historical odds depth. Evaluate
cost/effort next off-season; do NOT build during the season. - Ingest observability: every feed row above appears on
/opswith an
explicit cadence and staleness status — a feed that isn't on the Jobs page
doesn't exist operationally. Fan picks and roster refresh added to
feed_monitor_service.py. - Backfill discipline: historical gaps (team lists pre-2016, odds
history granularity) are accepted and documented in coverage tables —
spend effort on capture-going-forward, not heroic backfills.
Odds API quota and fallback
Business usage is tracked with x-ratelimit-* response headers. The legacy
fallback remains capped at 500 requests/month; one request per collector
call (h2h + spreads + totals bundled).
As of 2026-07-20: 392/500 used, 108 remaining. No upgrade planned — adjust schedule to stay within budget.
R21 allocation (week of 2026-07-21):
- 4 tlwindow CronJobs (Tuesday) + 1 open + ≤3 refreshes + 8 closes = ≤16 calls
- Mid-week refresh CronJob (0 */6 * * *) adds ~44 calls for the remaining 11 days of July
- Combined worst-case ≈ 60 calls → leaves ~48 buffer before August reset
Schedule rule: do NOT increase the 6h refresh frequency or run manual API calls this week. Monthly counter resets ~1 Aug, restoring 500 for R22 onward. Detailed breakdown and per-round budget table: docs/nrl-operator-runbook.md → Quota management.
Scheduling split (who runs what)
- VM cron: weekly round script (scrape+stage+dbt), close-snapshot cron
lines, monthly players, DuckDB → PVC sync (to be documented per
docs/ui-deployment-gaps.mdgap 4). - k8s CronJob:
- Odds refresh collection —
scrapersCronJob,0 */6 * * *, runs the
bucket-first collector only. Detection runs on the VM after reconciliation. - Team lists —
scrapers-team-listsCronJob,0 6 * * 2(Tue 06:00 UTC
= 4pm AEST), runsscripts/run_team_lists.py. Hop-1 only; VM staging
runs after. - Candidate for close snapshots if the VM proves unreliable at kickoff times.
- Human-triggered: backfills,
fan-pick-loader, anything touching raw
bucket layout.
Failure playbook
| Symptom | First check | Fix |
|---|---|---|
| /ops shows close-snapshot STALE in round week | VM crontab installed for this round? | schedule_close_snapshots.py --print-cron and reinstall |
| CLV missing for settled trades | close snapshots exist pre-kickoff? | update_nrl_movement_clv.py --settle; legacy label fallback covers old rounds |
| Player stats behind draw | dbt step in weekly run | dbt build --select gold_nrl_player_name_map gold_nrl_player_round_stats |
| Team lists empty for a round | spider vs nrl.com markup | bucket-stage-debug skill |
| Hosted UI stale vs VM | image build date vs last phase commit | rebuild/redeploy image (gap 3) |