EDGE Bet Model dashboard

RLWC 2026 ingestion — Phase 1 implementation report

Branch rlwc-2026-ingest (from main @ 422733c). Phase 1 of the "Rugby League World Cup 2026 Ingestion" action brief: make collection competition-safe, capture the full World Cup baseline, and quarantine past contamination. Phase 2 work is listed at the end.

Status

Brief item Phase 1 state
1 Competition registry Done. nrl_scraper/competitions.yaml + competitions.py (one loader/validation API)
2 Draw fails closed Done. Competition always pinned; competition/season/round echo checked; mismatch → quarantined run, no records
3 Contamination audit Done. scripts/audit_competition_contamination.py; 2 runs quarantined (below)
4 Raw/stage contracts Done for draw, team lists, match stats. raw/rugby_league/competition=…/stage=…/run_id=…; legacy raw/nrl/… untouched
5 Competition-aware staging Done for draw, team lists, match stats (scripts/stage_rugby_league.py). Season stats, standings, odds → Phase 2
8 Team lists / match stats Done. Competition argument, competition-scoped fixture selection, match-id keys, no fixed roster size
11 Recommendations disabled Done in the pre-recommendation gate. Web display of the "Unavailable" message → Phase 2 (no World Cup data in the UI yet)
6 dbt, 7 standings, 9 task surface, 10 odds, scheduling Phase 2

Commands

# Collect + stage everything for all three World Cups, then reconcile to the contract
NRL_UPLOAD_ENABLED=true python scripts/world_cup_ingest.py bootstrap --season 2026
python scripts/world_cup_ingest.py reconcile --season 2026

# Single competition / stage
NRL_UPLOAD_ENABLED=true python run_spiders.py --spider draw --competition rlwc_men --year 2026 --round 2
python scripts/stage_rugby_league.py --season 2026 --competition rlwc_men --dataset draw

# Contamination audit (report only), then quarantine, then release after review
python scripts/audit_competition_contamination.py --since 2026-10-15
python scripts/audit_competition_contamination.py --since 2026-10-01 --quarantine
python scripts/audit_competition_contamination.py --release <run_id> --reason "<why>"

# Gate: World Cup competitions are always blocked
python scripts/check_nrl_recommendation_gate.py --season 2026 --round 2 --competition rlwc_men --no-persist

Results (bootstrap 25 Oct, refreshed 26 Oct 2026)

Fixture baseline: 53 of 53 staged, matching the registry contract exactly.

Competition ID Per stage (R1, R2, R3, SF, F) Total Complete
rlwc_men 131 5, 5, 5, 2, 1 18 10
rlwc_women 157 4, 4, 4, 2, 1 15 8
rlwc_wheelchair 130 4, 4, 4, 4, 4 20 0 (starts later)

Completed matches have lineups and post-match stats: 667 lineup rows and 667 player-stat rows.
- Lineups and stats agree for every match: men's 37–38 players per match, women's 36.
- No stats exist for incomplete matches, and no scores are stored on fixtures that aren't full time.
- Every draw row has a stable match_id, which embeds its competition ID. No natural key is duplicated.

Nicknames stay separate. Australia has team ID 500149 (men), 500216 (women) and 500712 (wheelchair). England, France, New Zealand, PNG, Samoa, Fiji and Wales are likewise separated by competition_key + match_id.

NRL data is unchanged. The full NRL dbt build has 153 PASS / 1 WARN / 0 ERROR. The warning is the existing ops_nrl_latest_run_in_gold finals-numbering issue.

Contamination audit

Window Raw runs scanned Critical Info
Since 15 Oct (brief window) 23 0 2
Since 1 Oct 56 4 (2 runs) 2
  • The 2 info findings are legitimate fixtures shared across tournaments: Australia v New Zealand in both the men's and wheelchair R1, and England v Wales in both the women's and wheelchair R1. They show why (season, round, teams) can never be a key.
  • Quarantined: 20261006T024432Z and 20261006T025320Z, legacy NRL draw runs for R24 and R25 from the 6 Oct backfill.
  • Each holds a World Cup Final fixture labelled as NRL (premiership='Rugby League World Cup', slug rugby-league-world-cup).
  • Their raw objects are untouched (4 each), and the markers are at control/quarantine/runs/.
  • Both NRL and World Cup staging now skip them.
  • Reports: reports/audits/competition_contamination_20261025T082754Z.json (the quarantine run, also at control/quarantine/audits/) and …083008Z.json (15 Oct window).

Tests

  • tests/test_world_cup_ingest.py: 35 passed, 1 live smoke test (opt-in). It uses recorded fixtures in tests/fixtures/nrl_api/ (all 15 World Cup draw rounds, one NRL round, one completed and one upcoming match-centre payload).
  • Brief validation items covered: 1–8, 10, 13 and 14.
  • Item 9 (standings) and item 11 (odds) are Phase 2.
  • Item 12 (existing suites) is covered by the runs below.
  • The live smoke test (RUN_LIVE_SMOKE=1 pytest -m live tests/test_world_cup_ingest.py) passes against NRL.com.
  • Full root suite: 564 passed, 1 failed. The failure is the known test_line_move_leakage 11-vs-12 count, which also fails on clean main.
  • One earlier run also hit test_run_cluster_task::test_odds_close_without_round_runs_idempotent_close_window. It didn't reproduce in two further full runs or in isolation, so it's treated as flaky.
  • App tests (nrl-bet-advisor/tests): the same 3 failures as on main; nothing new.

Design decisions

  • Draw match IDs. /draw/data has no match ID, and undecided knockout fixtures use placeholder URLs (…/semi-final/game-1/) that change once teams are known. So tournament draw runs fetch each fixture's match-centre /data to get its stable matchId, checking competition.competitionId there as well. NRL draw runs keep their single request.
  • There is no URL fallback. If any fixture's ID can't be resolved (HTTP error, missing matchId, request failure), the run is marked failed and the next refresh retries.
  • Staging also rejects any draw run that contains a row without a match ID.
  • A competition-mismatch quarantine is never downgraded to failed.
  • Natural keys:
  • draw: competition_key|match_id (match ID required)
  • team lists: competition_key|match_id|player_id
  • match stats: competition_key|match_id|player_id
  • Run IDs are unique for tournament runs. They carry a random suffix (20261026T114048Z-d6d52b14), because second-resolution timestamps alone can collide between concurrent jobs. A collision would make two runs share a raw path (one overwriting the other's raw evidence) or a staging replace.
  • Run-scoped replaces are additionally keyed on (competition_key, run_id).
  • Legacy NRL run IDs keep their format for existing consumers.
  • Quarantine is append-only. A release writes a second marker instead of deleting the first.
  • Legacy compatibility. NRL keeps raw/nrl/…, the stg_nrl_* tables and its finals-at-R1–R3 convention. Draw records add competition fields plus round_number (the true source round, e.g. 28 for Finals Week 1).

Other fixes on this branch

  • Match-stats and team-lists staging matched round=3 against round=31; they now match the whole /round=N/ path segment.
  • run_spiders.py --year was silently ignored by the team-lists spider (which then scraped every season) and by match stats (which fell back to the current calendar year). Both spiders now accept year.

Known API coverage limitations

  • The NRL ladder route returns 404 for World Cup competitions, so standings must be derived (Phase 2).
  • The men's pools are published as "Pool A" and "Pool B & C" (combined).
  • The wheelchair Semi Final and Final rounds each carry 4 fixtures, including classification matches.
  • Squad sizes vary: women's teams name 18 per side, and men's 18–19.
  • Odds coverage for the World Cup hasn't been probed yet (Phase 2). No odds are mapped to World Cup fixtures.

Phase 2 (next)

  1. dbt: competition-neutral silver/gold models and compatibility views over stg_rl_*; competition dimensions in ops/DQ, load assurance and snapshot coverage.
  2. Derived standings with official tie-break rules, reconciled to the official RLWC pools page.
  3. run_cluster_task.sh world-cup-bootstrap / world-cup-refresh (due work at match grain, no-op when idle, publish only after validation).
  4. Odds: provider sport-key adapter and capability probe; missing coverage reported as unavailable.
  5. Web: World Cup views with "Unavailable: World Cup model not validated" in place of recommendations.
  6. go-cicd-ops contract: scrapers-world-cup-pipeline, */30 * * * *, active 2026-10-15 → 2026-11-16, using the existing PVC, writer Lease, working DuckDB copy, dbt/DQ gates, atomic promotion and immutable snapshot publication.
    - Don't edit the manifests until the image contract is tested.
    - Note that deploys are currently blocked (Tekton has produced no deploy since 17 Sep; probably an expired DigitalOcean token).
    - Until that's fixed, run World Cup refreshes from the VM.