Match-observation archive — the proprietary dataset (2026-07-21)
⚠ API correction (2026-07-21): we bought a DIFFERENT vendor
The Business purchase is theoddsapi.com — NOT the the-odds-api.com
our legacy fallback collector (odds_api.py) uses. Different domain, auth
(x-api-key header vs ?apiKey=), no /v4/, and premium endpoints the old
one lacks. Client: data/theoddsapi_business.py; verify first with
scripts/probe_theoddsapi_business.py. The two keys (ODDS_API_BUSINESS
new, ODDS_API_KEY old) never cross. The live NRL collectors use the Business
provider by default; the old provider remains an explicit switch-back.
This vendor changes the archive economics: fair odds, consensus, and
Pinnacle-anchored edges are PRE-COMPUTED by the API (/intelligence/,
/edges/, /best-lines/). We still keep market_reference.py — not as the
only source, but as the independent validator: when the vendor's fair
odds disagree with our own de-vig, that disagreement is information. A vendor
"edge score" or "value bet" is a hypothesis to run through our own CLA/CLV
gates, never a signal to trust on faith.
Why this exists (the correction that forced it)
The Odds API BUSINESS plan gives historical point-in-time snapshots — but
for rugbyleague_nrl the historical archive only begins ~May 2026. The
multi-season intraweek backfill (2023→2026) that the ceiling-break argument
assumed does not exist for this sport. What the plan actually buys us:
- A one-time grab of 2026 intraweek history from ~May to now (~R10–R21,
~11 round-weeks) — small, bounded, and unrepeatable, so grab it. - Richer live access going forward — more books, more regions
(Pinnacle lives ineu), higher rate limits.
The real ceiling-break is therefore forward: from now, build our own
history nobody else has. Every week we don't capture is a week we can never
backfill. This archive is the highest-leverage build on the roadmap.
The atomic contract
One immutable fact table, stg_match_price_ticks (extends the existing
movement_odds_snapshots), one row per:
(match_id, snapshot_ts, bookmaker, market, side)
→ price, line, captured_at_utc, collector, run_id
Plus a match-level dimension, gold_match_observation, one row per match,
joining the context feeds and the derived reference columns below. The
dimension is what makes the archive queryable as "what did the whole market
+ world look like around this game."
What to store (owner's list ✓, plus what else is worth it)
Prices — every snapshot
| Field | Source | Have? | Note |
|---|---|---|---|
| Every bookmaker's price (h2h/line/total) | Business /odds/ with no shortlist |
✅ | decision snapshots retain every returned AU book |
| Pinnacle price | theoddsapi.com — all regions, 50+ books | ✅ available on new tier | the sharp anchor; confirm it returns for NRL via probe |
| Betfair Exchange price | betfair_ex_au |
✅ | real matched money |
| Fair odds (de-vigged) | vendor /intelligence/fair-odds/ + our market_reference.devig_two_way cross-check |
✅ both | store both; disagreement is information |
| Consensus price | vendor /intelligence/consensus/ + our consensus_price |
✅ both | store both |
| Overround per book | computed market_reference.overround |
✅ built | vig = liquidity/appetite proxy |
| Sharp-vs-soft divergence | our soft_vs_sharp_divergence (+ vendor /edges/) |
✅ built | THE open-softness signal; vendor edge score stored as a VALIDATION target, not trusted |
| Best line per outcome | vendor /best-lines/ |
✅ available | the price we could actually have taken |
Context — per match
| Field | Source | Have? | Note |
|---|---|---|---|
| Team lists | nrl.com Tue spider | ✅ | + capture time vs drop (timing channel) |
| Weather | Open-Meteo | ✅ | 99% coverage post-backfill |
| Injuries / availability | implicit: named r−1 w/ minutes, absent r | ❌ build | delta_availability_loss; casualty-ward scrape is a candidate feed |
| Final score | draw/results | ✅ | settlement |
What else is worth capturing (my additions — cheap, high-value)
- Betfair traded volume / matched £ (weight of money) — the single
most valuable field we don't have. The Odds API does NOT expose it; the
Betfair Exchange API does. Candidate feed — abetfair_client.py
scaffold already exists at repo root. Volume-weighted price movement is
a far stronger steam signal than price alone. - Market width / book count per snapshot — how many books are up on
each market. Books pulling a market = uncertainty (late mail, scratching
rumours). Free (count the payload). - Line/total availability timing — when each market first appears
relative to kickoff. Free (first-seen from the tick stream). - Kickoff-relative snapshot cadence — capture time as minutes-to-
kickoff, not just wall clock, so the pre-kickoff curve is comparable
across games. Free (computed). - Fan-pick % (public money proxy) — ESPN, already collected. The
divergence between public % and the sharp price is the classic
contrarian input. - Referee appointment — some refs correlate with penalty counts →
totals. Cheap scrape, speculative; log as a candidate, don't build yet.
Player props are now cheap to COLLECT on this tier (/props/: NRL try
scorer first/last/anytime, tries-over) — collect-and-archive for future
modelling, but no props model exists, so they are observation-only and never
drive a recommendation. In-play prices: still out of scope.
Multi-sport expansion (owner-directed 2026-07-21)
The confirmed archive sports in this branch are NRL, AFL, and UFC. Rugby union
and WNRL are not registered because the provider catalog does not expose them
on this plan. The sport archive is parameterised, but modelling remains NRL-only.
Capture cadence (forward)
| Phase | Cadence | Purpose |
|---|---|---|
| Monday | 1 snapshot | the open |
| Tuesday window | +5m/+30m/+1h/+2h/+4h/+12h from team-list drop | repricing curve (k8s tlwindow jobs, live) |
| Wed–Thu | every 6h | ambient path |
| Final 60 min pre-kickoff | every 5–10 min | the close curve — where CLV is won |
| Post-match | final score | settlement |
The final-hour 5–10 min cadence is the owner's core ask and the piece we
don't yet run. It is per-kickoff, so it costs ~6–12 calls per game per
week — the quota reason to keep the sharp/Pinnacle add scoped to the
snapshots that matter (open, Tuesday window, and the final-hour curve), not
every ambient tick.
Quota arithmetic (business plan)
Each derived column is FREE (computed from stored prices). The cost is
regions × markets × snapshots. Adding Pinnacle means regions=au,eu — call
cost roughly doubles. Keep the two-region calls to the decision snapshots
(open, Tuesday window, final-hour curve); use au-only for ambient ticks.
The backfill walker's measured-cost discipline (odds_api_history.py)
applies here too — measure before scaling cadence.
Build order (spec for the CLI round)
market_reference.py— DONE (de-vig, consensus, anchor, divergence + tests).- Add Pinnacle:
regions=au,eu, bookmaker keypinnacle, on the decision
snapshots only. Confirm it returns for NRL before relying on it. - Final-hour 5–10 min capture: new cron lines (VM) or a k8s CronJob keyed
off the staged draw's kickoff times. stg_match_price_ticksstaging +gold_match_observationdimension
(dbt) joining team lists, weather, availability, final score, and the
derived reference columns.- Grab the May-2026→now historical intraweek data ONCE via the backfill
script (it's real 2026-season data, unrepeatable) into the same tables. - Only then: the repricing-curve and line_move_v1 evidence run — now on a
proprietary, richer dataset than the open/close file ever was.