EDGE Bet Model dashboard

NRL Operator Runbook

Scope

This runbook assumes the repo root is the canonical working directory on the Ubuntu VM.

Current operating context for the current-season flow: June 2026, Round 14.

It covers:

  • one-time historical backfill for completed seasons 2016 through 2025
  • one-time 2026 recent-round catch-up
  • weekly regular-season updates for 2026
  • separate monthly players snapshots for 2026
  • a local metadata catalog for quick bucket coverage checks
  • weekly finals updates for 2026
  • separate scrape and stage phases using the existing bucket-first architecture
  • optional dbt and Elementary data-quality build for draw, ladder, players, and stats gold models after staging
  • root-level commands as the supported operating pattern for this repo

Preflight

From repo root:

source .venv/bin/activate
python -c "import duckdb,boto3,scrapy; print('duckdb', duckdb.__version__); print('boto3 ok'); print('scrapy ok')"
python test_spaces.py --check-vars
python test_spaces.py

Install dbt and Elementary dependencies when you want to build the analytics models and local data-quality report:

pip install -r analytics/dbt_nrl/requirements-dbt.txt

Before any scrape batch that should write to Spaces:

export NRL_UPLOAD_ENABLED=true

Canonical working directory for every command below:

source .venv/bin/activate

Important Rules

  • Keep competition=111.
  • Keep historical 2016-2025 backfill separate from the in-progress 2026 workflow.
  • Do not use --stage-after-run for operator batches. Scrape first, then stage explicitly.
  • For seasonal stats scraping, omit --stat so the spider collects all stat categories for that season.
  • Current-season 2026 stats runs are YTD snapshots. They do not reconstruct prior weekly 2026 states unless the scraper logic is extended.
  • Weekly 2026 stats runs are point-in-time YTD checkpoints because the spider is called without --stat.
  • Current-season 2026 draw and ladder should use a one-time last-five-round catch-up, then continue with one weekly round input at a time rather than a full season reconstruction.
  • Current-season 2026 players is a season snapshot dataset, not a round-based dataset. Keep it in catch-up, then refresh it with a separate monthly operator run.
  • Close DBeaver before any staging step that writes duckdb/nrl_bucket.duckdb.
  • Preserve raw and staged history, but use stg_nrl_*_latest views or ops_nrl_run_inventory.is_latest_run = true when you want deduped analytics.
  • Run dbt only after staged Parquet exists. dbt reads artifacts/stage_local and writes silver/gold draw, ladder, players, and stats tables to duckdb/nrl_bucket.duckdb.
  • The dashboard Events tab now reads one unified event flow for upcoming, live, and completed matches; completed recommendation badges are derived from the recommended side versus the final score.
  • Stats gold dedupe grain is season + stat_category + player/team key + metric key, with the latest canonical row chosen by scraped_at_utc desc, then run_id desc, then rank_position asc.

Historical Backfill: 2016-2025

From the repo root:

export NRL_UPLOAD_ENABLED=true
./scripts/backfill_nrl_2016_2025.sh

What this does:

  • players: one run per season
  • stats: one run per season with --competition 111 --year <YEAR> and no --stat
  • draw: one run per round per season
  • ladder: one run per round per season

Round stop rule

The backfill script loops rounds 1..40 by default.

Primary stop condition:

  • draw writes 0 meaningful data rows
  • ladder writes 0 meaningful data rows

Fallback stop condition for site clamping:

  • draw repeats the same meaningful data rows as the prior round
  • ladder repeats the same meaningful data rows as the prior round

This keeps the operator flow simple and avoids needing perfect season-length metadata upfront.
The wider upper bound is intentional because finals can still appear as late round values, and out-of-range rounds can clamp to the terminal finals state instead of returning empty output.

Equivalent manual command shapes

For one season YEAR:

python run_spiders.py --spider players --competition 111 --year "$YEAR" --load-type historical
python run_spiders.py --spider stats --competition 111 --year "$YEAR" --load-type historical
python run_spiders.py --spider draw --competition 111 --year "$YEAR" --round "$ROUND" --load-type historical
python run_spiders.py --spider ladder --competition 111 --year "$YEAR" --round "$ROUND" --load-type historical

Historical staging

After the raw backfill scrape is complete and DBeaver is closed:

./scripts/stage_nrl_datasets.sh historical

Then build the dbt gold models and local docs:

bash scripts/dbt_build_nrl.sh --docs

To also refresh the Elementary local data-quality report:

bash scripts/dbt_dq_report_nrl.sh

Direct per-dataset alternatives:

python scripts/stage_bucket_runs.py --dataset draw
python scripts/stage_bucket_runs.py --dataset ladder
python scripts/stage_bucket_runs.py --dataset players
python scripts/stage_bucket_runs.py --dataset stats

Use the direct per-dataset commands when you want tighter control or to rerun a single dataset.

Current-Season Workflow: 2026

Treat 2026 as a separate current-season operator flow.
Do not reuse the 2016-2025 historical round sweep for 2026 draw.

Choosing the round value

Use the round currently published on nrl.com for the refresh you want to capture:

  • regular season: use the currently published regular-season round page
  • finals: use the currently published finals round page

The draw and ladder scrapers are being used here as a recent-round catch-up plus ongoing weekly refresh flow, not as a full 2026 round reconstruction.
The players scraper remains current-season snapshot style and is intentionally kept outside the weekly round wrapper.

One-time 2026 catch-up

Run this once when establishing or repairing the current-season starting point on the VM.
As of May 8, 2026, the target is to catch up the latest five published rounds for the round-based datasets and then take fresh current-season snapshots for the season-based datasets.

Recommended wrapper:

export NRL_UPLOAD_ENABLED=true
bash scripts/catch_up_2026_recent_rounds.sh <CURRENT_ROUND>

Equivalent manual sequence:

CURRENT_ROUND=<CURRENT_ROUND>
START_ROUND=$((CURRENT_ROUND - 4))
if [ "$START_ROUND" -lt 1 ]; then START_ROUND=1; fi

for ROUND in $(seq "$START_ROUND" "$CURRENT_ROUND"); do
  python run_spiders.py --spider draw --competition 111 --year 2026 --round "$ROUND" --load-type weekly
  python run_spiders.py --spider ladder --competition 111 --year 2026 --round "$ROUND" --load-type weekly
done

python run_spiders.py --spider players --competition 111 --year 2026 --load-type weekly
python run_spiders.py --spider stats --competition 111 --year 2026 --load-type weekly

What this catch-up means:

  • draw: one raw run per round across the latest five published rounds
  • ladder: one raw run per round across the latest five published rounds
  • players: one 2026 current-season snapshot
  • stats: one 2026 all-category YTD snapshot because --stat is omitted
  • the catch-up script stages with historical mode at the end so all newly caught-up round grains land in DuckDB and Spaces, not just the latest round

Repairing a specific 2026 round gap

Use this when the metadata catalog shows a hole inside the current-season round sequence, for example rounds 2-5 missing while later rounds are already present.

Recommended wrapper:

export NRL_UPLOAD_ENABLED=true
bash scripts/backfill_2026_round_range.sh <START_ROUND> <END_ROUND>

export NRL_UPLOAD_ENABLED=true
bash scripts/backfill_2026_round_range.sh 2 6

Round-gap example for rounds 2-5:

export NRL_UPLOAD_ENABLED=true
bash scripts/backfill_2026_round_range.sh 2 5

What this repair means:

  • it reruns only draw and ladder, because those are the round-grain datasets
  • it does not rerun players or stats, because those are season snapshots and do not need one run per round
  • it stages with historical mode at the end so each repaired round stays queryable in DuckDB and Spaces

Weekly regular-season run

Once the catch-up exists, use the weekly wrapper for in-season refreshes:

export NRL_UPLOAD_ENABLED=true
bash scripts/weekly_2026_round.sh <PUBLISHED_REGULAR_SEASON_ROUND>

What this weekly run is for:

  • draw: refresh the current-season round page so the prior week's completed results are reflected in the latest snapshot
  • ladder: refresh standings
  • stats: refresh the 2026 all-category YTD snapshot with no --stat
  • stage: run weekly staging automatically at the end after reminding the operator to close DBeaver

This is a refresh flow, not a repeated full reconstruction of every 2026 round.

Round 10 example while the season is partway through:

export NRL_UPLOAD_ENABLED=true
bash scripts/weekly_2026_round.sh 10

This captures the current published state of round 10 for draw and ladder, plus the current 2026 stats YTD snapshot.

After weekly staging completes, refresh the dbt gold tables:

bash scripts/dbt_build_nrl.sh

Run the visual data-quality report when you want a local HTML view of dbt test health:

bash scripts/dbt_dq_report_nrl.sh

Cluster task wrapper

Use scripts/run_cluster_task.sh as the stable command surface for scheduled
pipeline tasks. Future Kubernetes CronJobs should call this wrapper rather than
embedding long scrape, dbt, odds, or prediction command sequences in YAML.

Validate the runtime first:

bash scripts/run_cluster_task.sh validate-env

Weekly ingest for a published round:

export NRL_UPLOAD_ENABLED=true
bash scripts/run_cluster_task.sh weekly-ingest --season 2026 --round 18

Six-hour odds refresh:

bash scripts/run_cluster_task.sh odds-refresh --season 2026 --round 18

Close odds for the explicit current round:

bash scripts/run_cluster_task.sh odds-close --season 2026 --round 18

If odds-close is run without --round, the wrapper prints that kickoff-aware
close scheduling is not implemented yet and does not fake round detection.

Prediction generation:

bash scripts/run_cluster_task.sh predict --season 2026 --round 18

predict is diagnostic-only. It does not seed paper trades or publish
recommendations. The authoritative publication command is:

bash scripts/run_cluster_task.sh recommend --season 2026 --round 18 --market all

recommend reconciles Kubernetes manifests, stages them into the VM-owned
DuckDB, runs dbt and the recommendation DQ gate, and only then predicts,
seeds paper trades, and publishes artifacts. A failed, pending, zero-row, or
missing reconciliation blocks the command.

Odds refresh and close collection are bucket-first and upload immutable tick
objects plus _run.json manifests to DigitalOcean Spaces. Kubernetes pods are
stateless collectors; the VM is the single DuckDB writer. Use
--local-only only for an explicitly disposable diagnostic run.

Canonical cluster runtime contract

The wrapper is the operational entry point for Kubernetes CronJobs and VM
workers. Pods collect immutable objects; the VM stages and publishes the
DuckDB snapshot.

Task Required timing/ownership Command
odds-open Opening collection, idempotent per match/market run_cluster_task.sh odds-open --season 2026 --round N
odds-refresh Scheduled refresh collection and movement detection run_cluster_task.sh odds-refresh --season 2026 --round N
odds-close T-45 to T-75 close collection and CLV update run_cluster_task.sh odds-close --season 2026 --round N
teamlist-window Team-list timing observation run_cluster_task.sh teamlist-window --season 2026 --round N
recommend Reconcile, stage, dbt, pregame gate, then publish run_cluster_task.sh recommend --season 2026 --round N
reconcile Reconcile manifests and stage pending runs run_cluster_task.sh reconcile --season 2026 [--round N]
publish-ui-snapshot Publish verified versioned DB and stable pointer run_cluster_task.sh publish-ui-snapshot --season 2026 [--round N]

Use dq-gate --gate-mode pregame for recommendation eligibility,
--gate-mode close-audit for the per-match close window audit, and
--gate-mode settlement only after close data exists. A pregame gate does not
require close snapshots. --event-id scopes close-audit/settlement to one
match grain.

Bucket configuration uses NRL_BUCKET_NAME, NRL_BUCKET_ENDPOINT,
NRL_BUCKET_REGION, NRL_BUCKET_ACCESS_KEY, and NRL_BUCKET_SECRET_KEY.
The older DO_SPACES_* and SPACES_* names remain read-only compatibility
aliases. Never print credential values.

Close runs with no eligible rows are successful zero-row manifests with
record_count: 0 and object_keys: []; they must not create a ticks object.
Non-empty runs publish a ticks object and manifest containing record count,
checksum, season, round, snapshot label/context, collector, Kubernetes/VM
context, source commit, image reference/digest, market, and task type.

UI snapshot publication and recovery

publish-ui-snapshot uploads the versioned object first:
sync/duckdb/nrl_bucket.duckdb/run_id=<run_id>/nrl_bucket.duckdb. Only after
that succeeds does it update the stable pointer
sync/duckdb/nrl_bucket.duckdb._run.json. The pointer includes the source
commit, image reference/digest, timestamp, byte size, and SHA-256 checksum.
Readers validate the pointer, download the versioned object, and reject a
checksum or required-table mismatch. If publication fails, rerun the publisher;
the previous stable pointer remains authoritative.

Promotion assurance

Promotion requires an explicit market-only baseline. If the baseline has no
usable profit/stake evidence, the evaluator reports insufficient_evidence
and promotion is not treated as proven. The strategy UI displays this status.

Data-quality report:

bash scripts/run_cluster_task.sh dq-report

For command previews that do not call APIs, DuckDB, or dbt:

DRY_RUN=true bash scripts/run_cluster_task.sh weekly-ingest --season 2026 --round 18

DQ dashboard (Elementary OSS)

The web app exposes a Data Quality link in the top navigation that opens the Elementary OSS report.

Item Detail
Dashboard Elementary OSS
Generate report bash scripts/dbt_dq_report_nrl.sh
Open in app /dq

How it works:

  • dbt tests own the DQ rules (schema tests and singular tests in analytics/dbt_nrl/).
  • Elementary visualises the dbt run and test results as a self-contained HTML report.
  • The web app is read-only — navigating to /dq never invokes dbt, edr, or any scripts.

Routing logic at /dq:

  1. If NRL_DQ_DASHBOARD_URL is set → redirects to that URL (for a hosted Elementary instance behind Nginx, Cloudflare Access, or VPN).
  2. Else if analytics/dbt_nrl/target/elementary_report.html exists → serves it via /dq/elementary.
  3. Else → shows a status page with the generate command and expected file path.

Environment variables:

# Redirect to a hosted Elementary report (leave empty for local-file mode)
NRL_DQ_DASHBOARD_URL=

# Override local report path (default shown)
NRL_ELEMENTARY_REPORT_PATH=analytics/dbt_nrl/target/elementary_report.html

Monthly players run

Run players separately during the first weekly refresh you perform each new calendar month, or whenever you explicitly want a fresh current-season player snapshot.

Recommended sequence:

export NRL_UPLOAD_ENABLED=true
bash scripts/monthly_2026_players.sh

Round 10 example if you also want the monthly players refresh:

export NRL_UPLOAD_ENABLED=true
bash scripts/weekly_2026_round.sh 10
bash scripts/monthly_2026_players.sh

Weekly finals run

When finals begin, keep the same current-season refresh model and keep historical backfill separate.

Recommended sequence:

export NRL_UPLOAD_ENABLED=true
bash scripts/weekly_2026_round.sh <PUBLISHED_FINALS_ROUND>

What this finals run is for:

  • draw: refresh the published finals round page so the snapshot captures both the teams scheduled for the upcoming finals round and the results from the just-finished finals round
  • ladder: refresh only if the source still publishes meaningful finals-era ladder content
  • stats: refresh the 2026 all-category YTD snapshot

If ladder stops being relevant during finals, skip it with:

SKIP_LADDER=true bash scripts/weekly_2026_round.sh <PUBLISHED_FINALS_ROUND>

Equivalent manual current-season command shapes

python run_spiders.py --spider draw --competition 111 --year 2026 --round <PUBLISHED_ROUND> --load-type weekly
python run_spiders.py --spider ladder --competition 111 --year 2026 --round <PUBLISHED_ROUND> --load-type weekly
python run_spiders.py --spider stats --competition 111 --year 2026 --load-type weekly

Use these manual command shapes when you want to rerun or skip a specific weekly dataset without changing the wrapper script. The compatibility wrapper bash scripts/weekly_nrl_2026.sh <ROUND> still works, but bash scripts/weekly_2026_round.sh <ROUND> is the primary entry point.

Manual monthly players shape:

python run_spiders.py --spider players --competition 111 --year 2026 --load-type weekly

Staging Sequence

For stage-only reruns after any scrape batch and with DBeaver closed:

./scripts/stage_nrl_datasets.sh weekly

What the wrapper does:

  • weekly: stages only the latest raw run per dataset
  • historical: stages all raw runs for the selected datasets
  • use historical after the one-time 2026 catch-up or the 2016-2025 backfill because those batches produce multiple new round grains
  • use weekly after the single-round weekly 2026 wrapper because that batch produces one new run for draw, ladder, and stats
  • use weekly after the separate monthly players wrapper because that batch produces one new players run

Direct per-dataset alternatives:

python scripts/stage_bucket_runs.py --dataset draw --max-runs 1
python scripts/stage_bucket_runs.py --dataset ladder --max-runs 1
python scripts/stage_bucket_runs.py --dataset players --max-runs 1
python scripts/stage_bucket_runs.py --dataset stats --max-runs 1

DBeaver Lock Warning

If DBeaver has duckdb/nrl_bucket.duckdb open, staging can fail because DuckDB uses a file lock.

Recommended practice:

  1. Finish the scrape batch first.
  2. Close DBeaver.
  3. Run staging.
  4. Reopen DBeaver after staging completes.

The helper script ./scripts/stage_nrl_datasets.sh checks that the local DuckDB catalog can be opened before it starts.

Validation Checks

Local CSV checks (debug only)

These files should exist after a run:

  • Output_Sheets/draw.csv
  • Output_Sheets/ladder.csv
  • Output_Sheets/players.csv
  • Output_Sheets/stats.csv

Format-aware row counts:

python scripts/count_csv_rows.py Output_Sheets/draw.csv --dataset draw
python scripts/count_csv_rows.py Output_Sheets/ladder.csv --dataset ladder
python scripts/count_csv_rows.py Output_Sheets/players.csv --dataset players
python scripts/count_csv_rows.py Output_Sheets/stats.csv --dataset stats

Bucket metadata catalog

Refresh the local metadata catalog from raw bucket manifest keys:

source .venv/bin/activate
bash scripts/check_nrl_bucket_status.sh

What this produces:

  • local DuckDB catalog: duckdb/nrl_metadata_catalog.duckdb at the repository root
  • manifest inventory file: artifacts/catalog/nrl_manifest_inventory.ndjson at the repository root
  • DuckDB views for analysis:
  • cat_nrl_manifest_inventory
  • cat_nrl_round_coverage
  • cat_nrl_season_coverage
  • cat_nrl_status_check

Use cat_nrl_status_check for a compact year-by-year status view and cat_nrl_round_coverage when you want to inspect the individual round grains captured for draw or ladder.

DuckDB checks

python - <<'PY'
import duckdb
con = duckdb.connect("duckdb/nrl_bucket.duckdb")
for table in ["stg_nrl_draw", "stg_nrl_ladder", "stg_nrl_players", "stg_nrl_stats"]:
    print(table, con.execute(f"select count(*) from {table}").fetchone()[0])
for table in ["stg_nrl_draw_latest", "stg_nrl_ladder_latest", "stg_nrl_players_latest", "stg_nrl_stats_latest"]:
    print(table, con.execute(f"select count(*) from {table}").fetchone()[0])
for table in ["gold_nrl_draw_latest", "gold_nrl_ladder_latest", "gold_nrl_players_latest", "gold_nrl_stats_latest"]:
    print(table, con.execute(f"select count(*) from {table}").fetchone()[0])
print(con.execute("select dataset, run_id, record_count, status from ops_nrl_run_inventory order by started_at_utc desc").fetchall())
con.close()
PY

Gold stats duplicate check. Expect 0 rows:

select
    season,
    stat_category_key,
    player_key,
    team_key,
    metric_key,
    count(*) as duplicate_rows
from gold_nrl_stats_latest
group by 1, 2, 3, 4, 5
having count(*) > 1
order by duplicate_rows desc, season desc, stat_category_key, player_key, team_key;

Historical coverage check

After the one-time backfill stage:

  • stg_nrl_* data should include seasons 2016 through 2025
  • 2026 should only appear after you intentionally run the one-time catch-up or weekly current-season flow

The Odds API — Live Odds

Overview

Live NRL odds are fetched via nrl-bet-advisor/data/odds_api.py using The Odds API.
A single call retrieves h2h, spreads (line), and totals (over/under) for all upcoming fixtures.

  • Bookmakers (default): Betfair Exchange AU, PointsBet AU, SportsBet
  • Quota: 500 requests/month on the free tier. Each fetch_nrl_odds() call = 1 request.
  • Markets returned: h2h (win/loss), spreads (handicap line), totals (over/under)
  • Key env var: ODDS_API_LIMIT_500MONTHLY (or ODDS_API_KEY) in the root .env

Setup

Add to your .env:

ODDS_API_LIMIT_500MONTHLY=your_key_here

Usage

From within the prediction / paper trading workflow:

from data.odds_api import fetch_nrl_odds
df = fetch_nrl_odds()
# Returns DataFrame with columns:
# event_id, kickoff_utc, home_team, away_team,
# home_odds, away_odds, bookmaker_count,
# home_line, home_line_odds, away_line_odds,   ← spreads
# total_line, over_odds, under_odds             ← totals

Paper trades are seeded only by the gated recommend command above. Direct
calls to the setup script are unsupported for production operation.

python scripts/setup_paper_trading.py --season 2026 --round <ROUND>

Quota management

  • 500 requests/month on the free tier. fetch_nrl_odds() uses exactly 1 request per call regardless of how many markets are requested (h2h+spreads+totals are bundled).
  • Check remaining quota in the response headers — logged automatically.
  • Avoid calling more than once per hour intraday; once per day is sufficient.

Current quota status (as of 2026-07-20)

392/500 used · 108 remaining. No plan upgrade planned — work within budget.

R21 estimated spend (week of 2026-07-21):

Calls Count Notes
tlwindow CronJobs (Tuesday) 4 06:05 / 09:05 / 12:05 / 18:00 UTC
Open snapshot 1 Tuesday/Wednesday when market publishes
Mid-week refreshes ≤3 k8s CronJob at 6h intervals — do NOT increase frequency
Close snapshots 8 One per kickoff, runs ~T−40min
R21 total ≤16

Remaining budget after R21: ~92 calls. Monthly counter resets with the billing
cycle (~1 Aug) restoring the full 500 for R22 onward.

Schedule discipline for R21: do not run manual test calls against the API
this week. If verifying the tlwindow fix, use --dry-run (no API call made).
If the k8s refresh CronJob (0 */6 * * *) is running through the week, that
adds ~44 calls (11 days × 4/day) — still within the 108 budget combined with
R21's 16 operational calls, but leaves no headroom for unplanned reruns.