EDGE Bet Model dashboard

NRL Bucket-First Target State

Summary

  • Canonical working directory: repo root
  • Runtime: local Ubuntu VM
  • Bucket: DigitalOcean Spaces
  • Stage engine: DuckDB
  • SQL UI: DBeaver
  • Scope: draw, ladder, players, stats
flowchart LR
    A["Scrapy spiders<br/>draw / ladder / players / stats"] --> B["Shared pipeline sink"]
    B --> C["Local debug CSVs<br/>keep for convenience"]
    B --> D["DigitalOcean Spaces raw zone<br/>html/json/ndjson.gz + manifest"]
    D --> E["Local DuckDB stage job"]
    E --> F["Spaces stage zone<br/>partitioned parquet"]
    F --> G["DuckDB catalog / views"]
    G --> H["DBeaver on local machine"]

Environment

The pipeline reads these variables first:

  • NRL_BUCKET_ENDPOINT
  • NRL_BUCKET_REGION
  • NRL_BUCKET_NAME
  • NRL_BUCKET_ACCESS_KEY
  • NRL_BUCKET_SECRET_KEY
  • NRL_UPLOAD_ENABLED

For the current VM setup, the code also accepts the older SPACES_* names as a compatibility fallback.

Raw Layout

Every spider run writes immutable artifacts under:

raw/nrl/dataset=<dataset>/load_type=<historical|weekly>/season=<yyyy|unknown>/round=<rr|unknown>/run_date=<yyyy-mm-dd>/run_id=<utc_ts>/

Each run writes:

  • raw source payloads under sources/
  • records.ndjson.gz
  • _run.json

Each canonical record carries:

  • run_id
  • dataset
  • spider_name
  • scraped_at_utc
  • source_url
  • load_type
  • season
  • round
  • schema_version

Stage Layout

The local stage script writes partitioned Parquet to Spaces under:

stage/nrl/dataset=<dataset>/season=<yyyy|unknown>/round=<rr|unknown>/snapshot_date=<yyyy-mm-dd>/

It also creates a local DuckDB catalog plus:

  • stg_nrl_draw
  • stg_nrl_ladder
  • stg_nrl_players
  • stg_nrl_stats
  • ops_nrl_run_inventory
  • ops_nrl_file_inventory
  • optional local metadata catalog views such as cat_nrl_round_coverage and cat_nrl_status_check for raw bucket manifest coverage checks

Commands

Smoke-test bucket connectivity:

.venv/bin/python test_spaces.py

Run one spider and keep CSV debug output:

python run_spiders.py --spider draw --competition 111 --year 2025 --round 1

Run scraping plus local stage:

python run_spiders.py --spider draw --competition 111 --year 2025 --round 1 --stage-after-run

This is suitable for small spot checks.
For the 2016-2025 historical backfill and the 2026 operator flow, keep scrape and stage as separate phases when running spiders manually, and use the runbook at nrl-operator-runbook.md.

Run the stage script directly:

python scripts/stage_bucket_runs.py --dataset draw

2026 Operator Note

Keep the in-progress 2026 season separate from the 2016-2025 historical backfill:

  • historical 2016-2025: use the dedicated backfill flow with round-by-round draw and ladder
  • current-season 2026 draw and ladder: do a one-time catch-up for the latest five published rounds, then continue with one weekly round refresh at a time
  • current-season 2026 players: capture the current season snapshot during the one-time catch-up, then refresh it monthly with a separate operator command because it is not round-based
  • current-season 2026 stats: capture point-in-time YTD snapshots with no --stat so every available category is included; do not claim round-by-round reconstruction unless the scraper logic is extended
  • stage remains a separate explicit step after scrape batches; do not rely on --stage-after-run for operator batches
  • preserve raw and stage history, but use stg_nrl_*_latest views or ops_nrl_run_inventory.is_latest_run = true for deduped analytics

Validation

  • Confirm a run uploads sources/*, records.ndjson.gz, and _run.json with the same run_id.
  • Compare _run.json.record_count to the staged row count in DuckDB.
  • Check that stage files land under the correct dataset/season/round partitions.
  • Keep Output_Sheets/*.csv available only as local debugging outputs, not as the analytics source of truth.

DBeaver

Minimal local workflow:

  1. Open the local duckdb/nrl_bucket.duckdb file after a stage run.
  2. Query the stg_nrl_* views directly.
  3. If you want to query Spaces from DuckDB on another machine later, configure DuckDB httpfs plus S3-compatible settings for the Spaces endpoint and bucket credentials.

Close DBeaver before running staging commands that write duckdb/nrl_bucket.duckdb.