NRL Data Quality v2

Stack overview

Scrapy spiders
  → DigitalOcean Spaces (raw NDJSON + _run.json manifests)
    → stage_bucket_runs.py (partitioned Parquet + ops_nrl_run_inventory)
      → dbt silver (read_parquet, normalise types)
        → dbt gold (latest-run dedup, canonical grains)
          → dbt DQ: load assurance (tag:dq_load)
          → dbt DQ: business rules (tag:dq_business)
            → Elementary report / alerts
              → FastAPI website reads gold tables

Layers and ownership

Layer	Owner	Location
Raw scrape	Scrapy pipeline	Spaces `raw/nrl/dataset=*/…/_run.json` + `records.ndjson.gz`
Stage	`stage_bucket_runs.py`	`artifacts/stage_local/dataset=/season=/round=/.parquet`
Ops inventory	`stage_bucket_runs.py`	`ops_nrl_run_inventory` (DuckDB table)
Silver	dbt	`silver_nrl_{draw,ladder,players,stats}`
Gold	dbt	`gold_nrl_{draw,ladder,players,stats}_latest`
Load assurance	dbt	`ops_nrl_load_assurance` + `tag:dq_load` tests
Business DQ	dbt	`tag:dq_business` tests on gold tables
Reporting	Elementary `edr report`	`target/elementary_report.html`
Alerts	`dbt_dq_monitor_nrl.sh`	Optional; enabled via `NRL_DQ_ALERTS_ENABLED=true`

dbt owns all data rules and their severity. Elementary visualises them and
optionally routes alerts. OpenMetadata is explicitly out of scope.

Severity guide

Severity	Meaning	Action on failure
`error`	Hard data contract broken — gold output is not trustworthy	Block consumption; investigate before shipping
`warn`	Expectation gap or schedule ambiguity — likely a known edge case	Log; review in Elementary report; escalate if persistent

What is always `error`

Duplicate rows in a canonical gold grain (match_grain, player_key, stats_grain)
Required-field nulls (season, player_key, stat_category_key, business_entity_key)
Successful scrape runs with zero staged rows
played != wins + drawn + lost in ladder
Home and away teams are the same match
is_latest_run = true count per grain is not exactly 1

What starts as `warn` (promote to `error` once seeds are reliable)

Match count per round differs from round_match_expectations seed
Ladder team count outside ladder_team_count_expectations seed
Scores partially present (one score null, other not)
Duplicate ladder positions per season/round
Negative stat values
Latest inventory run not reflected in gold

Load assurance (`tag:dq_load`)

Load assurance answers: did the raw scrape, staging, and dbt build all complete
correctly?

Model: `ops_nrl_load_assurance`

Joins ops_nrl_run_inventory (populated by stage_bucket_runs.py) to silver
row counts grouped by (dataset, run_id).

Key columns:

Column	Meaning
`dataset`	draw / ladder / players / stats
`season`, `round`	Scrape target grain
`run_id`	Unique scrape run identifier
`status`	success / failed / … from the raw manifest
`is_latest_run`	True for the canonical run per (dataset, season, round)
`expected_record_count`	`record_count` from the `_run.json` manifest
`staged_row_count`	Actual rows found in silver for this run_id
`record_count_ok`	NULL for non-success runs; true/false for success runs

Tests

Test	Severity	What it checks
`ops_nrl_successful_runs_have_staged_rows`	error	Successful runs with `record_count > 0` must have staged rows
`ops_nrl_record_count_matches_staged`	error	`staged_row_count` must equal `expected_record_count` for all success runs
`ops_nrl_one_latest_run_per_grain`	error	Exactly one `is_latest_run = true` per (dataset, season, round)
`ops_nrl_latest_run_in_gold`	warn	Latest draw run_id in inventory must appear in `gold_nrl_draw_latest`

Business-rule DQ (`tag:dq_business`)

Business DQ answers: does the NRL data make sense as a sports dataset?

Draw rules

Test	Severity	Rule
`gold_nrl_draw_no_duplicate_match_grain`	error	One row per match_grain
`gold_nrl_draw_home_away_different`	error	Home team ≠ away team
`gold_nrl_draw_scores_consistency`	warn	Scores are either both present or both absent (no partial score)
`gold_nrl_draw_round_match_count`	warn	Match count per round matches `round_match_expectations` seed

Ladder rules

Test	Severity	Rule
`gold_nrl_ladder_no_duplicate_season_round_team`	error	One row per (season, round, team_key)
`gold_nrl_ladder_current_no_duplicate_season_team`	error	One row per (season, team_key) in current view
`gold_nrl_ladder_played_formula`	error	`played = wins + drawn + lost` where all non-null
`gold_nrl_ladder_points_diff_formula`	error	`abs(points_diff − (points_for − points_against)) ≤ 0.5`
`gold_nrl_ladder_unique_positions`	warn	No two teams share the same ladder_position in a round
`gold_nrl_ladder_expected_team_count`	error	Team count matches `ladder_team_count_expectations` seed

Players rules

Test	Severity	Rule
`gold_nrl_players_unique_player_grain`	error	One row per (season, player_key)
`gold_nrl_players_no_duplicate_active_records`	error	No duplicate active (team non-null) records per player
`gold_nrl_players_required_fields`	error	season, player_key, player_name, team all non-null

Stats rules

Test	Severity	Rule
`gold_nrl_stats_unique_stats_grain`	error	One row per stats_grain
`gold_nrl_stats_no_duplicate_canonical_rows`	error	No duplicate (season, stat_category_key, player_key, team_key, metric_key)
`gold_nrl_stats_required_fields`	error	season, stat_category_key, metric_key, business_entity_key, stat_value_raw all non-null
`gold_nrl_stats_nonnegative_values`	warn	stat_value ≥ 0 where non-null

Expectation seeds

`round_match_expectations`

Controls how many matches are expected per round. The test selects the most
specific matching row (exact season+round > range > default).

Add a row here — rather than SQL logic — whenever a round has a known
non-standard match count:

expectation_name,enabled,season,round,round_start,round_end,expected_match_count,notes
state_of_origin_round,false,2025,12,,,7,Template: enable with exact season+round for Origin-impacted rounds

Rows with enabled = false are ignored by the test.

`ladder_team_count_expectations`

Controls expected team count per season range. Currently covers 2013–2026.
Add season-specific rows for expansion years (e.g., if a 19th team enters).

Running DQ

# Full build + all tests
bash scripts/dbt_build_nrl.sh

# Load assurance only
cd analytics/dbt_nrl && dbt test --select tag:dq_load

# Business rules only
cd analytics/dbt_nrl && dbt test --select tag:dq_business

# Elementary HTML report
bash scripts/dbt_dq_report_nrl.sh

# Alert wrapper (no-op unless NRL_DQ_ALERTS_ENABLED=true)
NRL_DQ_ALERTS_ENABLED=false bash scripts/dbt_dq_monitor_nrl.sh

Elementary alert wrapper (`dbt_dq_monitor_nrl.sh`)

The script wraps edr monitor and is safe by default — it exits cleanly with
a dry-run notice unless NRL_DQ_ALERTS_ENABLED=true is explicitly set.

Environment variables

Variable	Default	Purpose
`NRL_DQ_ALERTS_ENABLED`	`false`	Set to `true` to enable `edr monitor`
`NRL_DQ_ALERT_SUPPRESSION_HOURS`	`24`	Elementary alert suppression window
`NRL_DQ_ALERT_FILTERS`	`tags:critical`	Elementary test tag filter for alerts
`ELEMENTARY_SLACK_TOKEN`	—	Slack bot token (never commit)
`ELEMENTARY_SLACK_CHANNEL`	—	Slack channel for alert posts

Secrets (ELEMENTARY_SLACK_TOKEN, etc.) are read from the environment only and
must never be committed to the repository.

Intended future work

Promote warn rules to error once the expectation seeds are complete for all
Origin and finals rounds.
Add per-team stat anomaly detection via Elementary's elementary_tests.yml.
Extend ops_nrl_latest_run_in_gold to cover ladder, players, and stats.