NRL Model & Edge Risk Audit — July 2026

Reviewer stance: sports analyst / former coach lens, plus quantitative risk audit.
Scope: all prediction models (Elo, logit bench, ridge line/total, LightGBM), the
edge-candidate framework (docs/nrl-edge-goal-10.md, docs/edge-5-movement-refinement-v1.md),
CLV machinery, paper trading, and staking policy.
Language rules respected: everything below is a candidate or hypothesis. Nothing
in this repo is a proven edge today — and the audit's headline is that two of the
promotion gates cannot currently be trusted as implemented.

Executive summary

The framework design is genuinely good — walk-forward by season, per-market separation,
CLV-gated promotion, paper-trade-first discipline, evidence-weighted voting. The problem
is that several implementations undercut the framework they're supposed to enforce:

The LightGBM model is trained on closing-line data (home_line_close, closing-odds
implied probability) — a direct violation of the repo's own leakage rules — and it
escapes the leakage guard because that guard only covers the model_bench specs.
The live LightGBM prediction path is broken: nrl/src/predict.py builds 23 features
for a model trained on 25 and raises LightGBMError (verified by execution). The
"Elo + LightGBM agreement" filter in the edge-review skill is therefore built on a
model that is both leaky in backtest and inoperable live.
The CLV measurement protocol is structurally wrong: the "close" snapshot is
collected once per round, after the last game completes. Closing lines for Thursday/
Friday games cannot be captured on Sunday night — The Odds API drops events at
commence time. CLV is the single most important promotion gate in the framework, and
as specified it will be null or corrupted for most games in a round.
CLV is computed averaged-vs-averaged (consensus taken_price, AVG(price) close),
which nrl-edge-goal-10.md itself forbids ("raw odds snapshots are the source of
truth for CLV").
The headline ROI narrative is post-hoc selection. "2024–2026 is positive" was
observed after scanning 17 seasons × 4 models × 2 policies. No significance testing,
no multiplicity control. At ~150 bets/season, ±20% season ROI is well within noise —
the status doc admits this but the recency-weighted framing still leans on it.

None of this means the project is unsound. It means the current green lights are not
yet earned. Fix the measurement instruments first; only then do model improvements mean
anything.

Scorecard — what the numbers actually say

Model family	Honest read
`elo_current`	Solid baseline. Log-loss 0.640 vs 0.693 coin-flip — real signal, no market edge. Long-run ROI −2.4% ≈ the vig. Behaving exactly as an Elo should.
`logit_core`	Marginally better than Elo recently; coefficients show it is ~90% Elo restated (`elo_rating_diff`, `elo_home_prob` dominate). It is not a second opinion; it is the same opinion with extra steps.
`logit_core_market` / `lineup_prior_year_v1`	Log-loss 0.619 — the best in the shop — because they include the closing market. That is the market's skill, not the model's. Identical outputs (lineup coefficients are 0.000) means the bench is carrying a duplicate.
`line_ridge_core`	+$0.38 over 1,727 bets is a coin toss with a spreadsheet. MAE 13.4 pts on a sport where the average line error is ~12–13 means the model has not beaten the market's own margin distribution.
`total_ridge_core`	−0.6% long-run. The `_market` variant's 2022–24 profit is driven by `close_total_points` (+2.23 coeff) — i.e. "the closing line predicts the total". True, and unusable.
`lgbm_v1`	62.15% accuracy vs 61.46% for "pick the odds favourite" — +0.7pp while being handed the closing market as input. Zero demonstrated skill beyond market, plus the defects in Findings 1–2.

Coach's translation: every model in the shop is a re-derivation of Elo plus the
market. There is no feature yet that the market doesn't already have by kickoff. The
market's information advantage is team news, and the repo scrapes team lists but the
lineup features contribute literally nothing (coefficients 0.000, coverage starts 2022 R4).
The one place a private edge could live is currently the deadest feature family in the bench.

Critical findings

F1 — Closing-line leakage in the LightGBM pipeline (severity: critical)

nrl/src/features.py:324-329 puts home_line_close and home_implied_prob in
FEATURE_COLS, and home_implied_prob is de-vigged from closing odds
(features.py:159-171, with close imputed from open when missing). nrl/src/train.py
trains and reports test metrics on these features.

The leakage guard (data/model_bench.py:219,
assert_no_actionable_closing_line_leakage) only inspects MODEL_BENCH specs — the
nrl/src pipeline is invisible to it. The guard passes while the flagship ML model
violates the rule it enforces.

Fix: retrain on opening odds/line only (or better: current-snapshot odds at a
declared decision time), and extend the leakage test to assert no
CLOSING_LINE_FEATURE_TOKENS appear in nrl.src.features.FEATURE_COLS.

F2 — Live prediction path is broken and train/serve-skewed (severity: critical)

Verified by execution: the shipped lgbm_v1.pkl expects 25 features
(incl. crowd_home_pct, kickoff_hour, is_evening, is_weekday); predict.py
constructs 23 and predict_proba raises
LightGBMError: number of features in data (23) is not the same as it was in training data (25).
The fallback logic at predict.py:119-124 cannot recover because the live frame also
contains two features (home_player_rating, away_player_rating) the model was never
trained on.

Even if the shape matched, predict.py:100-108 stubs home_line_close=0.0 and all
rolling-form features at neutral priors — the model's most informative inputs are
zeroed at exactly the moment it's asked for a real opinion. A model served with
placeholder features is a different model from the one that was validated.

Fix: single source of truth for the feature list (import FEATURE_COLS from
features.py), a CI test that round-trips load → build live frame → predict_proba,
and real feature computation (rolling form is computable pre-kickoff from staged data —
there is no excuse for a 0.5 stub).

F3 — CLV protocol cannot produce true closing lines (severity: critical)

docs/edge-5-movement-refinement-v1.md §CLV: "close snapshot: collected after last
game of the round completes." data/odds_movement.py:738-741 confirms the label is
collected post-kickoff, once per round. NRL rounds span Thursday→Sunday; The Odds API
removes markets at commence time. The "closing" price for a Thursday game either won't
exist on Sunday or won't be a closing price.

Every promotion gate in the framework routes through CLV. If the close snapshot is
wrong, the entire validation layer is measuring noise, and the strategies could be
"promoted" or "killed" on corrupted evidence.

Fix: capture a per-game close snapshot at T−5 to T−0 minutes before each kickoff
(the 6-hourly refresh cadence plus a kickoff-triggered snapshot job; kickoff times are
known from the draw). Define closing line = last pre-kickoff snapshot per event, per
bookmaker, and store it raw.

F4 — CLV computed on synthetic averaged prices (severity: high)

scripts/setup_paper_trading.py:225 books trades at consensus_avg;
data/odds_movement.py:749-756 computes the close as AVG(line), AVG(price) across
bookmakers. nrl-edge-goal-10.md explicitly says averaged odds must not be the only
retained market record. A consensus price is not a bettable price: average-taken vs
average-close CLV can be positive while every individual book's line was worse than
its own close. Fix: record per-bookmaker taken price (best available at decision
time, named book) and compute CLV against that same book's close, with the consensus
kept as a display aggregate only.

F5 — Backtest settles every bet at closing odds (severity: high)

data/backtest.py:_best_side prices EV and settles P&L from close_home_odds /
close_away_odds for all models. Two problems: (a) it assumes you always achieve the
closing price — the whole point of the CLV program is that you want to beat it, so ROI
at close understates a genuinely sharp strategy and flatters a square one; (b) the
_market model variants are given close_home_implied_prob as a feature and then
"bet" into that same closing price — the doc labels them diagnostics, but they sit in
the headline win-model table, in the season-ROI narrative, and in the paper-trading
seeds (32 rows = 8 matches × 4 models). Diagnostics in the recommendation path is how
leakage launders itself into decisions.

Fix: backtest actionable models at opening odds (that is the price you can
actually see at decision time in the historical data), and keep _market variants out
of any seeded paper trades and out of leaderboards that feed the app.

F6 — Open/close imputation destroys the movement signal (severity: medium)

backtest.py:442-465: close is imputed from open and open from close, so rows missing
one side get market_move = 0, home_line_move = 0, total_points_move = 0 exactly
where data is thin (older seasons). The movement features that Edge 5 depends on are
trained partly on manufactured zeros. Fix: add has_true_open / has_true_close
flags, exclude imputed rows from any movement-feature training, and report coverage.

F7 — Feature joins at the wrong grain (severity: medium, silent-corruption risk)

nrl/src/features.py:194-249 joins lineup, crowd-pick, and weather features on
(season, home_team, away_team) with no round or date, while the underlying table is
round-grained (the backtest queries round from the same table at
backtest.py:545-560). Any pairing that occurs twice in a season with the same home
team — which happens every NRL season, plus finals rematches — becomes a many-to-many
merge: duplicated training rows (sample-weight distortion) and features from the wrong
fixture (e.g. a Round 3 game receiving Round 22's lineup deltas — cross-fixture
leakage). Could not be verified against the DB (not present in this clone), but the
schema mismatch is conclusive. Fix: join on (season, round, home, away) or
kickoff date, and add a post-merge assertion len(df_after) == len(df_before).

F8 — Fake granularity in lineup deltas (severity: low)

backtest.py:585-587 maps delta_forward_strength identically into
lineup_middle_delta, lineup_edge_delta, and lineup_lock_delta. Three "different"
features, one number. Harmless to ridge/logit numerically, but it makes coefficient
tables lie about what's being measured and inflates the apparent feature coverage.

F9 — Promotion gates disagree with each other (severity: medium, governance)

Four different thresholds are live simultaneously:
.claude/skills/model-edge-review (≥50 bets, +5% ROI), nrl-edge-goal-10.md
(100–200+ bets), edge-5-movement-refinement-v1.md (≥100 trades, +5% ROI, median-positive
CLV, concentration cap), strategy_weights.py:PROMOTION_MIN_BETS = 100. Whichever
document a future session happens to read determines the bar a candidate has to clear.
Fix: one constants module (promotion_gates.py) imported by code and referenced by
every doc; a test asserting docs and code agree (the skill-sync script already sets the
precedent).

F10 — No statistical rigour on the headline claims (severity: high)

The status doc scans 17 seasons × 4 win models × 2 policies (plus 2×2 market models)
and highlights the recent positive cells. With ~150 edge_capped bets/season at ~$1.90
average odds, one-season ROI has a standard error of roughly ±8%, so ±20% seasons are
~2σ events that will appear somewhere in a 130-cell grid. The Edge 5 plan then layers
3 strategies × 11 filter overlays — a 33-way garden of forking paths with no
multiplicity control and no pre-registration. Fix: block-bootstrap season-level ROI
confidence intervals in the status doc; pre-register the filter set and evaluation
window per strategy before data collection (the doc already gestures at this —
"collect raw first, filters second" — make it binding); hold 2026 H2 out as untouched
confirmation data.

F11 — Staking policy has no risk layer (severity: medium)

edge_capped doubles stake when EV ≥ 15%. In a market with ~2% vig, a model that
regularly emits +15% EV is not finding value, it is miscalibrated — the policy
systematically doubles down on the model's worst probability estimates (this is the
classic favourite–longshot trap: max-EV side selection plus high-EV thresholds
preferentially picks underdog prices where model error is largest). There is no
per-round exposure cap, no drawdown stop, no fractional-Kelly sizing, and
EDGE_TARGET_EV = 0.10 as a filter has the same problem. Fix: shrink model
probability toward the market before computing EV (p_bet = λ·p_model + (1−λ)·p_market,
λ fit on walk-forward calibration), cap recognised EV at ~8–10%, size with ≤¼ Kelly on
the shrunk probability, cap per-round exposure, and treat any bet the model prices >10%
from a 2-book consensus as a data-error check, not a bet.

F12 — Elo configuration is unvalidated in-sample convention (severity: low)

EloConfig (HFA=55, K=24, margin multiplier capped at 1.5, 0.75 season regression) is
sensible folklore, but there's no record of these being tuned on a holdout, and HFA is
a single constant. NRL home advantage is venue- and travel-specific (Suncorp vs a
Sydney derby vs Warriors crossing the Tasman are different animals). Cheap upgrade:
venue-adjusted HFA and a travel/short-turnaround adjustment — the flags already exist
in the lineup feature table (away_long_travel, home_short_turnaround) and are
currently doing nothing.

Where the edge realistically lives (coach's view)

The market's last mover is team news. Bookmakers reprice within minutes of the
Tuesday 4pm team-list drop and again at final 1–17 confirmation an hour before kickoff.
An automated pipeline can be faster and more systematic than a recreational book's
trading desk on:

Spine quality deltas, properly measured. Losing a first-choice halfback is worth
multiple points; losing a bench utility is worth nothing. The current
lineup_*_delta features average squad strength — build player-level ratings from
the stats you already scrape (minutes-weighted, position-adjusted), then price the
specific 1–17 change. This is the single highest-value modelling investment in the
repo, and it's the only feature family the market can be slow on.
Late outs vs open snapshot (lineup_disruption_flag is scaffolded — populate it
first among the filters).
Goal-kicker availability for line bets. A 70% vs 85% kicker swings expected
margin by ~1.5–2 points on 4–5 attempts — invisible in squad-average strength,
decisive at the 2.5-point line granularity you're paper trading.
Weather at kickoff vs total priced at open. Totals are set days out; a
late-arriving 20mm forecast in Townsville is real information. The
weather_contradicts_total_move filter is the right idea — it needs the per-game
close snapshot (F3) to be measurable.
Origin-window rotation and short turnarounds. Clubs rest stars after team
lists confirm it; the market partially prices announced rest but is slow on
predictable rotation patterns (coach-specific tendencies are learnable from 3
seasons of team lists).
Totals microstructure: referee penalty/set-restart tendencies and bunker-era
scoring drift are slow-moving signals recreational totals lines underweight —
season_to_date_avg_total is a start, referee identity is scrapeable.

What will not move the needle: more re-combinations of Elo + rolling form. Six of
the seven current models are that, and they all converge to the same answer minus vig.

Foundations to move the needle — priority order

P0 — Fix the instruments (this fortnight, before more data collection)
1. Per-game pre-kickoff close snapshots; closing line = last pre-kickoff observation
per book (F3). Every week of Round 14+ paper trading collected under the current
protocol is unusable evidence — this is the most time-sensitive item.
2. Per-bookmaker taken/close prices for CLV; consensus for display only (F4).
3. Retrain lgbm_v1 without closing features; extend the leakage guard to
nrl.src.features.FEATURE_COLS; add a train/serve parity test (F1, F2).
4. Fix feature-join grain to include round; assert row counts after merges (F7).
5. Consolidate promotion gates into one importable module (F9).

P1 — Fix the evaluation (next month)
6. Re-run the backtest pricing actionable models at opening odds; move _market
variants into a clearly separate diagnostics table and out of paper-trade seeding (F5).
7. Bootstrap CIs on all season ROI figures in model-edge-status.md; add a
calibration-curve section (reliability by probability bucket) — calibration is the
stated primary metric and is currently never plotted.
8. Market-shrunk EV + capped stakes + per-round exposure limit (F11).
9. Movement-feature training restricted to rows with true open and true close (F6).

P2 — Build the actual edge (season-long)
10. Player-level rating system from staged player stats → priced 1–17 deltas → a margin
model whose H2H, line, and totals outputs all derive from one predicted margin
distribution (replaces 4 near-duplicate weak learners with one coherent one).
11. Venue/travel-adjusted Elo HFA (F12).
12. Populate the scaffolded filters in priority order: lineup_disruption_flag,
key_spine_out_flag, goal_kicker_missing_flag, weather_contradicts_total_move —
each pre-registered with its evaluation window before first data point.

Bottom line

The discipline in this repo — hypothesis language, paper-first, CLV gates — is better
than most professional syndicates start with. But right now the scoreboard is broken in
three places (leaky flagship model, inoperable live path, mis-specified closing
snapshot), and a broken scoreboard is worse than no scoreboard because it manufactures
conviction. Fix P0 before Round 15's snapshots, treat every ROI number currently in
model-edge-status.md as unaudited, and put the modelling budget into player-level
team-list pricing — the only information channel where this pipeline can plausibly be
faster than the market it's trying to beat.