EDGE Bet Model dashboard

Scheduled pipeline

Who runs what, on what schedule, and how to run it by hand. Decision taken 2026-10-28: the cluster executes, this repo controls it.

Why the cluster, not the VM

  • The VM can't be a production scheduler. It's VirtualBox on a desktop that sleeps. When the host sleeps the VM pauses, its clock stops, and cron doesn't fire. In 2026 that caused a 5-week outage (19 Aug–25 Sep) and clock skew of 2.5–7h after each pause, which makes DO Spaces reject every signed request (RequestTimeTooSkewed).
  • The VM half was never scheduled anyway. Reconcile, dbt/DQ, predict and publish were always run by hand, so when the sessions stopped, nothing ran and nothing alerted.
  • Why the cluster had no DuckDB. In July 2026, jobs sharing one ReadWriteOnce volume with a DuckDB file collided. The fix made every cluster job stateless, and the go-cicd-ops contract test keeps scrapers PVC-less.

How it works

scrapers-pipeline CronJob (go-cicd-ops), hourly at :15, concurrencyPolicy: Forbid:

run_cluster_task.sh pipeline
  -> scripts/run_pipeline.py
       clock check (Date header of the bucket endpoint, ±120s)
       acquire control/pipeline/lock.json   (owner = pod name, 60 min TTL)
       restore: published snapshot (fetch_duckdb_snapshot.py, checksum-verified)
                + stage/nrl/ parquet -> NRL_STAGE_LOCAL_DIR
       decide due work
         world_cup  fixture within -48h/+72h, a past fixture not FullTime,
                    or no World Cup draw staged -> world_cup_ingest.py bootstrap
         nrl        NRL fixtures within -3d/+7d and season not complete
                    -> reconcile + dbt build (only if PIPELINE_ENABLE_NRL=true)
       run due steps (any failure: stop, exit non-zero, no publish)
       publish_duckdb_snapshot.py (new immutable snapshot + pointer)
       release lock (always)
  • The published snapshot is the system of record. Each run starts from it, so a failed run never corrupts state, and every hourly snapshot can be rolled back to.
  • Nothing due means nothing published, and the previous snapshot stays live.
  • The image carries dbt (analytics/dbt_nrl/requirements-dbt-core.txt) and the full dbt project, with packages installed at build time (DBT_SKIP_DEPS=true).

Single writer

Writes are protected three ways:
- The CronJob uses Forbid, so it never overlaps itself.
- control/pipeline/lock.json excludes everyone else, including manual VM runs.
- publish_duckdb_snapshot.py refuses to publish while another owner holds an unexpired lock.

An expired lock (from a crashed run) is taken over by the next run.

Manual runs from the VM

Check the clock first. The VM's clock drifts after the host sleeps.

python scripts/pipeline_lock.py clock-check
python scripts/run_pipeline.py --season 2026 --plan          # what would run (local DuckDB)
PIPELINE_LOCK_OWNER=vm-manual python scripts/run_pipeline.py --season 2026   # full restore->publish
python scripts/pipeline_lock.py status

--no-restore runs on the local DuckDB instead of the published snapshot; use it only when the local copy is known to be newer.

Not automated yet (2027 pre-season)

During NRL rounds the pipeline only detects the round until PIPELINE_ENABLE_NRL=true. Still to design before enabling:
- the weekly ingest
- publishing round control
- predict, recommend and paper-trade seeding
- in-season reconcile semantics: today, any failed collector run fails the whole reconcile

The cluster image also needs ml/ and nrl/ for predictions.

Watching it

  • /readyz on the dashboard shows the live snapshot_run_id and when it was generated.
  • PIPELINE_SUMMARY {...} is the last log line of every run.
  • An off-VM watchdog (GitHub Actions) alerts when the snapshot goes stale during an active competition.

Deploys (lessons from 29–31 Oct 2026)

  • Merge one PR at a time. Each push to main starts a Tekton build. Five merges within two minutes produced five concurrent builds: one deployed, three failed pulling the Docker Hub base image (PROTOCOL_ERROR), and one lost the race to update go-cicd-ops. Wait for the "Deploy scrapers " commit in go-cicd-ops before merging the next PR.
  • Redelivering a GitHub webhook doesn't start a build. The GitHub source accepts it (202), but no PipelineRun is created. To redeploy main, merge a new commit.
  • Check a deploy with:
  • kubectl get pipelineruns -n doks-ci-cd | grep scrapers
  • the go-cicd-ops "Deploy scrapers" commits
  • /ops on the dashboard (it shows the running commit)