Load Bearing

An agent that finds newly published forecasting methods, judges whether they fit a real power-grid problem, and runs them on a frozen benchmark: claimed vs. measured, with sources.

Comment

Inspiration

New forecasting methods are published every week, each claiming to beat the last. Most practitioners never find out whether those claims hold on their own problem, because checking takes days. I work on electricity load forecasting and wanted an agent that does the checking for me: which claims are load bearing, and which collapse on contact with a real grid.

What it does

Load Bearing runs three steps:

  • Scan searches the live web (arXiv, Hugging Face, Zenodo) for forecasting claims from the last 120 days.
  • Judge scores each candidate against a written problem spec, citing a source for every claim it relies on.
  • Run executes accepted methods on a frozen benchmark: PJM hourly load, forecast a week ahead every Monday.

Each method gets a receipt: what the paper claimed, what we measured, and a verdict of "worth switching," "practical tie," or "inconclusive," based on a 90% interval against a bar set in advance.

The first question it answered: a pretrained foundation model (Chronos-2) roughly ties a plain regression on Great Britain demand. Does that transfer to US grid zones?

How I built it

  • Three Claude Code skills (/scan, /judge, /run-candidate) chained by one pipeline command
  • Live web search through You.com, with the provider swappable so results can be compared on identical inputs
  • A Python evaluation harness with frozen, versioned protocols; each protocol is scored once and never revised
  • Foundation models run through TimeCopilot's FoundationForecast; baselines include seasonal naive, lag regression, the GEFCom "Vanilla" benchmark, and LightGBM
  • Load from EIA-930; weather from ERA5, GFS/HRRR, ECMWF IFS and AIFS
  • A static dashboard on GitHub Pages, rebuilt by GitHub Actions

Prior work disclosure: the data pulls, harness, baselines, protocols, and skill scaffolding existed before the event and are tagged pre-hackathon in the repo. Built today: [live You.com runs, fixes for the real service, the two-provider comparison, and the receipts from today's scan — edit to match what you finished].

Challenges

  • Getting an agent to judge relevance from evidence only. Every claim on a scorecard must point to a source, and a validator rejects scorecards that don't.
  • Keeping the benchmark honest. Zones, weather points, and the pass/fail bar were fixed before any model ran on them.
  • One year of weekly forecasts gives wide intervals, so the honest answer is often "inconclusive."
  • Models whose weights were published during the test year can't be called out-of-sample, and are labelled that way.

Accomplishments I'm proud of

  • Chronos-2 beat a temperature-only regression clearly in one zone (27% lower error), but against the field's standard benchmark the lead disappeared.
  • On two new zones chosen by a rule written in advance, neither result was conclusive, so the Great Britain tie is not confirmed for the US.
  • Adding calendar inputs made Chronos-2 worse in all four zones.
  • The agent reports "inconclusive" when the data doesn't support a verdict, rather than picking a winner.
  • Claude and You.com found and verified new methods using /scan and /judge agents - adding new candidates using /run agent

What I learned

The baseline matters more than the model. A headline win over a weak baseline shrank to nothing against a standard one. An agent with live search is only useful here if every judgment is traceable to a source.

What's next

  • A prospective panel: commit forecasts before the load exists, then score at 13, 26, and 52 weeks
  • Scheduled runs that end in a pull request for human review
  • More candidates, including TabPFN-TS
  • Blind human labels to measure how well the judge agrees with people

Built With

Share this project:

Updates

Submission history