ForecastProof

An evidence-first agent that validates forecasting papers, challenges them statistically, and runs one governed next experiment.

Comment

Inspiration

Financial forecasting papers often look precise while leaving the most important questions unanswered: Was the data frozen? Did the split leak future information? Did the model beat a simple baseline? Is a higher point estimate statistically convincing? And did the proposed improvement really run, or was it only suggested by an LLM?

ForecastProof turns those questions into an executable, auditable workflow. Instead of asking an AI to declare whether a paper “works,” deterministic evidence gates decide what is true and GPT-5.6 handles the part where language reasoning is useful: synthesizing verified evidence into a clear research decision.

What it does

ForecastProof takes an evidence-backed forecasting paper through one end-to-end flow:

  1. Analyze the paper. Inspect pinned evidence, unknowns, the proposed model family, and every adaptation required to map the paper onto a shared task.
  2. Verify on a common task. Run Random Forest, GBDT, or LSTM on the same frozen SPY next-day-direction dataset, with identical 12-lag features, 41 purged walk-forward folds, 656 aligned out-of-fold targets, and a fold-train-only majority baseline.
  3. Challenge the result. Check four adaptation-integrity gates, then compare accuracy against the baseline with a Wilson interval and an exact binomial test. A better point estimate alone is not enough.
  4. Generate an audited decision memo. GPT-5.6 reads evidence through two read-only tools and returns a strict structured memo. Eight deterministic checks verify grounding, citations, completeness, safety, adaptation honesty, and statistical honesty.
  5. Run one governed iteration. Evidence-bound proposals may change only allow-listed parameters. After human approval, ForecastProof trains a real child model on reserved folds and promotes it only if deterministic primary, secondary, slice, runtime, finiteness, and integrity gates all pass.

The demo contains three S&P-related papers and three genuinely executed model families. All three pass adaptation-integrity checks. None passes the statistical-skill gate, so deployment stays on HOLD. That honest negative result is the point: ForecastProof refuses to turn uncertainty into a confident-looking recommendation.

How we built it

ForecastProof is a Python and Streamlit application backed by pandas, NumPy, scikit-learn, and PyTorch. A frozen research-suite artifact keeps all paper cases on one comparable protocol. Reproduction tiers distinguish strict reproduction from paper-inspired adaptation, and every paper-to-run delta remains visible.

Live synthesis uses GPT-5.6 Luna through the OpenAI Responses API. The model must call get_evidence_brief and get_verification_result, both read-only, before returning a strict DecisionMemo Structured Output. Deterministic code—not the model—owns every pass/fail gate, promotion decision, and deployment hold.

The default experience is a verified replay, so judges can run the complete golden path without an API key or model cost. Live mode is opt-in, capped, uses low reasoning, sets store=false, and records usage telemetry without storing the key.

How we used Codex

Codex helped audit the existing research harness, isolate hackathon work on codex/openai-build-week, design the reusable S&P 500 common-task contract, implement the Streamlit product flow and Responses tool loop, create tests that really train all three model families, and keep scope disclosure honest. The pre-event baseline is commit ffa048e; Build Week work is visible as later commits on the submission branch.

Challenges

The hardest choice was refusing two tempting shortcuts: calling a shared benchmark a strict paper reproduction, and treating a slightly higher accuracy as proven skill. We also rejected a fourth candidate because its local MethodCard lacked a valid evidence quotation. Fewer trustworthy cases made a stronger product than a larger unsupported claim.

The other challenge was separating the agent's useful language work from decision authority. GPT-5.6 is excellent at connecting evidence, risks, and next actions, but a polished memo should never overwrite a failed statistical gate. The model is a grounded analyst, not its own grader.

Accomplishments

  • Three evidence-backed papers run end to end across Random Forest, GBDT, and LSTM.
  • All cases share exactly 656 aligned out-of-fold targets and pass four comparison-integrity gates.
  • The UI separates “adaptation succeeded” from “forecast skill is established.”
  • The bounded iteration loop really changes parameters, trains, evaluates reserved folds, and retains the parent when guardrails fail.
  • 131 automated tests pass, including real training smoke tests for all three model families.
  • A live GPT-5.6 Luna acceptance run called both tools and passed all eight memo audits at 100/100 while preserving deployment HOLD.

What we learned

An agent becomes more trustworthy when its authority is deliberately narrow. The LLM should explain verified evidence and expose uncertainty; deterministic protocols should own data integrity, statistical conclusions, and promotion.

What's next

  • Add an independent frozen out-of-period SPY regime.
  • Add paired parent-child confidence intervals and McNemar or paired-bootstrap evidence.
  • Add preregistered multi-seed evaluation.
  • Route uploaded papers into strict, adaptation, or blocked paths with a visible compatibility explanation.
  • Add user-provided US-equity data mapping and cost-aware economic validation only after forecast skill passes.

ForecastProof is a research-support system, not investment advice, and never authorizes trading.

Built With

Share this project:

Updates