Inspiration
Link to video in case it doesn't load: https://vimeo.com/1246920796?fl=tl&fe=ec Quant papers on arXiv constantly claim high Sharpe ratios and clean returns, but almost nobody actually re-tests them with realistic trading costs on fresh data. At a real quant fund, a team codes each strategy up and stress-tests it before trusting a dollar to it. Outside those firms, setting up the market data pipelines and verification harness takes too much time, so most claims just go unchecked.
This project is step one of a larger research flywheel:
- The Harness (Built here): A deterministic evaluation arena that stress-tests strategies against reality.
- Agentic Discovery: Agents figuring out what hypotheses and market anomalies to test next.
- The Curated Graveyard: Compiling every failure, fee-drag threshold, and regime boundary into a structured knowledge base so future agents can read what failed and synthesize better strategies without repeating dead ends.
What it does
You upload a finance research paper. An AI pipeline extracts the core strategy and implements three variations of it:
- A strict literal version as written in the paper.
- A conservative, volatility-scaled version.
- A regime-filtered version that only trades under specific market conditions.
From there, deterministic Python code—never an LLM—acts as the judge and stress-tests each implementation across five institutional checks:
- Lookahead bias: Does it secretly leak future data?
- Execution & friction: Does it still make money after realistic slippage and trading costs?
- Parameter sensitivity: Does performance fall off a cliff if you nudge the settings by 10–20%?
- Out-of-sample stability: Does it hold up on unseen recent data?
- Multiple-testing luck: Could the Sharpe just be statistical noise from trying multiple variants?
If the generated code has a syntax or lookahead bug, it gets sent back to the agent to fix (up to three tries). If the strategy runs cleanly and simply loses money, it doesn't get a second try—otherwise the agent is just curve-fitting to the test. Every run produces a pass/fail verdict, a full tearsheet, and a plain-English post-mortem of what the paper missed.
Results so far: 5 papers, 18 strategy implementations, 0 verified. The best one hit a 0.74 Sharpe before costs (against a 1.33 claim in the paper) and lost money once trading costs were applied.
How I built it
I treated the build like an automated assembly line: I wrote a master spec and broken-down work orders with strict acceptance tests, then used coding agents over 20 hours to build and verify each piece.
- Market Data: 3 years of 1-minute bars across 50 tickers and 10 years of daily S&P 500 bars stored in Parquet, backed by a strict data contract that auto-generates schema validators and prompt context so agents can't hallucinate column meanings.
- Backtester & Evals: A custom vectorized backtesting engine and five deterministic verification gates in Python, with all pass/fail thresholds locked in a read-only config file that agents aren't allowed to touch.
- Agent Pipeline: An extractor, three parallel strategy workers, a bug-only self-repair loop, and a post-mortem critic, all tracked via typed records and a replayable event log.
- Web App: FastAPI backend streaming live pipeline events to a React + TypeScript frontend.
Challenges I ran into
- Timestamp & timezone hallucinations: On the first end-to-end run, all three generated strategies placed zero trades because the agents guessed wrong about what a bar's timestamp meant. I fixed this by enforcing a strict data contract and pre-computing session-time columns so agents never have to do timezone math.
- Drawing the line on self-repair: Letting an agent retry until a strategy passes the backtest would defeat the entire point—it would just overfit to the evals. I had to separate code/leakage bugs (allowed to self-heal) from bad trading logic (immediate fail).
- Diagnostic visibility: Early on, a strategy with zero trades and a strategy with a real signal that got wiped out by trading fees looked identical in the final verdict. I had to separate pre-cost signal diagnostics from post-cost execution results.
Accomplishments that I'm proud of
- Zero LLM grading: Keeping a hard wall between AI writing the strategy code and deterministic Python code stress-testing it, from the first commit to the final build.
- Speed and cost: Taking a raw PDF to three implemented, stress-tested strategies with full tearsheets in ~3 minutes for under a dollar.
- 375 passing backend tests and 17 frontend tests built solo in 20 hours.
- An honest null result: 0 out of 18 implementations surviving the arena is actually the most realistic finding possible—and the UI highlights why they failed instead of hiding it.
What I learned
- A tight spec with deterministic acceptance tests matters way more than which model writes the code.
- If a data schema leaves any room for interpretation, coding agents will fill the gap with a confident guess.
- Most academic trading edges are razor-thin before fees and completely gone after realistic slippage.
What's next for Financial Models In the Arena
- The Curated Graveyard & Wiki: Automatically compiling post-mortems across hundreds of papers into a structured, non-contradictory knowledge base that future agents read before proposing new strategies.
- Agentic Hypothesis Generation: Letting discovery agents query the graveyard to spin up and test novel hybrid strategies that avoid known failure modes.
- Alternative data with strict point-in-time rules: Starting with congressional stock trades keyed strictly to the public disclosure date rather than the trade date.
- Live forward-testing: Scoring strategies strictly on new market days recorded after the strategy was locked in.
- Integration with prediction markets data: Prediction markets data will provide an extremely dimension of data wrt how the world thinks on cultural events which could help in finding correlations (and possible causations) in other areas.
