Inspiration
Trading backtests routinely look brilliant because of bugs, not skill: a signal that uses today's close to trade today, a shift(-1) that pulls in tomorrow's return, a z-score over the whole history, or the luckiest of hundreds of settings. These bugs are easy to write and nearly impossible to spot by reading code. Retail traders, finance students and quant interview candidates trust numbers like these every day.
What it does
Backtest Auditor takes a trading strategy (Python) and:
- Scans it with deterministic code rules for known leak patterns.
- Attacks it with four deterministic tests (delay, hide-the-future, unseen data, deflated Sharpe), running the code only inside a Nebius Token Factory Sandbox.
- Explains the evidence with NVIDIA Nemotron: the responsible line, why it leaks, and a suggested change.
- Fixes it: Nemotron Super writes two candidate fixes, each re-tested in parallel in its own sandbox branch. A fix is accepted only if the tests pass and the strategy still trades.
The core rule: the tests decide, the AI explains. Nemotron's verdicts are overwritten with the test results, and every number it writes is checked against the evidence.
On the built-in gallery, a strategy claiming 50.6%/yr (Sharpe 3.43) is caught and fixed to an honest 1.0%/yr; a shift(-1) slip claiming 63.9%/yr is fixed to 4.9%/yr; a weekly back-fill bug that fools the delay test and every pattern rule is caught by hide-the-future. Two honest strategies pass, with no false alarms.
How we built it
- NVIDIA Nemotron Super (
nemotron-3-super-120b-a12b) on Token Factory plans the audit (flags lines, predicts each test's verdict, reviews rule hits), writes the report, and writes fixes. - NVIDIA Nemotron Nano (
NVIDIA-Nemotron-3-Nano-30B-A3B) writes cheap one-sentence explanations of rule hits, capped at 1,500 tokens with a fallback. - Nebius Token Factory Sandboxes: a tagged base image (≈ 3 s per audit), hardened I/O (per-run output markers, timeouts), and Git-like branching so each candidate fix runs in its own branch of one snapshot, in parallel.
- All LLM calls use Token Factory's OpenAI-compatible API with strict JSON-schema output; guardrails verify cited lines, retry incomplete responses, and drop no-op fixes.
- Streamlit front end with a landing page, an 8-strategy gallery, and a live "audit your own code" page. A measured spend budget makes public live runs safe.
Challenges we ran into
- An LLM scanner wasn't reliable. As a pattern checklist, Nemotron found the buggy line only ~1 in 3 runs, so we moved syntactic detection to code-parsing rules and gave Nano the explanation job.
- Schema-valid isn't complete. Super sometimes returned well-formed but empty plans; we added completeness checks and retries (8/8 afterwards on our measured samples).
- Small leaks are hard to see. A full-history z-score only flips some trades; checking 40 dates instead of 12 caught it on 4/4 random seeds with zero false alarms.
- Our acceptance rule was too strict. Requiring the delay test to PASS rejected a correct fix (it's noisy on random data); hide-the-future is now the decisive check.
Accomplishments that we're proud of
An agent that writes code, runs it in a sandbox, and accepts its own work only when deterministic tests agree. Every number on screen is reproducible.
What we learned
Let models do the reading, reasoning and writing, and let code do the judging. And measure everything: several "obvious" design choices failed when we tested them.
What's next for Backtest Auditor
Real-market data with licensed sources, more bias checks (survivorship, transaction costs), and walk-forward re-optimization as an honest alternative for overfit strategies.
