Paper Desk

Inspiration

AI can now write a trading strategy and a backtest in minutes, and the result usually looks amazing. Often it is not real. In one study, five LLM trading methods lost 50% to 72% of their return once they were tested after the model's training cutoff. Across 888 algorithms, backtest Sharpe explained almost none of the live result. The code can peek at future prices, the AI can simply remember the past, or the plan is just the lucky one out of many tries.

The tools that catch this exist, but they are built for quant developers. Most people building strategies with ChatGPT or Claude decide by looking at a chart. We wanted to give them a second opinion before they trust a backtest.

What it does

Paper Desk takes a trading plan in plain English, like "Buy Costco when it drops 5%, sell after 10%", and in about 30 seconds returns one verdict: Trust, Be careful, or Do not trust, with plain reasons.

Claude turns the plan into a Python strategy(df) function.
10 Daytona sandboxes run it on 30 stocks in parallel. The network is blocked and no keys are inside, because this is untrusted code.
A referee runs four checks:
- Peeking: we re-run the strategy at 50 cutoff dates with the future hidden. An honest strategy must satisfy $$\text{pos}_t(\text{df}_{\le c}) = \text{pos}_t(\text{df}) \quad \forall\, t \le c$$ If a past decision changes, the code read the future, and we name the exact line.
- Locked period: performance after the model's training cutoff, on data the AI never saw.
- Breadth: does the same rule work on the other 29 stocks?
- Trial count: test enough variants and one looks great by luck, so we count every try.
Arize AX records every step as a trace. The "Why?" answer is generated only from that trace, and then every number in it is checked against the trace.
Fix it: Claude writes 4 honest variants and they all run at once.
Seal: each verdict gets a date, a SHA-256 hash, an Alpaca paper trade and a shareable card. This builds a record of whether our verdicts held up.

Paper money only. Not financial advice.

How we built it

Frontend: Next.js 16, React 19, Tailwind and shadcn/ui. A live run view streams real sandbox output into terminal tiles as it happens.
Agent: Claude through the Anthropic SDK with tool use. It writes the strategy, the honest variants and the grounded explanation.
Sandboxes: the Daytona SDK with a custom snapshot (Python + pandas). Each run creates 10 ephemeral sandboxes with networkBlockAll. Prices go in as files. A Python runner backtests, runs the peeking test, and streams progress lines back through Daytona session logs.
Observability: Arize AX over OTLP. Each run is one trace with LLM, tool and evaluator spans, plus custom evals for the verdict and for "why is grounded in the trace".
Seal: SHA-256 over the plan, code and verdict, with an Alpaca paper order whose client_order_id is the hash.
Data: 10 years of daily, split- and dividend-adjusted prices for 30 large US stocks. 2016 to 2025 is the build period, and 2026 onward is locked.

Challenges we ran into

Daytona's 10-CPU limit. One run uses all 10 CPUs. Running 4 Fix it variants at once, or starting a run right after another, hit the limit. We split the variants into 2 sandboxes each, and sandbox creation now waits and retries for a free CPU.
Bundling the Daytona SDK. Next.js bundling broke the SDK's file upload (a dynamic require), so we load it as an external server package.
Streaming live output. We moved to async session commands so each sandbox's log lines stream into the dashboard as they happen.
Model and API details. The newer Claude model rejects forced tool_choice, so we switched to auto plus a system prompt. Our org-level API key also needed a workspace header.
Arize setup. Our first traces were rejected because the base64 Space ID had lost its == padding. We found it by listing spaces through the REST API.
Keeping the "why" honest. An LLM explanation can invent numbers, so we check every number in the answer against the trace and flag any that don't match.

Accomplishments that we're proud of

It runs end to end with real services: Claude, Daytona, Arize AX and Alpaca paper.
A planted look-ahead bug is caught on 30 of 30 stocks and pinned to the exact line, while honest variants of the same plan pass.
Untrusted, AI-written code runs safely: no network and no secrets in the sandbox.
The whole loop, from plain-English plan to a verdict with reasons, takes about 30 seconds.

What we learned

A great backtest is weak evidence. After about 7 variants, you expect a 2-year backtest with Sharpe above 1 even when the true Sharpe is 0 (López de Prado). Counting trials matters as much as any single result.
Look-ahead bugs are easy to write and hard to see. Re-running with the future hidden is a simple, convincing test.
Sandboxes are the right boundary for running other people's code, and parallel sandboxes make a 30-stock test feel instant.
Tracing every step makes the explanation auditable instead of just plausible.

What's next for Paper Desk

Grade the sealed record: score whether each sealed verdict held up over time on paper trades, and publish the track record.
A gate for agents: an API and MCP tool that a trading agent calls before it goes live.
Stronger checks: walk-forward tests, a stock list that includes delisted companies (to remove survivorship bias), and thresholds tuned on real results.
Platforms: screening for courses, marketplaces and prop firms that host strategies from others.