BOTLAB Bot Log Auditor

An AI agent that audits your trading bot's real trade history with 6 statistical gates and tells you, in plain English, whether its profits are an edge or just luck.

Comment

Inspiration

Inspiration We run BOTLAB, an independent lab that paper-trades crypto bots with real fees and publishes every result — including the ones that fail. Out of 100+ strategies we tracked, almost all failed at least one statistical test once fees, sample size and luck were accounted for. Meanwhile, bot sellers and exchange dashboards show only PnL and win rate. A 40-trade streak with an 80% win rate looks amazing and proves almost nothing. We wanted anyone to be able to check their own bot the way we check ours.

What it does You upload a raw trade-history CSV from Bybit, Binance or a generic export. The agent detects the format, maps the columns, sorts trades in time order and converts PnL into R-multiples (risk units). It runs 6 statistical gates: G1 Sufficient sample — at least max(50 trades, MinTRL), the minimum track record length for 95% confidence G2 Not just luck — Probabilistic and Deflated Sharpe Ratio (adjusts for how many variants you tried) G3 Still working recently — last 100 / last 30 trades G4 Not reliant on a few coins or days — top-3 coins and best-2 days must each be under 50% of profit G5 No losing side — longs and shorts checked separately G6 Acceptable drawdown An AI model writes a plain-English audit: verdict, which gates failed and why, what to test next, and risk warnings — never buy/sell advice, never profit promises. You can ask follow-up questions. Pro audit (month-by-month table with green/red days, per-coin breakdown, robustness checklist, printable report) is unlocked through PayPal Checkout (Sandbox): order created → buyer approves → server captures → Pro unlocked.

What it does

How we built it Statistics core: the open-source gates.js from our earlier project botlab-check-your-bot (MIT), ported to Python and kept in exact parity (tests compare both on hundreds of random logs). Agent pipeline (Python): parse → normalise → gates → sandboxed cross-check (the original gates.js re-run in an isolated subprocess: no network, temp dir, timeout) → model-written analysis code executed in a restricted sandbox → AI report → follow-up chat. Hosted demo: a stateless Cloudflare Worker with Workers AI for the report, nothing uploaded is stored. Payments: PayPal Orders v2 REST API in Sandbox, server-side capture. UI: single responsive page, light/dark, live progress of every agent step. 49 automated tests (parsing variants, gate parity, sandbox limits, payments with mocked HTTP). Challenges we ran into Exchange exports are messy: different column names, newest-first ordering, fees in separate columns, partial fills. We built heuristics first and only ask the AI for a column mapping when heuristics fail. Making the AI honest: the model must explain statistics without sounding like financial advice or hype. We constrained the prompt and always show the raw gate numbers next to the text. Running model-written code safely: the sandbox blocks sockets, subprocesses and file access outside a temp folder, and kills code that runs past its timeout. The public hosted demo only runs the safe deterministic subset. Accomplishments that we're proud of The same gates we use on our own 100+ bots now work on anyone's trade log in seconds. Exact parity between the JavaScript and Python implementations, verified by tests. A payment flow that unlocks real extra value (Pro analysis), not a paywall on basic honesty. What we learned Most "profitable" bots fail on sample size, concentration (a few coins or days carry all the profit) or a losing side hidden inside a winning total. Showing why matters more than a single score.

What's next Direct exchange import with read-only API keys (no CSV needed). Saved audit history to track a bot over time. Agent-to-agent mode: let trading agents request and pay for an audit automatically. Educational tool. Not financial advice. Past results don't predict future results.

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for BOTLAB Bot Log Auditor

Built With

Share this project:

Updates

Submission history