-
-
HomePage
Inspiration
Ask an AI assistant "Does retrieving more documents improve RAG accuracy?" and you get a confident, plausible answer, drawn from what it has already read. But for questions about how AI systems behave, the honest answer is almost always "it depends — measure it."
I wanted an AI that behaves less like an oracle and more like a scientist: one that turns a question into a falsifiable hypothesis, runs a controlled experiment, and lets the measured evidence decide. Then, instead of stopping at an answer, it proposes the next experiment.
AI that doesn't just answer questions. It runs experiments to find out.
What it does
ExperimentOS is an autonomous experimentation agent for AI/ML questions. Given a research question, it:
- Searches existing evidence with Tavily and identifies the research gap.
- Forms a falsifiable hypothesis about a measurable effect.
- Designs a controlled experiment: variables, baseline, configurations, metrics, success criteria.
- Writes the experiment code, safety-checks it, and runs it in a sandbox.
- Measures real results and computes the statistics.
- Decides whether the hypothesis is supported, partially supported, rejected or inconclusive, citing only measured numbers.
- Proposes the next experiment, which runs with one click, building a research trail.
Every step, tool call and Nemotron call (model, latency, tokens) streams live to the UI.
$$ \text{Question} \rightarrow \text{Hypothesis} \rightarrow \text{Experiment} \rightarrow \text{Code} \rightarrow \text{Evidence} \rightarrow \text{Conclusion} \rightarrow \text{Next experiment} $$
How we built it
- Reasoning: NVIDIA Nemotron 3 Super 120B on Nebius Token Factory writes the hypothesis, designs the experiment, generates the code, interprets results and proposes the next experiment. Nemotron 3 Nano 30B handles fast question parsing on the same OpenAI-compatible endpoint.
- Research discovery: a live Tavily search; Nemotron summarises the sources into known findings and a research gap that the experiment targets.
- Agent: I built a single FastAPI orchestrator with an explicit state machine (no multi-agent swarm), streaming events to a Next.js frontend over server-sent events.
- Trust layer: every Nemotron output is validated with Pydantic, a falsifiability gate and nine pre-execution checks.
- Execution: generated Python is AST-checked (import allow-list, no file, network or process access) and runs in a sandboxed subprocess with CPU, memory and file limits.
- Benchmark: a real, deterministic RAG pipeline (BM25, embedding and hybrid retrieval, chunking, reranking, an extractive reader) over 40 questions about fictional entities, so no model can answer from memory.
- Statistics in Python, not the LLM. For $n$ questions with accuracy $\hat p$ I report Wilson 95% intervals:
$$ \frac{\hat p + \frac{z^2}{2n} \pm z \sqrt{\frac{\hat p (1 - \hat p)}{n} + \frac{z^2}{4 n^2}}}{1 + \frac{z^2}{n}}, \qquad z = 1.96 $$
Configurations are compared on the same questions with an exact McNemar test, where $b$ is questions gained and $c$ questions lost:
$$ p = \min\left(1, \quad 2 \sum_{i=0}^{\min(b, c)} \binom{b + c}{i} 2^{-(b + c)}\right) $$
- Deployment: Render (API in Docker, web app on Node), with a daily token budget and rate limits for the public demo.
Challenges we ran into
- Running AI-generated code safely. An import allow-list alone was not enough:
json.codecs.opencould reach the filesystem through an allowed module. I added module-traversal checks plus a runtime audit hook, so either layer alone blocks it. - Preventing confident nonsense. Early analyses could cite plausible-looking numbers. Now every number must match a measured value or the analysis is rejected, and a "supported" verdict requires a statistically significant difference.
- Making sure the code ran what was planned. Generated code could in principle run $k = 5$ and label it $k = 10$, so each run's executed configuration is verified against the plan before it counts as evidence.
- Building an honest benchmark: deterministic (seed 42), reproducible, and hard enough that retrieval choices matter. Every entity has two look-alike decoys with conflicting facts.
- Working within a trial budget. I added a daily token budget, a per-experiment call cap and response caching, and cut analysis prompts by about 68%. If the budget runs out mid-experiment, the run finishes on a rule-based path, so measurements are never lost.
- Deployment surprise: my chosen Render hostname already belonged to an unrelated app also named "ExperimentOS API", so my frontend was quietly calling someone else's server until I caught it.
Accomplishments that we're proud of
- The loop really closes, and found something real. Retrieving more documents raised retrieval recall from 85% to 100%, yet accuracy did not improve (70.0% to 67.5%); the agent correctly called this inconclusive and traced it to distractor documents. Its own follow-up, retrieving 20 and reranking to the top 5, raised accuracy from 72.5% to 92.5%:
$$ \Delta = 92.5 - 72.5 = +20.0 \text{ points} $$
$$ \text{relative gain} = \frac{92.5 - 72.5}{72.5} \approx 0.276 \quad (\text{about } +27.6 \text{ percent}) $$
$$ b = 8 \text{ gained}, \quad c = 0 \text{ lost} \quad \Rightarrow \quad p = 2 \cdot 2^{-8} \approx 0.008 $$
- Every conclusion is traceable to specific runs and measurements; nothing is hard-coded.
- Fast and lean: a full loop takes about 42 seconds and 8 Token Factory calls.
- Nemotron output passed strict schema, falsifiability and number-grounding checks on the first try for nearly every step.
What we learned
- Recall is not accuracy. Putting the right answer somewhere in the context is not enough; distractors and position bias can cancel the gain. I would not have seen this without measuring.
- LLMs are excellent experiment designers but should not do the arithmetic. Letting Nemotron reason and interpret while Python computes made the results both fast and verifiable.
- Structured LLM output is dependable when you check it. Strict validation with targeted retry feedback turned occasional slips into automatic corrections.
What's next for ExperimentOS
- Experiment families beyond RAG: prompt variants, temperature, agent planning strategies.
- A live Nemotron reader in place of the simplified extractive reader.
- Larger and real-world evaluation datasets.
- Running heavier experiments as Nebius AI Cloud Serverless Jobs.
Log in or sign up for Devpost to join the conversation.