Valo - AI Scientist

An autonomous AI scientist that designs and runs real physics experiments, while trusted code, not the model, decides what counts as true.

Comment

Inspiration

Today's "AI scientists" mostly write papers about experiments whose results they report themselves. We wanted the opposite: an agent that does real computational research, choosing hypotheses, experiments and fidelities, and changing direction, without ever being able to invent a measurement, rewrite a test, or declare its own simulator valid. In VALO, Nemotron is the scientist; the harness is the laboratory.

What it does

You give VALO a research question in plain English, e.g. "Maximize efficiency of a III-V single-junction solar cell while reducing its surface area."

  1. Intake. The lead scientist (Nemotron 3 Ultra) reads the catalogue of simulated worlds, lists the ways the question can be read, and asks the researcher structured questions. It drafts a research specification. Trusted code checks every "can / cannot measure" claim against the catalogue. Only the human can approve the spec.
  2. World build and qualification. It builds a simulated world (Solcore/Sesame drift-diffusion for solar cells, or NVIDIA Warp for heat spreading). A qualification lab runs 16 fixed checks and issues a passport: trust state, the calibrated F1↔F2 error band, and the region where each fidelity is valid. A world that fails is quarantined and repaired.
  3. Research. The scientist opens tracks, writes hypothesis cards, and locks predictions (hashed) before any run. It designs experiments and picks a fidelity (F0 analytic, F1 depletion approximation, F2 drift-diffusion). The fidelity controller advises the cheapest admissible fidelity. 13 code gates refuse anything illegal: unlocked predictions, inadmissible fidelity, out-of-bounds designs, probing the sealed held-out conditions, over budget, out of scope.
  4. Judging. A deterministic judge compares locked predictions with measured runs: supported, refuted, unresolved or invalid. The model never writes a verdict.
  5. Wrap-up. Verification at the reference fidelity, a sealed held-out test revealed once, a referee/adjudicator council, and a report in which every number must carry a run or verdict id or it is dropped by the claim linter.
  6. Cockpit. A live web UI shows:
    • an intake stepper;
    • the world passport;
    • an Research Tree;
    • a best-so-far chart;
    • evidence and report;
    • full observability of every model call and its cost.

A real run (camp_6b88f4b0f7b8, on NIM, $0.26 total):

  • 134 scientist turns and 386 real Solcore simulations (236 at F1, 150 at F2), with 31 refusals by gates.
  • The scientist's main hypothesis, that the F1-optimal design transfers to F2, was refuted by the code judge: predicted +1.2%, measured −2.3% relative efficiency at F2.
  • An F2 optimisation found 27.00% vs a 26.55% baseline. This is an exploratory lead, not a confirmed claim, and the report says so.
  • The qualification lab quarantined a GaInP world variant for an F1/F2 ranking inconsistency.

How we built it

  • Stack: Python 3.12, FastAPI with Server-Sent Events, SQLite (WAL) evidence ledger, Pydantic v2 and SQLAlchemy 2. The cockpit is vanilla JS with d3-org-chart.
  • Models: NVIDIA Nemotron through Nebius Token Factory (the hosted configuration) and NVIDIA NIM (development), behind one OpenAI-compatible client. Roles are routed by tier:
    • Ultra 550B for the lead scientist;
    • Super-VL 120B for the specialists (hypothesis forger, methods, analyst, world builder, referee, writer);
    • Lightning 30B for fast parsing and memory.

We use strict json_schema outputs with the thinking toggle per role. There is an automatic cross-provider fallback (NIM → Token Factory) on rate limits. Every call is ledgered with tokens, cost and request id.

  • Physics: Solcore 5.10, Sesame drift-diffusion, tmm optics; NVIDIA Warp for the thermal world. And other world simulators.
  • Science core: prediction locking, the judge, uncertainty widening from the passport, a fidelity controller, validity domains that work for any world, sealed held-out tests, quarantine and repair with stale-result propagation, and wrong-world detection.
  • Memory: a port of Hindsight (FTS5 + dense embeddings + link expansion, fused by RRF). Code writes a belief timeline whenever a verdict or world change happens. Memory is never evidence.
  • Literature: Tavily and arXiv, reranked.
  • Built by: one person orchestrating a team of AI coding agents. Over 100 small reviewed commits and 1,069 automated tests.

Challenges we ran into

  • Keeping the model honest without scripting it. We wanted the scientist free to choose, but never free to grade itself. The answer was hard gates in code plus a refusal that always says how to unblock, instead of a fixed pipeline.
  • Fidelity gaps are real. F1 (depletion approximation) ranks designs differently from F2 (drift-diffusion). The agent's own hypothesis that "F1 optimum transfers" was refuted. The qualification lab must catch this, not the model.

Accomplishments that we're proud of

  • Real end-to-end autonomous campaign on real physics, where the model never touched a number it reported.

What we learned

  • The useful split is the model proposes, code disposes. Nemotron is very good at choosing what to try next. Trusted code must own what is true.
  • Cheap fidelity is for refuting, not for claiming; a passport makes that enforceable.
  • Long-running agents need state they can't lose: a ledger, a belief timeline and a digest, not a longer prompt.

What's next for Valo - AI Scientist

  • More world kinds (tandem cells, PhysicsNeMo surrogates as a qualified fidelity).

Built With

Share this project:

Updates

Submission history