Litmus

landing
litmus pondering
agent lightbulb

Inspiration

One of us spent last summer doing research and kept hitting the same wall. Research papers are full of claims, and you want to know which ones hold up in your own setup. In ML, checking even one means tuning, running, rerunning on more seeds, and comparing numbers by hand. Do that for every claim you highlight, and you've already lost a week. We wanted the reseach paper highlights we make to turn into a full pipeline for experimentation on their own.

What it does

Papiers makes this task somewhat simpler for us. You use your highlighted claims and notes in Papiers (which stores research papers, such as ones from arXiv). You then tell Litmus what you're researching, and it pulls out the highlights that match. You pick a claim to start from, and Litmus dispatches agents in parallel to complete experiments on a specific dataset (from the papers you read).

Each agent turns its claim into a real training run on a dataset, in our case, the dataset used to train object classification models, CIFAR10. It runs on Modal NVIDIA A100s, uses 5 paired seeds, and is compared fairly against its parent. A statistics module grades the result: confirmed, refuted, or inconclusive, based on claims, with a confidence interval. When something comes back refuted, the agent searches your library for why. Then it asks you what to do next and suggests next experiments.

You steer at every step. The whole thing grows as a map of experiments around Litmus; you can chat with him about any one agent running experiments or the research as a whole, through LangGraph's runtime context. When you're done, Litmus writes an abstract where every sentence cites each experiment and the highlight behind it.

How we built it

Experiments: Keller Jordan's airbench94 on Modal A100s, specifically for our runs on the CIFAR10 dataset, with each training epoch streamed live to the UI. Results are cached to reduce extra work.
Agents: a LangGraph engine with SQLite checkpoints. Each claim gets its own branch: run → grade → reflect → suggest. When an agent needs you, it pauses only its own branch, and a restarted server picks up exactly where it left off.
Grading: a fixed module that agents can't touch. Paired t-intervals on the same seeds, plus 5 extra seeds automatically when a result is borderline.
Reasoning: OpenAI structured outputs for Litmus's chat, each agent's next steps, the suggested questions and the abstract. Any citation the model invents gets stripped.
Library: the Papiers command-line tool, weighting your own notes above plain highlights.
Frontend: React, three.js and FastAPI over WebSockets, plus a demo mode that simulates the whole backend in the browser.

Challenges we ran into

Fair comparisons. Our first agents compared "train longer" against themselves. Now every child is graded against its parent re-run at the same settings.
Seed noise. Lots of results landed right on the line, so we added automatic extra seeds instead of guessing.
Agents running in parallel while waiting on a human. One agent asking a question can't freeze the others. Getting LangGraph interrupts, parallel branches, and answers that arrive mid-wave to play nicely took a while.

Accomplishments that we're proud of

It's real. Every node is a training run that happened, every verdict comes from actual statistics, every line of the abstract points back to evidence, and runs are fully transparent on Modal. We also love that it feels human-in-the-loop: instead of letting agents do everything for you, Litmus asks questions throughout the loop, digs deeper, and lets your notes shape what it tests.

What we learned

Many published claims don't hold up with a 10-epoch budget, and it's useful to know that quickly. We also learned that most of the work in "AI research agents" is unglamorous: fair baselines, seed counts, and not letting the model make up numbers.