Research War Room

An autonomous agent tunes a GPU kernel under two human lenses, Speed and Precision. It states predictions first, code judges the results, and it cites your Papiers highlights.

Comment

Inspiration

Autoresearch agents are good at running experiments and bad at being honest about them. They grade their own homework, quietly change the goal, and squash every tradeoff into one number. Real research teams don't work like that: one person cares about speed, another about numerical precision, and the important decisions happen where they disagree. For the Papiers Human-Centred Autoresearch Challenge I wanted an agent that works with that disagreement instead of hiding it. It says what it expects before every run, accepts a verdict it doesn't control, rejects its own failed ideas with a post-mortem, and asks the right human when it is unsure.

What it does

Research War Room is a web UI where an autonomous agent tunes a fused matmul + bias + GELU Triton kernel under two competing lenses, Speed and Precision. In this prototype one person plays both lenses by switching tabs. In a real team they would be two people. When the two lenses each find a gain, the agent can propose a hybrid that combines them for a human to approve. In the demo recording it declined, with a logged reason, because that combination had already been run. It also refuses to repeat a change that already lost and refuses a claim that does not match the exact parameters it changes. The session ends with a final report: the best configuration, what was ruled out, what to try next, what needs a human decision, and the measured outcome of each typed hunch.

Each iteration, the agent:

  1. Reads relevant notes and conversations from the Papiers library and cites what it used.
  2. Proposes a hypothesis with a falsifiable prediction stated before the run, for example "this config is at least 1.2x faster than the best so far".
  3. Runs the experiment in an isolated subprocess with a hard 15 s timeout.
  4. Gets a verdict computed in code, never by the LLM: confirmed, rejected, or inconclusive (inside the measured noise band).

Correctness is measured against an FP32 reference:

err = max|y - y_ref| / max(max|y_ref|, 1)

The error bound belongs to the human lens, not the agent. By default the strict bound is 1e-3. A Speed directive loosens it to 2e-2 on Speed branches. A Precision directive tightens every branch back to 1e-3 on both the standard batch (M = N = K = 2048) and two adversarial batches (long K = 16384, and 8x larger pre-activations).

The key moment: FP16 accumulation is about 1.5x faster than its FP32 parent. Its standard error of 1.3e-2 passes the Speed tolerance, so it is provisionally confirmed. When the Precision owner sets a directive, the agent re-judges the same logged run without re-running it. The error is 13x over the bound on the standard batch and 71x over on the adversarial batch, so the agent flags the conflict and rejects the branch. It then writes a post-mortem with an error-versus-K sweep (from 1.9e-2 at K=256 to 8.1e-2 at K=4096) and forks a pivot branch that rules FP16 accumulation out.

When a result is too close to call, the agent sends a help request to the right lens owner, with the evidence, a default and a deadline. Other branches keep running while it waits, and if nobody answers it goes with the default and says so. Typed hunches from a human become hypotheses the agent tests and judges like any other: in the demo, a tanh GELU hunch ran about 1.07x faster with error under the strict bound, and the code called it inconclusive because the gain sat inside the noise. Findings, negative results and the full ledger are exported as Markdown for Papiers.

How we built it

  • Harness (Python, PyTorch, Triton): A Triton kernel template whose only editable surface is a small Params struct: tile sizes, warps, stages, accumulation dtype and GELU formulation. The starting kernel is deliberately badly tuned (16x16x16 tiles, 1 warp, 1 stage) but correct, and must pass the 1e-3 gate before the agent starts. Every experiment runs in a fresh child process that is killed on timeout, and its evidence is saved per run.
  • Agent: An orchestrator that runs branches on the single GPU. It handles hypothesis parsing and validation (free-form code and implausible predictions are refused), lens bounds, verdicts, post-mortems, pivots, help requests and exports. Gemini models propose hypotheses and write summaries, with NVIDIA NIM as a logged per-call fallback and an on-disk response cache.
  • Server (FastAPI + WebSockets): It streams every session event and accepts validated commands (directives, help answers, start, stop, retry). Every session is logged to JSONL as the source of truth. A /preflight endpoint checks the GPU, the model APIs and Papiers for real.
  • UI (React + Vite + TypeScript): A branch graph, a metric chart with predicted-vs-measured markers and error bars, summary cards, help cards, post-mortems, a Papiers panel, the ledger, story chapters and a final result screen. It runs live or as a recorded replay.
  • Tests: 50 pytest tests (including a golden session fixture that pins event shapes), GPU tests that reproduce the precision flip from logged metrics, and 84 vitest tests.

Challenges we ran into

  • Timing noise. The same FP16-vs-FP32 comparison measured anywhere from 1.28x to 2.36x when baselines were timed in separate processes. Now every candidate is timed in the same process as its references, interleaved over 7 rounds with the order rotated, and the speedup is the median of per-round ratios. Across 10 separate processes the result now varies by under 1%. Differences under about 5% are still reported as inconclusive, and five of the eight runs in the demo landed there.
  • Model latency. The models I first planned to use took minutes per call. I benchmarked models on JSON validity, whether edits compiled and whether edits passed the bound, then moved to Gemini with NIM as the fallback. Rate limits and fallbacks still make some calls slow, so a full recorded session takes roughly three minutes.
  • Making the failure reproducible. After a strict rejection, the agent rarely proposed FP16 on a Speed branch on its own. The harness can seed it as a variant, which is clearly labelled in the hypothesis note. The agent still tests it, the code still judges it, and the agent writes the post-mortem.
  • Blackwell + WSL2. Triton runs on the RTX 5060 laptop GPU, but its shared memory limit (101376 bytes) and 255 registers per thread break large tile and stage combinations. The agent now checks shared memory and accumulator size before launch and cuts stages or refuses the candidate, and post-mortems use register and spill counts as evidence.
  • Papiers is read-only from the CLI, so findings are exported as Markdown instead of written back. The UI says "exported", never "written".

Accomplishments that we're proud of

  • The agent catches its own mistake: a result that looked like a win under one lens is rejected under the other, from logged evidence, with no re-run and a real post-mortem.
  • Verdicts the LLM can't fake. Predictions are stated before the run, the bound comes from the human, and the verdict is computed in code.
  • Trustworthy measurements. Interleaved same-process timing took speedup variation from "anywhere between 1.3x and 2.4x" to under 1%.
  • Every number in the recorded demo comes from a real logged run.
  • A full working stack (GPU harness, agent, server, UI, 130+ tests) built solo in one weekend.

What we learned

  • Separate judgment from generation. Once the verdict was code and the bound belonged to a human, the agent's honesty no longer depended on the model.
  • Measure your noise before trusting your speedups. Most of my early "wins" were measurement artifacts.
  • The honest headline is often smaller than the exciting one. The best FP32 configuration is about 2.6x faster than the naive start (2.5x to 2.8x across recordings) and slightly under PyTorch eager (0.93x to 1.05x depending on the timing window, 0.97x in the demo). The eager line drawn on the chart comes from the baseline window, while each run's eager ratio is timed in the same window as that run. PyTorch eager itself misses the 1e-3 gate in my harness (1.5e-3 on the standard batch), so the kernels that meet the strict bound are the ones the agent tuned. The value is in the process: predict, measure, reject, ask.
  • Every question to a human needs a default and a deadline, or the agent stalls waiting for people.

What's next for Research War Room

  • Two real people, two real lenses: multi-user sessions so the Speed and Precision owners can be on different machines.
  • Hybrid merges across more branches, with a human approving each merge.
  • Writing back to Papiers once a write API exists, so findings and negative results land next to the notes that inspired them.
  • More domains beyond one kernel: quantization, pruning, and other research problems where speed, accuracy and cost pull in different directions.
  • A comparative final report that sets each lens's best result side by side with the evidence and the human decisions that shaped it.

Built With

Share this project:

Updates

Submission history