CounterLab

• 

  Recorded live Nemotron investigation: failed first bet, repaired follow-up and replayable evidence.

• 

  Saved investigation activity keeps model decisions and execution progress visible.

• 

  The experimental plan and predictions are committed before the experiment runs.

• 

  Matched measurements and replay evidence make each reported result inspectable.

• 

  OpenLab accepts a paper and code, then builds a testable investigation.

Inspiration

I built CounterLab for a problem that comes up when checking research: two implementations claim to follow the same method, yet produce different results. An explanation only gets you so far. You need an experiment that can separate the explanations and a record of what happened.

Hugging Face's ICML 2026 reproduction effort reported 242 papers where independent teams reached opposing verdicts on the same claims. That helped shape the project. I wanted the AI to commit to a prediction before seeing the result, and to leave that prediction on the record when it was wrong. Source.

What it does

Give CounterLab a supported paper or code, a repository or paper link, or one of the included examples. Then ask the question you want tested. NVIDIA Nemotron 3 Ultra, served by Nebius Token Factory, reads the inputs and proposes a controlled experiment. It names competing explanations, predicts their outcomes, picks its own bet and writes the Python harness.

CounterLab checks the design before execution. The uploaded code and harness run in a restricted Docker container without network access or credentials. Every condition runs twice with the same seeds. The measurements must be valid and match before CounterLab computes a finding. Nemotron can propose one follow-up and repair a broken harness within fixed limits; its committed predictions stay unchanged during repairs.

The interface shows the plan, charts, verdicts, model activity and failures. It also has setup checks, progress and Stop controls, searchable history and downloadable records. The included Python replay verifies file hashes and recomputes findings from saved measurements without an API key. Optional code re-execution requires the experiment's dependencies.

A run I can show you

The recorded October 30 investigation took 87 seconds, with three Nemotron calls and 23,005 reported tokens through Nebius Token Factory. There were no transport retries.

Its first bet failed. Nemotron expected refreshing cache predictions only after a miss to cause fewer misses. That version averaged 601.9 misses; refreshing on every request averaged 557.3. None of the ten seeds supported the bet.

The follow-up also hit a problem: its first harness crashed on all 40 measurements. CounterLab returned the traceback, and Nemotron repaired the code. The repaired experiment matched its repeat execution, and both comparisons supported the follow-up bet on all ten seeds. You can still inspect the failed first bet and the broken harness attempt.

The other three bundled open examples use labelled simulated planning with real numerical execution. They check the integration, not Nemotron's experiment-design ability.

How I built it

I developed CounterLab as a solo entrant with AI assistance. Python and FastAPI handle intake, schema-checked model responses, design checks, Docker execution and record sealing. The React and TypeScript interface lets you inspect each experiment and its evidence.

Nebius Token Factory supplies inference for Nemotron's design, code generation, repair and follow-up. Deterministic Python computes the measurements and findings. Temporary provider errors have bounded, recorded retries; redesign and harness repair have separate limits. The hosted judge edition lets you explore recorded investigations. The full source build runs new investigations locally.

What I learned

Keeping the AI accountable took more work than displaying a convincing answer. A repeatable experiment can still measure the wrong thing, and a plausible prediction can fail.

The separate learning-augmented paging case study includes a frozen eight-task comparison with non-AI baselines. Adaptive Nemotron found both available finite differences, but a simple heuristic found them too. Both of Nemotron's predictions were correct on only 2 of its 11 measured tests. These results do not establish an AI selection advantage. They are why CounterLab preserves controls, predictions and failed attempts alongside successful results.

Limits and next steps

Open Lab currently runs bounded Python/CPU experiments. It does not support scanned PDFs, large training jobs or arbitrary services. Code in another language may be read and reimplemented in Python, with that limitation recorded. Generated experiments still need scientific review. One live open investigation demonstrates the workflow; it does not establish general reliability. The paging study does not settle the full published disagreement or theorem.

Next I would build a frozen benchmark of open investigations, add more language runtimes and support hosted execution. The current project uses Nebius Token Factory and NVIDIA Nemotron. It does not use Nebius AI Cloud or Tavily.

Built With

• docker
• fastapi
• nebius-token-factory
• nvidia-nemotron
• pydantic
• pypdf
• python
• react
• sqlite
• typescript
• vite