Scientific Discovery Swarm

A brainstorming partner for scientists. You talk to it about your data, it turns vague hunches into experiments, runs them in background and comes back with real numbers and figures.

Comment

Inspiration

I wanted an AI that could help me think about my data. What I got, every time, was an AI that agreed with me fluently.

Ask a language model a research question and it returns confident, plausible, unfalsifiable text in seconds. It cannot tell you whether a connection is genuinely unmade or merely unmentioned in its training data, and it cannot show you its work.

Building a multi-agent system on top of that adds a second problem. The largest empirical study of these systems — MAST, arXiv:2503.13657, 1,600+ annotated traces across seven frameworks — found that 23.5% of failures are verification failures, and concluded they "require architectural solutions rather than improved base models." GPT-4o and Claude 3.7 Sonnet failed that category equally.

I did not believe that until it happened to me four times.

What it does

You talk to it about your data. It turns a half-formed hunch into experiments that can actually fail, runs them sandboxed in the background while you keep thinking, and comes back with real numbers and real figures.

Then it stops. It never tells you that you found something — every run lands inconclusive until a person reads the evidence and enters a verdict.

Working results from real data, all with their positive control passing first:

  • Polar amplification, recovered blind from 57 ERA5 weather stations — R² 0.498, p = 8.55e-10 — then the hemispheric asymmetry on its own initiative, taking R² from 0.498 to 0.609. Four cents.
  • A refutation. I predicted nodule size alone would predict lung malignancy as well as the full CT volume. It doesn't: AUC 0.567 against 0.803, non-overlapping bootstrap intervals.
  • A null. Asked whether warming depends on longitude — where nothing should be there — it returned p = 0.367, while the same code still recovered latitude at R² 0.498.

How we built it

Every agent writes three documents and dies: a full trace, a summary, and a log of every command with its exit code. Written evidence-first, so a claim cannot exist before the action that supports it. The summary is the only thing the next agent ever receives — there is no memory, no shared state, no conversation handle.

That layout makes claims mechanically checkable. Claims live in one file, evidence in another, and a deterministic join verifies that every claim points at something that actually happened.

Google ADK and Gemini 3.5 Flash-Lite on Vertex AI in every slot including the verifiers. Cloud Run for the control plane, scaling to zero; Cloud Run Jobs for fan-out — one task, one agent, one process. Firestore for the live manifest and the persisted approval gate, GCS as the system of record, BigQuery for the cost mirror. Model-generated code runs in a pinned container with no network, a read-only root, all capabilities dropped and an unprivileged user.

Challenges we ran into

Every real defect produced a confident, well-formed, wrong result. Not one produced an error. That single observation reorganised the entire project.

An agent asked to implement a discrete-element model substituted an analytic proxy formula and reported success — one step from being fitted as evidence for a preregistered hypothesis. Another was asked to sweep 1,300 parameter points, ran 3, called it a simplification, and reported success. Two named constants drifted between specification and code and the run reported success.

The worst one happened this week. An agent couldn't find my dataset — my plumbing bug, not its — and instead of stopping, it generated a thousand synthetic lung nodules with the exact size-malignancy relationship it was testing for written into the generator, then prepared to verify that relationship as its positive control. A hypothesis confirmed by data built to confirm it. Two gates caught it and killed the run for three and a half cents.

None of these would have been caught by reading the output. All were caught by a structural check comparing what was asked for against what was run.

Accomplishments that we're proud of

The number that justifies the architecture. On thirty injected defects, the deterministic evidence join catches 40 of 40. An LLM verifier reading exactly the same documents catches 4. Paying 12.4× for a larger model moves it to 8 — still worse than the mechanical check, at twelve times the price. We publish both numbers because the gap is the design.

It reports its own failures. swarmctl status prints the project's open gaps, unprompted, from the preregistration ledger. Literature-gap retrieval — the most exciting thing I set out to build — scores 0/3, and the system says so every time you ask. With an empty ledger it prints NO SCIENTIFIC DISCOVERY HAS BEEN MADE.

Gate immunity. It learns how you work into a readable profile.md and accepts standing instructions for individual agents, but neither can widen a gate. "Skip the positive control" is refused by the same allowlist in both places, and the four obvious attack strings are pinned in a test.

$5.84 across 57 runs and 369 agents, with 343 tests.

What we learned

Assume the failure is silent. Ask of every component: if this were broken, what would I see? If the answer is "correct-looking output", you need a mechanical check, not a better prompt.

Make the model declare, then check the declaration. "Did you implement the requested method?" is answerable and checkable. "Did you do a good job?" is neither.

Let the positive control fail the run. Controls were named in every objective and enforced nowhere, until an agent computed an AUC of 0.3966 against a threshold of 0.5, wrote "the value 0.3966 was not strictly greater than 0.5, but the script's condition for positive control passed", and marked itself successful. It noticed. It wrote it down. Nothing joined the number to the verdict.

Use an allowlist, never a blocklist. Eight API routes shipped unauthenticated because the gate was a per-route decision and each route was added on a different day — two of them spent model credits.

And the bug I am most embarrassed by: import importlib does not bind importlib.util, so the package probe had been broken since the first commit. Every sandbox reported "no packages available", every agent was told the standard library was all it had, and that is why one of them substituted a proxy formula for a simulation it could not write in pure Python. The visible failure was three layers downstream of a typo.

What's next

Fix literature-gap retrieval — the 0/3 is a taxonomy-resolution problem we have diagnosed and not yet demonstrated fixed. Run the Swanson replication that has never been run. Add pycocotools and a GPU path so the swarm can enter the MAGFiLO solar filament segmentation challenge at IEEE BigData 2026, which is a real benchmark with real ground truth.

And a genuine discovery, preregistered before the result is read. The system is built so that when one arrives, it will be checkable — and so that until then, it says so.

Built With

Share this project:

Updates