meta-science

Hypothesis: a model can improve its own scientific method. Thesis: nothing counts until it beats the champion, by a margin, on worlds it cannot see. …We built that gate — and it holds.

Comment

Inspiration

"Self-improving agent" is the most common unfalsifiable claim in this field. Every demo shows successes, which proves nothing: a system that reports it improved is indistinguishable from one that logs "improved!" and changes nothing.

We wanted to build the version that can be checked — where the interesting artefact is not a system that improves, but one that refuses to. Our first axiom, in the old tongue: Ex probatione propria non sequitur veritas — from one's own proving, truth does not follow.

What it does

An agent meets a world it has never seen: variables with opaque names, no documentation, and two affordances — look, or act. It forms a falsifiable hypothesis, designs its own intervention, runs it, and finds out it was wrong.

Then it does the same thing one level up. It proposes a change to its own experiment-design strategy, and that proposal is evaluated on 24 held-out worlds it has never seen and cannot enumerate. It is promoted only if it beats the champion by a margin — and refused otherwise, with a receipt saying why.

In a live run, Gemini found a genuine improvement — half the measurement for a small, audited accuracy cost — and it was promoted. It then proposed two further refinements. One scored higher and was refused anyway, for gaining +0.0139 where the margin asks +0.02; the other scored worse. Refusing a real gain is the point: it is what stops a system ratcheting forward on noise.

We ran the loop three times and published all three, receipts included: one promotion and two refusals every time, and four of the six refusals were of candidates that scored higher and missed the margin.

One number up front: reading observations alone, Gemini recovers the causal direction in 0 of 4 confounded worlds; the same loop, allowed to intervene, recovers 4 of 4.

How we built it

Worlds. Structural causal models seeded from real scientific topologies — state relations, exponential response, transport chains, compartment flows, inheritance, confounding. Real science supplies non-trivial structure and known ground truth for free.

Anonymisation, as a tested property. Every retrievable surface is stripped at birth: domain names become X₁…Xₙ, labels shuffle per world, constants and exponents randomise, the functional family varies. Otherwise the agent retrieves rather than discovers — Gemini already knows the gas laws and would emit one on recognising the shape, then design experiments confirming what it had already said.

The thesis, as the gate enforces it:

$$M' \succ M \iff S(M', W) \ge S(M, W) + \varepsilon,\quad W \cap \mathrm{view}(M') = \varnothing$$

The gate. Every record lives in one of three tiers — raw (anything the agent produces), wiki (candidate findings), output (accepted canon) — and a record's tier is its status: nothing is promoted by assertion, only by moving up. Gemini proposes; it never sees the held-out seeds, never runs the scorer, and never writes above raw. The gate computes the evidence itself.

Google stack. Gemini 3.6-flash (falling back to 3.5-flash) through the GenAI SDK drives both reasoning roles — the scientist that experiments and the proposer that suggests method changes. Firestore holds the ledger of every verdict; Cloud Run executes the runs. All of it declared in Terraform, with the API key in Secret Manager and a service account scoped to roles/datastore.user.

Challenges we ran into

We built a metric we could have gamed. Cost initially charged the number of experiments, which left sample size free — so a challenger could buy accuracy by drawing more data and appear to have learned something. One did exactly that. Cost now charges total measurement, and that candidate loses.

A silent failure that looked like an empty answer. The model cascade caught every exception and moved on, so a 503 UNAVAILABLE from gemini-3.7-flash surfaced as a model with nothing to say. Failures are now recorded, only a transient overload is retried — and the cascade starts at 3.6.

An auditor that flagged everything. We added a second Gemini model to check whether each promotion was earned rather than gamed. It initially flagged every single promotion — which carries exactly as much information as flagging none. The fix was not a better prompt but a better receipt: the score split into its accuracy and cost parts, plus a noise floor below which movement is just sampling variation. "Spent less and kept the answers" is now distinguishable from "spent less and lost accuracy" — the two cases it exists to separate.

Receipts that did not replay. A test pinning the README's figures to the code failed on its first run, for a cause worse than a stale number: world generation was seeded with Python's builtin hash(), which is randomised per process. The same seed built a different world in every run, so the replayability the receipts promise did not hold and a judge would have got different numbers than we published. Fixed with a stable hash, and determinism is now asserted in a subprocess — an in-process check cannot see it.

Proving anonymisation rather than asserting it. Checking that banned words are absent is easy. The real question is whether a model can answer without experimenting. So we tested it adversarially — and it cannot.

Accomplishments that we're proud of

The measurement that settles the whole design — four worlds with a hidden confounder, scored on whether the agent recovers the true causal direction:

confounded worlds recovered
Gemini, observation only 0 / 4
The same loop, allowed to intervene 4 / 4

Identical worlds. The only difference is the ability to act. That single comparison shows the anonymisation holds — if Gemini could recognise these worlds, observation alone would have sufficed — and that intervention is doing real work rather than decorating a conclusion the model already had.

Our benchmark was flattering us, so we changed it. Late on we checked whether the two arms of an intervention — the treated and untreated runs — share random draws. They did — common random numbers, which makes the paired difference isolate the causal effect. Elegant, and a trap: with the noise cancelled, estimates stay precise however few samples are drawn, so cutting measurement was free score. That is very likely why the proposer kept winning by asking for fewer samples. We measured both regimes: at 25 samples per arm, accuracy holds at 98.1% under shared noise but falls to 84.3% under honest, independent noise. Independent noise is now the default, the extreme strategy correctly loses, and the promotion the demo shows survives the harder regime. Both facts are asserted by tests.

What we learned

That the hard part of a self-improving system is not the improving. It is building the evidence the improvement has to survive — and then not letting the proposer anywhere near it.

And where the claim stops. Automated scientific discovery has substantial prior art — active learning, causal discovery, rule-induction in the Zendo/Eleusis tradition. This is not the first AI scientist and does not claim to be. The narrower claim is falsifiable self-improvement: a loop whose every promotion is earned against evidence the proposer cannot see, score, or tune against, leaving a receipt either way.

What's next for meta-science

Level 3 — after level 1 (do science) and level 2 (improve the method) — is open-ended, repeated self-evolution scored on law recovery under replication. Every hypothesis is replicated across fresh seeds, with verdicts by meta-analysis, and the model proposes functional forms rather than effect sizes. The promotion gate then does for many generations what it does today for one: margins against noise-ratcheting, an independent auditor against metric-gaming, receipts for the whole lineage, and ever-fresh generated worlds as the pre-registration analogue that keeps repetition from becoming p-hacking.

The Kepler test. The obvious next dataset is the one that started modern astronomy: Tycho Brahe's Mars oppositions — the numbers Kepler ground through for years before the eight arc-minutes of residual that would not go away forced him to an ellipse. It is harder than it looks, in exactly this project's way. Mars is the most recognisable dataset in the history of science, so a model shown labelled longitudes answers "ellipse" from memory before fitting anything: retrieval, not discovery — which is why our worlds are anonymised at birth. And nobody intervenes on a planet, so the law must be earned by fitting rather than by acting. Which is the reason to run it twice: once labelled, once with the dates stripped and the axes renamed. The gap between those two runs measures how much of a model's "discovery" is recall, on the one dataset where memorisation is guaranteed.

Plus the smaller debts: non-linear effect estimation, and a larger held-out set so smaller margins become measurable.

Try it yourself

Live demo — meet a world it has never seen, run an experiment, watch the gate decide World inspector — what the agent is allowed to know, beside the truth it is not Evidence base — the 384-run study behind every figure here Receipts — three live runs, committed in full, replayable offline without an API key Source — GPL-3.0, free for everyone, human or AI

Link Label

1 https://science.meta-agentic.ai/ Live demo — run the gate yourself 2 https://github.com/meta-agentic/meta-science Source (GPL-3.0) 3 https://github.com/meta-agentic/meta-science/tree/main/docs/receipts Three live runs, receipts committed 4 https://science.meta-agentic.ai/world/7/inspect World inspector — what the agent may know vs. the truth 5 https://science.meta-agentic.ai/evidence Evidence base — the 384-run study

Built With

  • cloud-run
  • docker
  • fastapi
  • gemini-3.5-flash
  • gemini-3.5-flash-lite
  • gemini-3.6-flash
  • google-cloud-firestore
  • google-genai
  • pytest
  • python
  • secret-manager
  • terraform
Share this project:

Updates

Submission history