Replicator

An autonomous agent fleet that turns published research claims into runnable experiments, measured evidence, and signed verdicts—without inventing results.

Comment

Inspiration

AI can now summarize papers, generate convincing explanations, and even write research-style text. But scientific trust requires something harder: evidence.

Reproducing a published result still means manually extracting the exact claim, locating datasets and dependencies, reconstructing an environment, executing the experiment, repairing failures, and tracing every conclusion back to a real measurement. This can take researchers days or weeks.

I built Replicator to answer one question:

AI can make a research claim—but can it prove one?

Replicator is not another chatbot. It is an autonomous scientific-verification system that takes responsibility for the operational workflow from paper to experiment to evidence.

What it does

Give Replicator an arXiv paper and explicit limits for cost, runtime, and repair attempts.

A fleet of six specialist agents then:

  1. Reader downloads the paper, examines its text and figures, and extracts measurable quantitative claims.
  2. Planner selects a defensible claim and designs a bounded experiment that fits the submitted guardrails.
  3. Builder generates contract-bound experiment code, validates it, and creates an immutable container image.
  4. Runner executes the experiment inside an isolated Cloud Run Job and autonomously diagnoses and repairs failures within the attempt limit.
  5. Verifier compares the paper's reported value with persisted measurements and evidence artifacts.
  6. Reporter produces a signed evidence report connecting every verdict to its protocol, code, logs, metrics, cost, and trace.

The researcher can leave while the workflow runs asynchronously and later return to the complete mission history, activity stream, repair memory, evidence ledger, and shareable report.

Replicator is deliberately fail-closed. If a claim is too expensive to test, it is marked NOT_ATTEMPTED. If generated code or infrastructure fails before producing a measurement, Replicator records a system failure—not a verdict against the paper. It never invents an obtained value to make a run look successful.

How I built it

Replicator is deployed as an asynchronous, event-driven system on Google Cloud.

The public Mission Control interface and API run on Cloud Run. Mission state, attempt history, idempotency records, verdicts, and repair memory are persisted in Firestore. Pub/Sub decouples the API from the six asynchronous workers and supports authenticated delivery and retries.

The agents are implemented with Google ADK and use Gemini 3.5 Flash through Vertex AI for multimodal claim extraction, structured planning, code generation, failure diagnosis, and repair.

Generated experiment containers are built with Cloud Build, stored in Artifact Registry, and executed as isolated, budget-capped Cloud Run Jobs using a restricted service identity. PDFs, generated source, logs, figures, and metrics.json files are stored as immutable evidence artifacts in Cloud Storage.

The Verifier derives numerical verdicts from persisted artifacts rather than free-form model output. The Reporter then creates a signed, shareable evidence trail. Cloud Logging and Cloud Trace correlate activity across the complete asynchronous mission.

The frontend was built with HTML, CSS, and JavaScript, while the API and workers use Python and FastAPI. Terraform and deployment scripts reproduce the Google Cloud infrastructure.

Challenges I ran into

The hardest problem was distinguishing failure of the system from evidence against a paper. Early experiments could fail because of unavailable datasets, invalid generated code, dependency conflicts, network restrictions, or resource limits. Treating those failures as “not reproduced” would be scientifically dishonest.

I introduced an explicit system-outcome path so only a persisted measurement can produce a scientific verdict.

Generated code also required stronger boundaries than ordinary application code. Replicator now validates source structure and syntax before dispatch, uses immutable containers, isolates execution identities, validates artifact paths, and limits every mission by attempts, runtime, and accounted cost.

The asynchronous architecture introduced redelivery and duplicate-execution risks. Workers therefore claim event IDs transactionally before creating side effects, while Firestore preserves the state needed to recover safely.

Finally, I had to make a complex autonomous workflow understandable. The interface now exposes the current stage, elapsed time, attempts, spend, agent decisions, next action, final system outcome, and every claim-level verdict.

Accomplishments that I'm proud of

  • Built and deployed a real six-agent workflow rather than a conversational prototype.
  • Completed end-to-end missions against genuine arXiv papers.
  • Executed generated experiments in isolated Google Cloud jobs.
  • Implemented autonomous diagnosis, repair, and retry within explicit guardrails.
  • Created artifact-backed numerical verdicts and shareable signed reports.
  • Added persistent run history and reusable repair memory.
  • Implemented fail-closed handling for infeasible and failed experiments.
  • Added correlated logs, trace IDs, immutable image digests, runtime, and cost evidence.
  • Created a public Mission Control interface where judges can inspect the system working.
  • Built 73 automated tests covering the API, workers, security boundaries, redelivery, execution, verification, reporting, and telemetry.

I am especially proud that Replicator remains useful when an experiment fails. Instead of hiding the failure or manufacturing a result, it shows precisely what happened and what remains unknown.

What I learned

I learned that trustworthy autonomy is less about making an agent sound intelligent and more about controlling what it is allowed to claim.

Model output should propose plans, code, and repairs, but it should not be the source of scientific truth. Persisted measurements and inspectable artifacts must remain the authority.

I also learned that budgets, isolation, idempotency, observability, and honest failure states are not secondary infrastructure concerns. They are core product features for any agent expected to perform consequential work without constant supervision.

What's next for Replicator

Next, I plan to add domain-specific experiment adapters for more classes of machine-learning and scientific papers, along with a public benchmark measuring claim-extraction accuracy, execution success, repair effectiveness, cost, and verdict reliability.

I also want to add researcher feedback loops for approving claim selection and experiment protocols, collaborative evidence reviews, stronger dataset provenance checks, and reusable organization-level experiment policies.

The long-term goal is a living verification layer for scientific literature: every published quantitative claim connected to executable protocols, measurements, provenance, and an evidence history that evolves as tools and datasets change.

Built With

Share this project:

Updates

Submission history