ReproSeal

• 

Claim boundary first: ReproSeal does not attest scientific correctness. For the demonstrated invocation, the Gemini/ADK Investigator's observable request, prompt, tools, environment, and response contained no target, tolerance, or Verifier verdict before candidate freeze. Cloud audit logs then recorded one KMS signature before the Verifier read the exact target Secret.

Inspiration

A scientific reproduction agent can accidentally optimize for the answer it is supposed to verify. If it sees a paper's reported value while changing dependencies, parameters, or evidence, a numerically close result may reflect target-aware fitting rather than an independent repair.

Telling an agent to “ignore the answer” is not an enforceable boundary. We wanted to turn the established logic of blind analysis into a runtime order of operations:

commit target
→ issue blind packet
→ investigate
→ freeze and sign
→ reveal to a separate Verifier
→ return a terminal receipt

The project began as ReproPilot, a broader paper-reproduction agent. Prior-art and competitor research showed that generic reproduction agents were already a crowded category, so the project pivoted to the narrower problem of outcome isolation.

What it does

ReproSeal separates scientific repair from outcome reveal.

The Investigator selects an allowlisted repair, but its observable request, prompt, tools, environment, and response omit the exact target, tolerance, and Verifier verdict. Metadata such as metric, unit, direction, and completed-trial count may still be bound into the candidate envelope; those fields are not claimed to be absent. Deterministic code validates, content-addresses, freezes, and signs the candidate before the Verifier reads the exact target Secret.

The Verifier performs a deterministic comparison and returns a target-free receipt without the target, observed value, delta, raw signature, or another feedback signal for iterative fitting. Within one live Verifier process, target access closes the episode; restart-durable one-shot state is not claimed.

ReproSeal preserves three honest outcomes:

• PASS: the isolated candidate met the precommitted computational criterion.
• FAIL: the process was valid, but the candidate missed the criterion.
• INDETERMINATE: the evidence cannot support a trusted comparison.

How we built it

The observed MVP path is:

authenticated demo driver
→ private Cloud Run Investigator
→ Gemini 3.6 Flash through Google ADK 2.6.2
→ reviewed local allowlisted runner
→ content-addressed candidate
→ private Cloud Run Verifier
→ Cloud KMS signature
→ exact Secret Manager version read
→ terminal target-free receipt
→ Cloud Audit Logs order check

The Investigator and Verifier are IAM-private Cloud Run services with distinct service accounts; unauthenticated requests are rejected. The Investigator has model access but no KMS signer or Secret accessor role. The Verifier holds the bounded reveal-side roles.

Google ADK constructs the agent run, Gemini returns a structured choice, and deterministic code validates it against the blind packet and output policy. The model gets neither a generic shell nor an arbitrary network tool. For the reduced UQ360 fixture, the reviewed local runner applies the selected SciPy migration and emits a self-content-addressed candidate.

The Verifier signs that candidate with Cloud KMS before reading one exact Secret Manager version. Data Access logs independently recorded one sign event followed by one exact-version read. The Python/Flask services remain separate from the React, vinext, and TypeScript viewer, which consumes only sanitized static evidence.

What makes it different

ReproSeal does not claim to invent blind analysis, hidden tests, deterministic verification, or signed software provenance. Its observed live differentiation is narrower: under the demonstrated IAM configuration, the Investigator's observable surface omitted the target, tolerance, and Verifier verdict, and a signed candidate freeze preceded exact target-Secret access. The target-free receipt closes the episode only within one live Verifier process; restart durability remains future work.

Challenges we ran into

The hardest challenge was making the information boundary enforceable rather than cosmetic. We also found an unlimited-retry flaw that could turn the Verifier into a membership oracle. Terminal-claim behavior closes it within one Verifier process, while restart durability remains an explicit limitation.

To make the proof reviewable without exposing the protected result or cloud identifiers, the demo emits bounded booleans and event counts rather than raw requests, responses, names, or signature bytes.

Accomplishments

• Deployed two authenticated, IAM-separated Cloud Run services and ran Gemini 3.6 Flash through Google ADK 2.6.2 on the live Investigator path.
• Had the agent select the evidence-backed SciPy migration while rejecting a post-hoc var_smoothing=0 control as contaminated.
• Completed a live PASS, with one KMS sign before one exact Secret read and zero observed reveal-side events under the Investigator identity.
• Captured a privacy-safe, continuous 134.7-second proof-of-action run inside a 3:34.734 public demo video.
• Published 95 Python tests, rendered frontend checks, and reviewable architecture, deployment, evidence, competitor, and disclosure artifacts.

What we learned

A matching number is not enough; the order in which information becomes available is part of the evidence.

Gemini is useful for choosing among reviewed actions; deterministic code must validate the choice, bind the candidate, compare the result, and control what the receipt reveals. An honest FAIL remains evidence, while INDETERMINATE stops when the boundary cannot support a trusted comparison. A polished viewer is still not enforcement evidence, so it remains separate from the private runtime proof.

What's next

The next work is production hardening:

• make terminal claims restart-durable;
• deploy an authenticated, egress-denied runner with a pinned image and dependencies;
• verify runner and freeze attestations against pinned trust roots and an independent event anchor; and
• execute an authorized forbidden-Secret negative test.

These items remain future work and are not claimed by the live MVP.

Honest MVP boundary

The demonstrated run supports one bounded claim: a submitted, content-addressed candidate was signed by Cloud KMS before the separately authorized Verifier accessed one exact Secret version and returned a target-free receipt that was terminal within that live process.

It does not prove:

• that the candidate came from a separately authenticated remote runner;
• that one-shot state survives a restart or scale-to-zero;
• that Python wheels are fully artifact-hash locked;
• that a separately signed runner attestation was verified against a pinned external trust root;
• that an actual forbidden Secret request was denied under the Investigator identity;
• that a cloud project owner could not collude across roles;
• that a public-paper model lacked prior knowledge of a reported result;
• source anonymity, independent replication, scientific validity, or paper correctness; or
• that the public reference viewer is the private enforcement path.

ReproSeal's claim is intentionally narrower: freeze the candidate before reveal, and make that information boundary reviewable.

Built With

• cloud-audit-logs
• cloud-kms
• cloud-run
• flask
• gemini-3.6-flash
• google-adk
• python
• react
• secret-manager
• typescript