EvidenceBound Recovery Mesh

Trust-aware flight recorder and selective self-healing for autonomous agent fleets.

Comment

What it does

EvidenceBound Recovery Mesh is a trust-aware flight recorder and selective self-healing engine for autonomous agent fleets.

When one evidence or agent checkpoint becomes untrustworthy, the system does not restart the whole fleet. It deterministically:

  • detects the trust break;
  • freezes unsafe downstream action;
  • traverses the directed Trust Graph to calculate the exact dependency blast radius;
  • preserves checkpoints that remain verifiable;
  • invalidates only contaminated state;
  • recomputes only the affected agent branch;
  • re-verifies dependency digests, provenance, integrity and policy state; and
  • resumes the blocked action only after the final deterministic gate passes.

The judge moment is:

TRUST BREAK → BLAST RADIUS → ACTION BLOCKED → SAFE WORK REUSED → AFFECTED BRANCH RECOMPUTED → VERIFIED RECOVERY

Live production proof

The current submission-ready UI is deployed on Google Cloud Run as revision:

evidencebound-recovery-mesh-00005-82k

The deployment smoke on August 15, 2026 verified:

  • Vertex AI / gemini-3.5-flash: PASS
  • execution provider: google_adk_vertex
  • Google ADK baseline: 4 live agents
  • unauthenticated POST /api/runs: 401
  • controlled stale-evidence trust break: PASS
  • blocked unsafe action: publish_action
  • unaffected work reused: scout
  • selective recovery: 3 agents rerun / 1 agent reused
  • final action: VERIFIED

A reference production acceptance run, run-4707af5a2fb6, measured:

  • full restart: 4 model calls / 1781 input tokens
  • selective recovery: 3 model calls / 1358 input tokens
  • saved in that controlled run: 1 model call / 423 input tokens (~24%)

These are measured values from that exact controlled run, not a universal savings claim. A later smoke on the submission-ready revision measured 1728 vs 1393 input tokens, demonstrating why the Flight Recorder reports each live run's actual usage rather than hard-coding a percentage.

Verified Google Agent Registry proof

The existing Recovery Mesh Cloud Run fleet endpoint is now manually registered in Google Agent Registry at location global. Registration is a separate keyless control-plane workflow using the existing Workload Identity Federation deployer identity. It did not change the Recovery Mesh backend, Cloud Run revision, judge API, runtime behavior or Gemini/ADK execution path.

Verified workflow 31871557186 produced:

  • AGENT_REGISTRY=PASS
  • Agent Registry Service: projects/evidencebound-rm-c977c1/locations/global/services/recovery-mesh-fleet
  • generated read-only Agent: projects/457699623691/locations/global/agents/agentregistry-00000000-0000-0000-a7f5-b9837959f789
  • interface: https://evidencebound-recovery-mesh-i3lzjodgra-ew.a.run.app
  • AGENT_REGISTRY_DISCOVERY=PASS

The Registry entry represents the Recovery Mesh fleet entry point. We do not claim that Statistician, Scout, Skeptic and Orchestrator are separately registered.

Flight Recorder UI

The hosted UI renders the Trust Graph from runtime state and makes the recovery story visible in seconds:

  • the stale history_snapshot is marked as the TRUST BREAK source;
  • affected graph edges/checkpoints are highlighted;
  • publish_action becomes BLOCKED before the side effect;
  • Scout remains visibly REUSED;
  • Statistician, Skeptic and Orchestrator are selectively recomputed;
  • after recovery, the graph preserves the historical break/reuse markers while showing re-verification;
  • the benchmark panel shows full-restart vs selective model calls and actual input-token counts.

The public app is easy to discover, while run/read/fault/recovery APIs require the private testing key supplied to judges. The key is stored in Google Secret Manager and is never committed or embedded in the public UI.

How we built it

Google stack

  • Google ADK 2.7.0 — specialized Statistician, Scout, Skeptic and Orchestrator agents
  • Vertex AI / Gemini 3.5 Flash — live bounded agent reasoning
  • Google Cloud Run — production runtime, scale-to-zero (min=0, bounded max=1)
  • Google Agent Registry — verified catalog/discovery entry for the Recovery Mesh fleet endpoint
  • Google Secret Manager — private judge testing credential
  • Google Workload Identity Federation — keyless GitHub → Google Cloud deployment and Registry control-plane identity

Application/runtime stack: Python, FastAPI, Pydantic, Docker, GitHub Actions.

Deterministic trust boundary

Gemini may analyze evidence and return bounded structured output, but it cannot:

  • mark a checkpoint VERIFIED;
  • choose the blast radius;
  • override provenance/integrity/policy checks;
  • authorize publish_action while dependencies are unsafe.

Those decisions stay in deterministic Recovery Mesh code. Gemini worker output is constrained to a strict JSON contract and to the exact Trust Graph dependency IDs that the worker is allowed to cite. Unsupported provenance references fail closed.

Agent Registry is catalog/discovery control plane only. It cannot mark a checkpoint trusted, calculate blast radius or authorize an action.

Checkpoints and Trust Graph

Each material checkpoint binds IDs/versions, dependency checkpoint IDs, parent-output digests, evidence/tool digests, policy version, output digest, verification state, provenance/integrity metadata and timestamps where applicable.

Minimum states are VERIFIED, INVALIDATED, RECOMPUTE, and BLOCKED.

Controlled faults for stale evidence, malformed worker output and policy drift enter the same verification/recovery contracts. They are clearly labeled controlled fixtures and are not represented as live provider failures.

Data used

The judge workload uses safe controlled fixture/history/policy data to create a genuine multi-agent dependency chain. It does not present fixtures as live SignalReview provider truth and does not copy SignalReview production source into this repository.

A separate deterministic scale probe exercises 100 synthetic agent checkpoints with the same blast-radius planner. Its locked controlled receipt is 14 affected / 86 reused / 1 blocked action. This is explicitly a graph-scale test, not 100 live Gemini calls.

Security and production discipline

  • fail-closed action gate;
  • application-level protected judge APIs;
  • Secret Manager credential, never in source;
  • keyless WIF deployment restricted to the repository/owner/main branch;
  • separate runtime/build/deployer service accounts;
  • bounded Agent Registry API Editor role on the deployer for the separate registration workflow;
  • idempotency protection for side effects;
  • bounded live model calls;
  • no silent fallback from failed Google execution to deterministic output;
  • secret scanning and container build in CI.

Run objects in this bounded hackathon slice are process-local and are not claimed as durable enterprise memory after Cloud Run scales to zero. Judges create a fresh reproducible live incident through the UI. Firestore / BigQuery / Cloud Logging remain enterprise persistence extension targets, not active integrations. We do not claim Agent Runtime, Memory Bank, Model Armor or durable multi-week context without a separately verified integration.

Findings and learnings

Two implementation details mattered in production:

  • Cloud Run reserves some URL paths ending in z; moving the health endpoint from /healthz to /health fixed a Google Frontend interception that never reached the revision.
  • Gemini 3.5 Flash's default thinking budget could consume a small structured-output budget. Setting bounded execution to minimal thinking and constraining the response JSON schema produced reliable worker output while keeping deterministic post-generation validation authoritative.

A third enterprise integration lesson came from Agent Registry: manual Service registration is asynchronous. The fail-closed workflow waits for the Google long-running operation and requires the generated read-only Agent to be observable before it reports PASS.

The larger lesson is that persisted state is not automatically trusted state. Recovery should reuse work only while its evidence, provenance, integrity, dependencies and policy context remain verifiable.

New-project boundary

EvidenceBound Recovery Mesh is a new isolated hackathon repository built during the August 2026 submission period. No SignalReview production source and no prior EvidenceBound implementation source is copied into the project. Pre-existing concepts are disclosed in the repository; hackathon-built implementation is traceable in git history.

Category

Fortified Enterprise Fleet. Recovery Mesh is the fleet-integrity/recovery layer: it demonstrates specialized agents, deterministic trust propagation, fail-closed action, exact blast radius, selective recomputation, auditable recovery, production Google Cloud execution and a verified Google Agent Registry catalog/discovery entry for the live fleet endpoint.

The current demo still does not claim durable multi-week context. That limitation remains explicit rather than being hidden behind an unverified persistence diagram.

Built With

Share this project:

Updates