EvidenceBound Recovery Mesh

Selective self-healing for agent fleets: exact trust blast radius, fail-closed actions, durable revalidation, and exact-run Google Cloud audit.

Comment

What it does

EvidenceBound Recovery Mesh is a trust-aware flight recorder and selective self-healing control plane for autonomous agent fleets.

When one evidence source, tool result, policy binding, or agent checkpoint becomes untrustworthy, Recovery Mesh does not restart the whole fleet and it does not continue from contaminated state. It takes a third path:

TRUST BREAK → EXACT BLAST RADIUS → ACTION BLOCKED → SAFE WORK REUSED → AFFECTED BRANCH RECOMPUTED → VERIFIED RECOVERY

The recovery decision is autonomous. The operator does not choose the repair set or decide which agents rerun.

The twist / what is novel

Most agent reliability approaches focus on preventing a bad answer, logging what happened, or restarting after failure. Recovery Mesh treats trust itself as a dependency graph that can be repaired selectively.

The hackathon-built mechanisms are:

  1. Directed Trust Graph + exact blast radius — every material checkpoint binds dependencies and cryptographic digests, so one invalid source produces a deterministic minimal repair set rather than a fleet-wide reset.
  2. Persisted state is not trusted state — Firestore gives durability, but persisted checkpoints regain VERIFIED status only after dependency, evidence/input digest, provenance, integrity, and active-policy validation.
  3. Fail-closed durable side-effect protocol — production acceptance proves 0 durable action receipts while publish_action is BLOCKED → exactly 1 receipt after verified recovery.
  4. Model/authority separation — Gemini reasons; deterministic Recovery Mesh code owns trust state, blast radius, policy checks, reuse decisions, and action authorization.
  5. Cross-plane evidence binding — the same run_id connects the Flight Recorder, Firestore durable state, and an independently queried Google Cloud Logging audit trail.
  6. Measured selective recovery — the reference live run reuses still-verifiable work and measures actual model-call/token reduction instead of claiming hypothetical savings.

This is more than an audit dashboard: the system detects the trust break, blocks the unsafe action, computes the repair plan, executes selective recomputation, re-verifies state, and resumes only when the deterministic gate passes.

An “Unlikely Hero” workload

The judge workload models a football performance / match-intelligence operations analyst whose autonomous fleet must never publish a recommendation derived from stale, contradictory, malformed, or policy-invalid evidence.

That is deliberately outside a standard corporate back-office role while still exercising a real institutional problem: multiple specialized agents transform evidence into a downstream operational recommendation, and a single contaminated input can poison the chain.

The demo uses safe controlled fixture/history/policy data. It does not present fixtures as live sports-provider truth and does not copy SignalReview production source into this repository.

Live production proof

Hosted Flight Recorder:

https://evidencebound-recovery-mesh-i3lzjodgra-ew.a.run.app/

Preferred hands-off judge route:

https://evidencebound-recovery-mesh-i3lzjodgra-ew.a.run.app/?autorun=stale_evidence&recover=1

Current verified production revision:

evidencebound-recovery-mesh-00007-bjm

Reference acceptance:

  • deployment workflow 33196157041 — SUCCESS
  • live acceptance + exact-run cloud-proof workflow 33196523402 — SUCCESS
  • acceptance run run-4f1eba151be7
  • execution provider google_adk_vertex
  • model gemini-3.5-flash
  • Google ADK baseline: 4 specialized live agents
  • Firestore persistence: durable=true
  • unauthenticated POST /api/runs → 401
  • trust break: history_snapshot / STALE_EVIDENCE
  • unsafe downstream action: publish_action = BLOCKED
  • unaffected agent work: Scout = REUSED
  • selective recovery: 3 rerun / 1 reused
  • final action: VERIFIED
  • exact-run Google Cloud Logging: PASS

Durable Trust Ledger proof

The current production revision uses Firestore as an active Durable Trust Ledger.

For the same live reference run:

FIRESTORE_BLOCKED_RECEIPT_COUNT=PASS count=0 run_id=run-4f1eba151be7

DURABLE_RECOVERY=PASS receipt=present rehydration=trusted

FIRESTORE_RECOVERED_RECEIPT_COUNT=PASS count=1 run_id=run-4f1eba151be7

Interpretation:

  • while the action is BLOCKED, no durable side-effect receipt exists;
  • after selective recomputation and deterministic re-verification, exactly one durable receipt exists;
  • persisted state is revalidated before reuse;
  • Firestore supplies durability, not trust authority.

The exact acceptance did not deliberately kill a Cloud Run instance between write and replay, so we do not present it as a separate forced-instance-restart exactly-once benchmark. Crash/restart invariants are covered by repository tests; the live receipt proves the active Firestore data path and rehydration gate.

Exact-run Google Cloud audit proof

A separate read-only auditor identity queried Google Cloud Logging for the exact same run-4f1eba151be7 and returned:

EXACT_RUN_CLOUD_LOGGING=PASS

GCP_PROOF_RECEIPT=PASS

The observed causal sequence includes:

TRUST_BREAK_DETECTED → BLAST_RADIUS_COMPUTED → ACTION_BLOCKED → CHECKPOINT_REUSED → RECOMPUTE_STARTED → CHECKPOINT_REVERIFIED → ACTION_RESUMED → RECOVERY_COMPLETED

Cloud Logging provides external audit evidence; it does not decide trust.

Autonomous recovery flow

After the private judge key is entered once and controls are unlocked, the preferred route requires no human recovery choice. Recovery Mesh:

  1. creates a fresh live Google ADK / Gemini baseline;
  2. injects the controlled stale-evidence fault through the production API;
  3. freezes publish_action before the unsafe side effect;
  4. traverses dependencies and computes the exact downstream blast radius;
  5. preserves Scout because its inputs remain verifiable;
  6. selectively reruns Statistician, Skeptic, and Orchestrator;
  7. deterministically re-verifies the recomputed checkpoints;
  8. commits the durable action receipt only after trust passes;
  9. resumes to final VERIFIED state.

The manual route without recover=1 intentionally pauses in the blocked state so a judge can inspect the blast radius.

Measured selective-recovery ROI

For reference run run-4f1eba151be7:

  • full restart: 4 model calls / 1707 input tokens
  • selective recovery: 3 model calls / 1432 input tokens
  • saved in this run: 1 model call / 275 input tokens (~16.1%)

These are measurements from one controlled production run, not a universal savings claim. Gemini usage varies; the Flight Recorder reports actual run receipts.

Architecture and Google stack

Specialized fleet

  • Statistician — verifies statistics and metrics
  • Scout — discovers and validates independent checkpoints
  • Skeptic — challenges assumptions and dependencies
  • Orchestrator — plans the final action and recovery path

The multi-agent structure is necessary because the core failure being solved is cross-agent contamination: one upstream trust break can invalidate only part of a larger fleet while other specialized work remains valid.

Google services

  • Google ADK 2.7.0 — specialized multi-agent fleet
  • Vertex AI / Gemini 3.5 Flash — bounded live reasoning
  • Google Cloud Run — production runtime, scale-to-zero
  • Firestore — durable checkpoints, Flight Recorder events, rehydration state, action receipts
  • Google Cloud Logging — exact-run external causal audit proof
  • Google Agent Registry — verified catalog/discovery entry for the Recovery Mesh fleet endpoint
  • Google Secret Manager — private judge credential
  • Google Workload Identity Federation — keyless GitHub-to-GCP deployment/audit identities

Application stack: Python, FastAPI, Pydantic, Docker, GitHub Actions.

Deterministic trust boundary

Gemini can analyze evidence and return bounded structured output, but it cannot:

  • mark a checkpoint VERIFIED;
  • decide that persisted state is trusted merely because it exists;
  • choose a smaller blast radius than the dependency graph requires;
  • bypass provenance, integrity, dependency, or active-policy checks;
  • authorize publish_action while dependencies are unsafe.

Recovery Mesh owns those decisions deterministically.

Every material checkpoint binds IDs/versions, dependency IDs, parent-output digests, evidence/tool-result digests, policy version, structured output digest, verification state, provenance/integrity metadata, and timestamps where applicable.

Minimum states are VERIFIED, INVALIDATED, RECOMPUTE, and BLOCKED.

Fortified Enterprise Fleet fit

Discovery & lifecycle: the production fleet endpoint is registered and discoverable in Google Agent Registry.

  • Service: projects/evidencebound-rm-c977c1/locations/global/services/recovery-mesh-fleet
  • generated read-only Agent: projects/457699623691/locations/global/agents/agentregistry-00000000-0000-0000-a7f5-b9837959f789
  • discovery workflow 31871557186 → PASS

The Registry catalogs the Recovery Mesh fleet endpoint; we do not claim separate Registry entries for every internal ADK role.

State & continuity: Firestore provides a durable cross-session state substrate. Recovery Mesh explicitly revalidates old state before reuse, which prevents “persistent” from becoming synonymous with “trusted.” We do not claim that the acceptance literally waited multiple weeks, and we do not claim Memory Bank or Agent Runtime integration without evidence.

Security & governance: fail-closed action gate, protected judge API, Secret Manager, keyless WIF, separate runtime/build/deployer/auditor identities, strict Gemini structured-output contracts, deterministic provenance/integrity/policy gates, durable idempotency receipt boundary, no silent fallback from failed Google execution.

Telemetry: the same production run is visible in the Flight Recorder and independently queried from Google Cloud Logging under a separate auditor identity.

Data sources

The judge workload uses controlled fixture/history/policy data to create a genuine multi-agent dependency chain. Controlled faults for stale evidence, malformed worker output, and policy drift enter the same verification/recovery contracts and are clearly labeled fixtures.

A separate deterministic scale probe exercises 100 synthetic agent checkpoints with the same blast-radius planner. Its controlled receipt is 14 affected / 86 reused / 1 blocked action. This is a graph-scale test, not 100 live Gemini calls.

Findings and learnings

  • Persisted state is not trusted state. Durable context must be revalidated before reuse.
  • Recovery can be selective without becoming permissive. Exact dependency traversal preserves safe work while failing closed around contamination.
  • Side-effect safety needs a durable boundary. Production proves 0 receipts while BLOCKED → exactly 1 after VERIFIED recovery.
  • Audit evidence should be independently queryable. The same run_id is visible outside our UI in Google Cloud Logging.
  • Gemini should reason, not grant trust. Deterministic verification remains authoritative.
  • Enterprise discovery is separate from trust authority. Agent Registry can discover the fleet, not change its trust state.

New-project disclosure

EvidenceBound Recovery Mesh is a new isolated hackathon repository built during the August 2026 Submission Period. Pre-existing EvidenceBound ideas and SignalReview concepts are disclosed, but no SignalReview production source and no prior EvidenceBound implementation source is copied into the project. Hackathon-built implementation is traceable in git history.

Explicit non-claims

We do not claim without separate evidence:

  • Agent Runtime;
  • Memory Bank;
  • Model Armor;
  • BigQuery export;
  • a forced Cloud Run instance-kill production replay proving restart-surviving exactly-once behavior;
  • universal token or cost savings;
  • 100 live Gemini agents;
  • that a Google service, rather than Recovery Mesh deterministic contracts, detected or resolved the trust break.

Category

Fortified Enterprise Fleet. Recovery Mesh provides autonomous trust-break detection, exact blast-radius computation, fail-closed action, verified-work reuse, selective branch recomputation, durable Firestore state, exact-run Google Cloud auditability, and production fleet discovery through Agent Registry.

Built With

  • docker
  • fastapi
  • firestore
  • gemini-3.5-flash
  • github-actions
  • google-adk
  • google-agent-registry
  • google-cloud-logging
  • google-cloud-run
  • google-secret-manager
  • google-workload-identity-federation
  • pydantic
  • python
  • vertex-ai
Share this project:

Updates

Submission history