RecoveryOS — Autonomous Recovery Control Plane

• 

• 

• 

• 

Inspiration

Autonomous AI agents are becoming capable of making decisions and executing real workflows. But during the Google All Things Agentic Hackathon, I became increasingly interested in a different question:

What happens after an autonomous agent takes an action and something goes wrong?

An agent can report success while an external service has failed. A workflow can be interrupted after an external mutation but before its internal state is persisted. Two systems can produce contradictory evidence. A retry can accidentally duplicate an action that already succeeded.

That led to the core idea behind RecoveryOS:

ACTION EXECUTED ≠ RECOVERY VERIFIED

RecoveryOS is designed as the recovery control plane around autonomous enterprise agents: a layer that detects failures, diagnoses them, governs recovery actions, reconciles uncertain execution, independently verifies business outcomes, and escalates when autonomous authority is no longer sufficient.

What it does

RecoveryOS provides a governed recovery lifecycle for autonomous enterprise agent workflows:

Detect → Diagnose → Plan → Govern → Execute → Reconcile → Independently Verify → Prove or Escalate

The system uses a multi-agent architecture with explicit separation of concerns.

The Orchestrator coordinates the workflow using Google ADK.

The Recovery Specialist is a dedicated Gemini reasoning agent with read-only diagnostic capabilities. It investigates failures, checks service health, discovers recovery options, and creates a structured RecoveryPlan without directly performing mutations.

Taskmaster is the shared mutating execution agent. State-changing actions pass through the RecoveryOS control boundary before execution.

The Verification Agent independently probes external service state and determines whether the intended business outcome actually occurred.

Around this core, RecoveryOS adds first-party enterprise fleet capabilities:

• Agent Registry
• Agent Identity and tenant/data scopes
• Agent Gateway
• Deterministic Agent Guardrails
• Durable Agent Context
• Correlated Fleet Observability
• Failure-Tolerant Routing with bounded recovery budgets

Three failure scenarios

Billing Provider Outage

A simulated Stripe HTTP 503 causes RecoveryOS to diagnose the failure, formulate a recovery plan, switch to PayPal under policy control, and independently verify the subscription before declaring:

RECOVERED • VERIFIED

Contradictory Evidence

An action reports success, but independent verification detects the wrong subscription tier.

RecoveryOS refuses to treat the action as proof:

Action reports success
        |
        v
Independent verification
        |
        v
Wrong plan detected
        |
        v
billing_configured = FAILED
        |
        v
AWAITING APPROVAL

Worker Interruption

The worker is intentionally interrupted after an external mutation succeeds but before internal persistence completes.

RecoveryOS marks execution uncertain, reconciles the external state, redispatches within bounded limits, prevents unsafe duplicate mutation, and independently verifies the resulting state.

How we built it

RecoveryOS is built around Google's agent and cloud stack.

Vertex AI provides production model serving.

Gemini 3.5 Flash is the primary reasoning model.

Gemini 3.5 Flash Lite is a bounded fallback used only for retryable model/provider failures such as quota exhaustion or transient model unavailability. Business failures such as a Stripe HTTP 503 do not trigger model fallback.

Google ADK (Agent Development Kit) provides agent orchestration and execution.

Cloud Run hosts the API and worker services.

Cloud Pub/Sub provides asynchronous workflow dispatch.

Firestore provides durable workflow state, agent context, evidence, optimistic concurrency control, and recovery history.

Production Vertex AI authentication uses Google Cloud workload identity / Application Default Credentials through the Cloud Run service identity rather than embedding a model API key in the application.

The Fleet layer adds:

• typed Agent Cards and capability definitions
• tenant and data-scope enforcement
• centralized gateway decisions
• deterministic guardrails
• durable structured context
• W3C-compatible trace correlation
• bounded fallback and recovery routing

The application uses synthetic enterprise scenarios so the recovery architecture can be demonstrated without exposing real customer data.

Challenges we ran into

The hardest part was not getting an agent to call a tool. It was making the system behave correctly when reality diverges from the agent's assumptions.

One major challenge was distinguishing tool success from business success. A tool returning success cannot be treated as proof that the desired enterprise state exists. This led to independent verification and the central RecoveryOS invariant.

Another challenge was distributed interruption. A worker can fail after an external system has changed but before RecoveryOS has persisted the corresponding internal state. Blind retrying can create duplicate mutations. RecoveryOS therefore reconciles external state before redispatching.

We also had to solve long-lived authentication, including automatic token renewal for the console so an expired access token does not interrupt a running operational session.

The Fleet architecture introduced another challenge: providing meaningful multi-agent separation without creating unnecessary autonomous mutation agents. We deliberately keep diagnosis and mutation separate while retaining a shared, policy-governed execution boundary.

Finally, moving production inference to Vertex AI required aligning Cloud Run service identity, IAM, ADK model configuration, global Vertex model availability, and a bounded fallback path without disturbing the existing recovery semantics.

Accomplishments that we're proud of

RecoveryOS turns failures into explicit, testable system states instead of treating them as edge cases.

The project now combines:

Autonomous diagnosis + governed execution + durable state + reconciliation + independent verification

into a single recovery control plane.

Some of the most important accomplishments are:

• A real multi-agent control plane using Google ADK and Gemini
• A dedicated read-only Recovery Specialist that creates structured recovery plans
• Centralized agent identity, gateway, guardrails, and policy enforcement
• Durable workflow and agent context across interruption scenarios
• Failure-tolerant routing with bounded recovery budgets
• Independent verification based on external ground truth
• Evidence-backed RECOVERED • VERIFIED outcomes
• A contradictory-evidence path that deliberately stops autonomous execution
• Post-mutation worker interruption recovery with reconciliation
• Production deployment on Cloud Run with Firestore and Pub/Sub
• Vertex AI production inference with Gemini 3.5 Flash and a bounded Lite fallback
• A large automated regression suite covering core recovery and Fleet behavior

The most important accomplishment is architectural rather than cosmetic:

RecoveryOS does not define recovery as “the agent finished.”

It defines recovery as:

the intended business outcome was independently verified.

What we learned

We learned that reliable agentic systems need to treat uncertainty as a first-class state.

A workflow should not simply be:

RUNNING → COMPLETED

It may need to become:

EXECUTING → UNKNOWN → RECONCILING → RECOVERING → VERIFYING → COMPLETED

We also learned that autonomy needs boundaries.

The agent can reason about what should happen next, but deterministic systems should decide whether a mutation is permitted, whether the operation is within scope, whether retrying is safe, and whether the final outcome is actually true.

Another important lesson was that multi-agent architecture is not about maximizing the number of agents. Separating diagnosis, mutation, governance, and verification can be more valuable than creating many loosely bounded autonomous agents.

Finally, production reliability is inseparable from the surrounding infrastructure: identity, persistence, event delivery, concurrency control, authentication, observability, and recovery all become part of the agent system itself.

What's next for RecoveryOS — Autonomous Recovery Control Plane

The next phase is to evolve RecoveryOS from a recovery system for individual workflows into a broader autonomous recovery control plane for enterprise agent fleets.

Future work includes:

• richer cross-agent discovery and lifecycle management
• stronger long-lived agent context and memory policies
• deeper identity and authorization boundaries
• distributed recovery coordination across multiple services
• expanded production observability and trace export
• policy-driven recovery simulation and testing
• additional enterprise failure domains beyond onboarding and billing
• recovery analytics focused on failures prevented, incidents recovered, and Human Hours Returned

The long-term vision is not an “autonomous company.”

It is a reliability layer that allows enterprises to deploy autonomous agents with greater confidence because the system surrounding those agents can:

detect failure, constrain action, recover safely, verify reality, preserve evidence, and return control to humans when necessary.

Govern. Recover. Verify.

Built With

• agentic-ai
• ai-agents
• ai-automation
• ai-governance
• ai-reliability
• ai-safety
• cloud-run
• developer-tools
• distributed-systems
• enterprise-ai
• fastapi
• fault-tolerence
• firestore
• gemini
• generative-ai
• google-adk
• google-cloud
• machine-learning
• multi-agent-systems
• observability
• pub-sub
• python
• saas
• vertex-ai
• workflow-automation