warranty-agent

Inspiration

Agents often report success when an action merely executed.

A restart can succeed while errors stay flat. Scaling can return 200 while p95 gets worse. Traffic shifting can complete while the real cause remains.

The problem is simple: nobody defined what “better” means.

That knowledge exists at provisioning time, but rarely survives into Day-2 operations.

What it does

When the agent provisions a resource, it also writes an operational contract:

• health signal
• recovery criterion
• rollback plan
• reversibility

During remediation it:

1. reads the contract-defined baseline,
2. gates on reversibility × verifiability × budget headroom,
3. executes,
4. measures the same signal again,
5. rolls back if recovery fails,
6. records executed, improved, and rolled_back separately.

The policy is simple:

If we cannot verify it, we do not automate it.

No contract or unreadable signal means MANUAL; the executor is never called.

Real deployed ledger:

executed 14 · improved 1 · rolled_back 12 · manual_required 3

Most systems would call this 14 successes. We call it one improvement.

How we built it

• ADK + Gemini 3.7 Flash / Vertex AI — model selects tools; deterministic code makes the gate decision.
• Cloud Run — agent and target; rollback uses traffic splits and is read back after execution.
• Firestore — contracts and immutable accountability ledger.
• Cloud Monitoring — identical p95 query logic before and after remediation.
• Ports and adapters — offline verification runs with zero billable calls.

The repository is spec-driven:

• IMPLEMENTED requires tests covering every promised behavior.
• VERIFIED requires a mutation proven to break the suite.
• make check fails if documentation claims exceed reality.

Challenges

The verification system repeatedly exposed problems that normal logs did not.

We found:

1. a 120s verification window still contained pre-remediation traffic,
2. a 60% recovery threshold was impossible when the maximum achievable improvement was ~31%,
3. updating code did not update previously issued contracts,
4. Cloud Run's default 300s timeout expired before the rollback verdict returned.

One live example improved from:

990.04 ms → 674.17 ms

yet was classified not_recovered because the contract itself was wrong.

We also discovered README metrics presented as measured results even though they came from a video script. We removed them.

The project’s own principle caught its own mistake:

executing is not improving.

What we learned

Verifiability must be part of the automation decision, not a postmortem step.

An action that is reversible but cannot be measured should still not run automatically.

We also learned:

• no traffic means no reliable health signal,
• a newly provisioned service is initially irreversible,
• verification is only as good as the contract's criterion.

What's next

• explicit access-grant policy for provisioned services,
• BigQuery billing reconciliation for measured cost,
• per-tenant service accounts and Workload Identity Federation.

Known limits

• During multi-revision traffic splits, Monitoring may return an ambiguous revision-level signal.
• Before/after measurement shows correlation, not causation.
• Only resources created with contracts are automation targets.
• A wrong recovery criterion produces wrong verification.
• Shared-resource cost attribution remains unresolved.

Built With

• adk
• cloud-run
• gemini
• google-adk
• google-cloud
• python
• vertex-ai