ControlGraph Canary

• 

  Controlgraph Architecture

Inspiration

I built ControlGraph Canary for one failure case: an operator approves and queues a rollout, then withdraws approval before the worker runs. The task still has a valid identity and signature, but no longer has permission to change Cloud Run traffic.

What it does

ControlGraph rechecks authority as the executor's final authorization step, immediately before its fixed adapter changes traffic. Each action is tied to one rollout, exact approval epoch, target, traffic plan, request, and expiry. A stale, future, missing, or unreadable epoch stops the action before the adapter runs.

The workflow captures the stable revision, sends 10% of traffic to a canary, checks fixed health rules, and either promotes the canary or restores the stable revision. A separate read-only verifier checks the result.

After a stale-authority denial, an authenticated operator can start an investigation. Gemini 3.5 Flash, coordinated by Google ADK, reads six fixed evidence sources: the rollout, target, health results, receipt, event timeline, and independent verification. It returns a validated, cited explanation and a bounded next-step suggestion for operator review. Gemini cannot start the investigation, approve authority, decide health, dispatch recovery, or change traffic.

How I built it

I separated the deterministic safety path from the AI investigator. The backend uses Python, FastAPI, and Pydantic. Cloud Run hosts the services; Firestore stores authority and receipts; Cloud Tasks delivers work; Cloud KMS signs capabilities and selected evidence; Cloud Monitoring supplies health data. Gemini runs on Vertex AI through Google ADK. The console uses React and TypeScript, and Terraform defines the cloud setup.

I started ControlGraph during the hackathon. It adapts disclosed patterns from my earlier RECONCILE work. The provenance register records the details.

Challenges I ran into

The hardest part was separating delivery from current authority. Exact duplicates had to be safe. If Cloud Run's response is lost, ControlGraph records AMBIGUOUS, avoids a blind retry, and checks traffic directly.

Accomplishments I'm proud of

The recorded reference run starts with a 90/10 canary. Promotion is queued under epoch N, authority moves to N+1, and the stale work is denied with DENIED / EPOCH_MISMATCH. Traffic stays at 90/10. After the operator confirms recovery, new epoch-N+1 authority returns traffic to 100% on the stable revision.

The Verified Replay lets judges inspect this credential-free recorded evidence. It is not a live control surface.

What I learned

Queue delivery and current approval are separate facts. Gemini investigates the evidence; deterministic services decide what production should do.

What's next

The current version covers one Cloud Run canary in one project and region. I want to support more rollout patterns and targets while keeping Gemini outside the mutation path.

Built With

• cloud-kms
• cloud-monitoring
• cloud-tasks
• docker
• fastapi
• firestore
• gemini-3.5-flash
• google-agent-development-kit-(adk)
• google-cloud-run
• pydantic
• python-3.12
• react
• terraform
• typescript
• vertex-ai
• vite