Chronos

Inspiration

At 3:00 AM, data engineering teams around the world get paged for pipeline failures. Troubleshooting schema drifts, API timeouts, and corrupted syncs requires hours of manual investigation.

While the obvious modern response is "let an AI agent fix it," granting autonomous LLMs write access to production introduces a dangerous new failure mode: an agent could hallucinate, swallow a prompt injection, or issue a command that drops a production table or corrupts customer records. Prompt engineering alone cannot prevent this.

We were inspired to build Chronos to solve this dilemma: How can we harness autonomous multi-agent reasoning to automate incident remediation while making destructive production mutations structurally impossible by code?

What it does

Chronos is a governed incident-remediation control plane built for the enterprise. When a data pipeline fails:

Detection & Triage: A typed DetectionAgent analyzes error telemetry and maps it to a structured FailureClassification. Adversarial Debate: A DebateProposer and DebateAuditor agent pair engage in a multi-round debate. The Proposer drafts remediation steps, while the Auditor attacks edge cases to harden the plan. Deterministic Policy Enforcement: Proposals are transmitted over the Agent-to-Agent (A2A) protocol to an isolated Go Action Broker. The broker evaluates every request against a strict, versioned policy model. Structural T3 Block: Destructive actions (such as DELETE_DATA or ALTER_PRODUCTION_SCHEMA) are classified as Tier 3 ((\text{T3})). The Go broker contains zero execution paths for (\text{T3}) actions, making production deletion structurally unreachable: [ P(\text{Execution} \mid \text{Action} \in \text{T3}) = 0 ] Tamper-Evident Ledger: Every action and decision is appended to a Firestore-backed, hash-chained ledger where entry hash (H_i) satisfies: $$H_i = \text{SHA-256}(H_{i-1} \parallel \text{Actor} \parallel \text{Action} \parallel \text{Decision} \parallel \text{Seq}_i)$$ The chain integrity can be verified at any time via verify_chain().

How we built it

Chronos was designed as a hybrid Python + Go architecture deployed on Google Cloud Run:

Orchestrator (Python / Google ADK): Implements the state machine (RECEIVED (\to) CLASSIFIED (\to) DEBATING (\to) POLICY_REVIEW (\to) VERIFIED (\to) CLOSED). It uses Gemini 3.5 Flash on Vertex AI via GEAP Interactions API with strict Pydantic schemas. Inline Defense (Model Armor): Evaluates input telemetry before it reaches the LLMs, redacting PII (emails, JWTs, AWS keys) and quarantining prompt injections. Go Action Broker (Go 1.23 / A2A): A lightweight microservice enforcing zero-trust policy evaluation. It exposes A2A endpoints (/.well-known/agent.json and /a2a/v1/invoke). Governance Verification: A custom Go AST static analysis test walks the codebase AST at build time to verify that no execution handler exists for forbidden actions. Storage & Telemetry: Firestore (Native mode) for transaction-safe sequence numbering, Pub/Sub for incident ingress and dead-letter queues (DLQ), and OpenTelemetry for reasoning-chain spans.

Challenges we ran into

Enforcing Zero-Trust without Breaking Autonomy: Balancing agent reasoning freedom with hard policy boundaries required establishing a deterministic tier ranking function (f): [ f(\text{Proposal}) = \begin{cases} \text{T0 (Sandbox)}, & \text{if Reversible} \land \text{Impact} = $0 \ \text{T1 (Approval)}, & \text{if Reversible} \land \text{Impact} > $0 \ \text{T2 (High Risk)}, & \text{if Non-Reversible} \ \text{T3 (Blocked)}, & \text{if Action} \in {\text{DELETE_DATA}, \text{ALTER_SCHEMA}} \end{cases} ] Type-Safe Multi-Agent Debate Control: Preventing infinite debate loops between the Proposer and Auditor required implementing a deterministic state machine controller capping the interaction at (\max(\text{Rounds}) = 3). CORS and Cloud Run Edge Authentication: Ensuring seamless dashboard interactions while maintaining service-account security between the Orchestrator and Broker required configuring strict CORS preflight handlers and OIDC token verification

Accomplishments that we're proud of

Proven Structural Safety: Proving that (\text{T3}) destructive mutations are blocked by static code analysis and compiler guarantees, not just LLM prompts. Full GEAP Capability Integration: Wiring together GEAP Registry, Memory Bank, Model Armor, A2A Protocol, and OpenTelemetry in a single unified architecture. Instant Cold Starts & Scale-to-Zero: Optimizing container builds with multi-stage Dockerfiles so idle cost is $0/month.

What we learned

Separation of Reasoning and Policy is Essential: LLMs excel at creative proposal generation, but execution boundaries must be owned by deterministic, statically-typed code. Multi-Layer Defense in Depth: Guardrails must exist at 4 distinct layers: system prompts, Pydantic enums, JSON schema validations, and broker policy rules

What's next for Chronos

Expanding the Action Broker registry with pluggable WebAssembly (Wasm) policy modules. Integrating real-time anomaly detection triggers directly from Google Cloud Monitoring and BigQuery audit logs.

Built With

• a2a-protocol
• cloud-run
• css3
• docker
• fastapi
• firestore
• gemini-api
• golang
• google-adk
• google-cloud
• html5
• javascript
• model-armor
• opentelemetry
• pubsub
• pydantic
• python
• streamlit