Night Watch

What it does

Night Watch is the AI crew that works the night shift. At 2 AM at Meridian Freight — a simulated logistics operation with five services — one on-call engineer is asleep. Night Watch is awake: a five-agent fleet on a Google Agent Development Kit (ADK) graph that ingests alert webhooks, gathers evidence, diagnoses, executes one playbook action, verifies it against live numbers, and files a hash-chained postmortem. No human turn anywhere in the loop.

Inspiration

On-call engineering at night is a trust problem, not a talent problem. The failure mode isn't "nobody can fix it" — it's "nobody can safely act at 2 AM without making it worse." We wanted agents that behave like a good night crew: they act autonomously, they refuse to act when evidence doesn't back the alert, and everything they do is written down in a ledger that can't be quietly edited.

How we built it

• Five agents on a Google ADK graph — router, diagnostician, remediator, verifier, reporter. Deterministic nodes handle routing, policy and execution; Gemini 3.5 Flash does the three reasoning jobs (diagnosis, proposal, verification) behind strict JSON schemas.
• Gemma triage tier at the gateway — Gemma triages every inbound alert (severity, fault class, duplicate check) fail-open and audit-chained, before Gemini 3.5 Flash does the deep reasoning: two Google AI models, one crew.
• Runs on Google Cloud Run (europe-west1) as two services: the agent gateway (FastAPI) and a Meridian Freight simulator (Node.js) that exposes real telemetry (Prometheus metrics, logs, snapshots) and a control plane for faults and repairs.
• Tamper-evident audit — every agent action lands in a hash-chained ledger; /audit/verify recomputes the chain on demand.
• Model Armor screen — inbound alerts are pattern-screened; prompt-injection payloads get an instant verdict: block.
• Incident memory bank + idempotency ledger — kill the fleet mid-run, restart it, and the run resumes instead of double-executing repairs.

Built as a Gemini Enterprise Agent Platform component set. Night Watch deliberately implements every GEAP surface:

• Agent Registry — every agent is cataloged with declared identity and scopes; remediation actions draw from a closed, versioned registry of approved playbooks
• Agent Runtime — ADK 2 graph engine with checkpoints: kill the fleet mid-run and it resumes from the last completed node
• Memory Bank — persistent incident memory that survives sessions and feeds future diagnoses
• Agent Identity — per-agent credential scopes (telemetry:read, actions:propose, actions:execute, audit:write); zero-trust by construction
• Agent Gateway — unified ingest + run lifecycle + policy enforcement at the edge
• Model Armor — inline screening on every inbound webhook before any LLM turn (prompt-injection blocked at the gate)
• Agent Observability — hash-chained, tamper-evident audit ledger plus per-node OpenTelemetry spans

What we learned / Challenges

Live models improvise. Our first live GCP runs taught us that Gemini will invent its own JSON key names mid-crisis, so we pinned exact schemas into agent instructions — after which runs completed cleanly. The best moment came from an accident: we fired a jam alert before injecting the fault, and the fleet refused to act — "telemetry indicates normal operations… this is a false alarm" at 0.95 confidence. We kept that refusal as a feature. An agent that declines to act on uncorroborated claims is worth more than one that always acts.

Verified evidence (live, not staged)

• Health: provider: gemini, model: gemini-3.5-flash, triage tier active, audit chain verified
• Live remediation run on GCP: fire → triaged by Gemma → diagnosed (conf 1.0) → action executed → verified
• Injection trap: blocked at the gate with matched patterns
• 9/9 graded eval scenarios pass in CI, 25/25 unit and integration tests

Built With

• docker
• fastapi
• gemini-3.5-flash
• gemini-api
• google-adk
• google-cloud
• google-cloud-run
• json
• model-armor
• node.js
• observability
• prometheus
• python
• rest-api
• webhooks