GAFFER

One Gemini-fused release-night verdict, human-approved and replayable in ClickHouse.

Comment

Inspiration

A global streaming premiere can fail in one territory while playback telemetry, infrastructure state and audience impact live in separate tools. Operations teams lose critical minutes reconciling dashboards and agreeing on what is actually happening. GAFFER turns that fragmented evidence into one accountable incident call.

What it does

GAFFER is a live release-night control room for media and entertainment operations. It detects a regional playback anomaly, gathers independent evidence, asks Gemini for a bounded causal verdict, calculates the affected audience and stages a safe response. Consequential actions remain blocked until a named human approves them. Every crew step, verdict and approval is written to a replayable ClickHouse ledger.

The fixed crew is:

  1. WATCH: Query ClickHouse for regional playback deviation.
  2. LIGHT: Gather supporting Grafana context plus ClickHouse error and subscriber evidence.
  3. CUT: Fuse the evidence into a schema-validated Gemini verdict.
  4. ROLL: Propose actions through deterministic policy gates.
  5. PRINT: Record the complete decision path in ClickHouse.

How we built it

The backend is FastAPI with a live WebSocket control room. The judged path uses Gemini 2.5 Flash through Google Agent Development Kit on Vertex AI Agent Engine. Cloud Run serves the public application and scales to zero. Secret Manager supplies runtime credentials. Cloud Billing alerts and one-instance limits keep the hackathon deployment cost bounded.

Playback telemetry is a reproducible synthetic corpus of one million events. No customer or personal data is used. Live mode performs real service calls and the separate ?mode=demo replay is always labeled.

Why ClickHouse is load-bearing

ClickHouse is the only partner track entered. The official mcp-clickhouse runtime performs every judged telemetry read used for anomaly detection, error breakdown and subscriber reach. Direct ClickHouse access is limited to controlled schema setup and parameterized ledger writes. The same ClickHouse ledger powers the filmstrip replay, so the partner service drives both the live decision and its audit receipt.

Grafana supplies supporting operational context and the one implemented external action: a human-approved dashboard annotation. It is not a second track.

Trust and human control

Gemini interprets bounded evidence but cannot choose the tool order, execute arbitrary SQL or bypass an approval gate. Deterministic code owns orchestration, significance thresholds and dispatch policy. The public approval endpoint requires a bearer token plus named attribution. Unimplemented actions never appear as dispatched.

What we proved

  • Public Cloud Run deployment with healthy Agent Engine status
  • Real official ClickHouse MCP queries against the synthetic release corpus
  • Schema-validated Gemini verdict from Vertex AI Agent Engine
  • Named Grafana approval and external annotation
  • Parameterized ClickHouse audit receipt and replay
  • 71 passing backend tests plus browser and media validation

How Codex helped

Codex helped audit the hackathon requirements, harden MCP response handling, repair structured ADK output, add policy and security tests, deploy the managed agent path, rebuild the responsive control room and challenge unsupported claims before the final demo.

Try GAFFER

On a cold start, leave the public tab open until the status reports live synthetic telemetry. Watch DACH cross the incident threshold while peer regions remain within baseline, inspect the vertex-agent-engine verdict source and replay the recorded crew cycle.

Known limitations

The telemetry is synthetic. Financial exposure uses clearly labeled planning assumptions. Grafana annotation is the only implemented external dispatcher. Rollback, configuration and public communication actions remain approval records only.

Built With

  • clickhouse-cloud
  • fastapi
  • gemini-2.5-flash
  • google-agent-development-kit
  • google-cloud-run
  • grafana-cloud
  • mcp-clickhouse
  • mcp-grafana
  • vertex-ai-agent-engine
  • websockets
Share this project:

Updates