Dashbord
Dragon Lair
Neural Lab
Scribe
Governor
Chaos
Qoe Studio
Terminal

Inspiration

During massive live broadcasts, traditional Network Operations Center (NOC) teams face catastrophic alert fatigue. When an edge gateway fails or a transpacific fiber link severs, hundreds of asynchronous Grafana panels flash red at once. Monolithic AI chatbots cannot solve this: LLMs are too slow and expensive to evaluate continuous 50Hz telemetry, and they hallucinate under stress.

The industry needs a hybrid paradigm: Deterministic sub-millisecond mathematical signal filtering combined with grounded multi-agent AI reasoning and strict human governance.

KALMAN was engineered for the Google Cloud Agentic Cinema Hackathon (Grafana Labs Partner Track) to eliminate the "first hour of NOC panic" during live 4K global streams.

What it does

KALMAN partitions live broadcast reliability across specialized, non-collapsible operational tiers:

1. Multi-Agent Separation of Duties Matrix

Subsystem / Familiar	Architectural Core	Operational Domain	Trust & Safety Boundary
🐱 Watcher (Kalman)	Scalar 1D Kalman Filter (<0.05ms)	50Hz continuous variance tracking & anomaly gating	Mathematical Gate: Uses zero LLM tokens during nominal operation; immune to hallucinations.
⏱️ Forecaster	Rolling Linear Regression (( \frac{dx}{dt} ))	Time-to-Breach (TTB) predictive forecasting	Early Warning: Triggers preemptive alerts before buffer underruns impact viewer QoE.
🔍 Diagnostician	Google Gemini 1.5 Pro (Vertex AI ADK)	Multi-signal residual correlation & root cause	Reasoning Only: Cannot execute infrastructure actions; must cite Grafana query IDs.
🛡️ Governor (Warden)	Pydantic Policy Engine + Audit Hashes	Separation of duties & allowlist validation	Authority Gate: Strictly enforces Human-in-the-Loop (HITL) sign-offs for high-risk operations.
⚡ Actuator	Multi-CDN Traffic Steering API	Closed-loop execution of approved edge actions	Execution Only: Cannot propose actions; only executes Governor-authorized directives.
🦉 Scribe	Gemini 1.5 Flash + Grounding Engine	Immutable postmortem authoring & proof citation	Audit Only: Strictly flags any unverified assertion as `[UNVERIFIED]`.
🐉 Ignis (Caldera)	Procedural Canvas Flame Synthesizer	Destructive origin incinerator & manual failover	Operator Gated: Awakens only when a NOC supervisor signs off on destructive action.

2. Disaster Archetypes & Automated Remediation Matrix

Disaster Scenario	Target Subsystem	Signature Metric Breach	Automated Agent Remediation
🔴 CDN 5xx Meltdown	`us-west` Edge Gateway	5xx error rate spikes 35x; packet loss > 18%	Flux diverts 80% viewer traffic to `us-east`/`eu-west`; Gemini isolates upstream edge timeout.
🔵 Ingress Origin Bombing	Primary Transcoder Cluster	CPU saturation 99%; frame drop cascade	Warden halts traffic, prompts operator for failover to backup cluster; Ignis severs primary.
⚠️ Transcoder Skew	Video/Audio Muxer	AV sync drift > 85ms (EBU R68 violation)	Chrono adjusts presentation timestamps and triggers adaptive GOP realignment.
🌐 Backbone Sever	`ap-south` Transpacific Fiber	Complete link drop; buffer underruns	Multi-CDN BGP re-convergence; traffic shed to Singapore auxiliary edge.
📢 Loudness Storm	Audio Processing Chain	Audio peak > +14 LUFS (EBU R128 violation)	Autonomous DSP limiter clamping to restore compliance within -24 LKFS ±1.

How we built it

Core Technology Stack

Layer	Technology	Role & Implementation Details
Cloud Hosting	Google Cloud Run	Serverless, autoscaling container deployment with low-latency edge serving.
Reasoning Core	Google Gemini 1.5 Pro	Vertex AI Agent Development Kit (`google-genai`) for multi-signal root-cause synthesis.
Observability	Grafana Cloud MCP	Official `mcp-grafana` protocol client querying live PromQL metrics and Loki error logs.
Metrics Pipeline	Prometheus & OpenTelemetry	Exposes live metrics at `/metrics`; exports telemetry traces via OTel.
Backend Engine	FastAPI & Python 3.11+	Asynchronous ASGI event loop and Server-Sent Events (SSE) real-time telemetry stream.
Kinetic UI	Dark Editorial Linen & Canvas	Bespoke CSS variables, procedural Web Audio synthesizer, and zero heavy frameworks.

Mathematical Formulations & Statistical Gates

Rather than relying on arbitrary static thresholds, KALMAN executes continuous, mathematically rigorous residual innovation filtering:

$$y_k = z_k - H \hat{x}_{k|k-1}$$

$$S_k = H P_{k|k-1} H^T + R_k$$

$$\epsilon_k = y_k^T S_k^{-1} y_k$$

Concept	Formulation	Purpose in Telemetry Pipeline
State Prediction	( \hat{x}_{k\	k-1} = A \hat{x}_{k-1\
Error Covariance	( P_{k\	k-1} = A P_{k-1\
Measurement Residual	( y_k = z_k - H \hat{x}_{k\	k-1} )
Innovation Covariance	( S_k = H P_{k\	k-1} H^T + R_k )
Normalized Innovation Squared (NIS)	( \epsilon_k = y_k^T S_k^{-1} y_k )	Scalar anomaly metric evaluated against ( \chi^2 ) threshold (( \epsilon_k > 3.5 )).
Outlier State Freezing	If ( \epsilon_k > 9.0 \implies \hat{x}_{k\	k} = \hat{x}_{k\
Recovery Convergence	(	\epsilon_k

Challenges we ran into

Preventing Outage Normalization: In naive Kalman implementations, an ongoing severe outage shifts the internal baseline, causing the filter to treat the outage as normal. We solved this by freezing state covariance updates whenever ( \epsilon_k > 9.0 ), maintaining baseline awareness until recovery converges (( |\epsilon_k| < 2.0 )).
Grounded SRE Diagnostics Without Hallucination: SRE investigations cannot afford fabricated root causes. We coupled Gemini 1.5 Pro to the Grafana Cloud MCP gateway, requiring every diagnosis to cite verifiable PromQL and Loki query identifiers (#Q-8491, #Q-9214).
Balancing Autonomy with Operational Safety: High-speed automated failover is required for minor edge routing, but catastrophic actions (like severing primary origin clusters) must have human oversight. We built a dual-track policy engine that enforces Human-in-the-Loop (HITL) approval gates for destructive operations.

Accomplishments that we're proud of

Sub-Millisecond Zero-Token Core: 50Hz Kalman filtering executes in <0.05ms, consuming zero LLM tokens during nominal broadcast operations.
100% Cryptographic Query Citations: Postmortem reports link every diagnostic statement to immutable query proof hashes.
68/68 Automated Tests Passing (100%): Comprehensive pytest suite covering Kalman convergence, Gemini reasoning failovers, and BGP steering.
Living Engineering Deck: An interface with kinetic familiars that conveys complex telemetry intuitively without sacrificing technical rigor.

What we learned

Deterministic signal mathematics and generative AI are complementary: math handles high-frequency millisecond anomaly detection, while Gemini handles contextual reasoning and multi-signal synthesis.
The Model Context Protocol (MCP) significantly accelerates building production SRE tools by standardizing tool calling into live platforms like Grafana Cloud.

What's next for KALMAN

Real-Time Video Ingress Frame Inspection: Expanding Gemini's vision capabilities to detect macroblocking and compression artifacts in raw video streams.
Predictive BGP Peering Shifts: Preemptively redirecting edge traffic before transit provider degradation impacts audience QoE.
Open-Source MCP SRE Workflows: Sharing modular Gemini-to-Grafana MCP agent runbooks with the broader DevOps and SRE community.