Release ops

Inspiration

Microdrama teams release short episodes across many languages on tight schedules. A release can look healthy in one market while a subtitle cue, dubbing asset, or latency regression makes another market unsafe. We wanted to build the missing control-room layer: a system that connects media validation, live observability, and incident response into one explainable release decision.

The key design question was not “Can an LLM make the decision?” It was “How can an LLM make the investigation faster without becoming an unsafe production authority?” That led us to give Gemini a meaningful diagnostic role while keeping policy, approvals, and writes in code.

What it does

ReleaseOps answers: Which markets are safe to release right now, and what evidence supports that answer?

It validates episode assets, publishes operational signals to Grafana Cloud, and uses the official Grafana MCP server to retrieve cited Prometheus and Loki evidence. Gemini proposes falsifiable hypotheses and orders the allowlisted diagnostic checks. Deterministic code then computes each market state and the release recommendation. A release manager sees the evidence ledger, reviews an exact payload hash, and approves a single idempotent Grafana IRM drill incident. After remediation, ReleaseOps publishes fresh telemetry and verifies recovery.

In the hosted demo, PT moves from BLOCKED to READY after the subtitle fix, while HI stays AT_RISK because its live evidence still warrants caution. That distinction is intentional: the system does not turn a partial fix into a blanket green light.

How we built it

We built a FastAPI service and React operator UI around a deterministic validation and policy core. FFmpeg/FFprobe validate media fixtures; Prometheus and Loki provide operational signals; Grafana Cloud provides Mimir, Loki, and IRM; and the pinned official Grafana MCP server is the only integration path for evidence reads and incident writes. Google ADK connects the service to Gemini 2.5 Flash on Vertex AI for the planner and narrative stages. Firestore stores workflow and run state, while Cloud Run, Cloud Build, Artifact Registry, and Secret Manager provide the hosted deployment boundary.

We also built a claim-to-evidence ledger: every material model claim must reference returned evidence IDs and citations. Approval tokens expire and are bound to the exact incident payload hash, and duplicate writes are idempotent.

Challenges we ran into

The hardest challenge was integrating a real incident workflow without allowing an agent to silently mutate production state. We had to separate model suggestions from code-owned truth, make approvals tamper-evident, and preserve useful failure visibility when an integration is unavailable.

We also encountered a tenant-level Grafana IRM issue on an earlier stack. Instead of hiding that failure, we isolated the deployment on a dedicated Grafana Cloud organization and re-ran the full hosted investigation and recovery gates with fresh telemetry.

Accomplishments that we're proud of

• Gave Gemini a real, bounded diagnostic job instead of a cosmetic chat interface.
• Connected the full loop: asset validation → live evidence → incident → remediation → recovery.
• Passed the 20-scenario golden evaluation and adversarial safety checks locally.
• Recorded 0 writes from 100 invalid approval attempts and 0 duplicate incidents across 20 retries.
• Proved hosted recovery on Grafana Cloud: PT became READY while HI correctly remained AT_RISK.
• Made every important decision inspectable through evidence IDs, citations, payload hashes, and deep links.

What we learned

Agentic systems are strongest when the model has a narrow job with observable outputs and a clear authority boundary. The model is valuable for search strategy, prioritization, and explanation; deterministic code is better for policy, state transitions, authorization, and idempotency. We also learned that “live” must be demonstrated with fresh telemetry and a repeatable recovery test, not just asserted in a UI.

What's next for ReleaseOps

Next, we want to support real catalog and localization-provider integrations, richer subtitle and dubbing quality signals, and configurable release policies for different content teams. We would add longer-running incident workflows, approval delegation with stronger audit controls, and trend-based detection that warns about a market before it becomes blocked. The same evidence-first architecture can extend beyond microdramas to any multilingual media pipeline where release confidence must be earned, explained, and reversible.

Built With

• adk
• antigravity
• firebase
• google-cloud
• grafana