Inspiration
It's premiere night for a streaming platform. Viewers are pouring in, and somewhere behind the scenes, an ops crew is watching dashboards, hoping nothing breaks. That moment — someone frantically clicking through Grafana panels under pressure — is what inspired Premiere Ops: an AI copilot that lets a crew member just ask about system health in plain English, instead of manually digging through charts during the most stressful hour of a launch.
What it does
Premiere Ops is a command-line AI agent that answers operational questions like:
- "Is the platform ready for tonight's premiere — any active alerts right now?"
- "How many dashboards exist and are all data sources healthy?"
- "Give me a quick ops status check before we go live tonight."
It answers only using real, live data it retrieves at runtime — nothing is hardcoded or hallucinated. If the data can't answer a question, it says so honestly instead of guessing.
How we built it
The agent is powered by Google Gemini (via the official google-genai
SDK) for reasoning, and connects to Grafana Cloud through its MCP
(Model Context Protocol) server for grounded, read-only access to real
observability data — data sources, alerts, incidents, and dashboards.
The flow works like this: a user asks a question → Gemini decides which Grafana tools it needs → those tools are called live through MCP → the real results are fed back to Gemini → Gemini reasons over the actual data and responds in plain English, citing which tools it consulted.
We built and iterated on this in Replit, using Python throughout.
Challenges we ran into
This was a first hackathon for one of us, and nearly every layer of the stack threw a real debugging challenge:
- Mismatched environment variable names between Grafana's service account token and what the code expected
- Grafana service account permissions defaulting too narrowly (403 errors) until the role was corrected
- Hitting Gemini's free-tier daily quota mid-testing, and discovering that some older model names are no longer available to new accounts
- Tuning how many "tool rounds" Gemini is allowed before it must answer, so it investigates thoroughly without looping indefinitely
Working through each of these — one real error message at a time — ended up being as valuable as building the feature itself.
What we learned
How MCP actually works as a bridge between an LLM and a real external service, how Grafana Cloud's permission and datasource model is structured, and just how much of "building an agent" is really about grounding it in real tool calls rather than letting a model guess.
Scope note
This proof-of-concept currently queries Grafana Cloud's own account and usage metrics to simulate a live platform-health signal. In a production deployment, the same agent would connect to a real streaming platform's Grafana instance, using the same architecture.
Built With
- ai-agent
- cli
- google-cloud
- google-gemini
- google-genai
- grafana
- grafana-cloud
- llm
- mcp
- model-context-protocol
- observability
- prometheus
- python
- replit