-
-
Dashboard
-
Description of failure and action plan
-
timeline of anomalies
-
list of all failures and decisions
Inspiration
We’ve all been there, debugging LLM pipelines is a nightmare. Traces are huge and messy, and finding out why a model hallucinated or why latency spiked takes hours. We wanted to build a tireless "first responder" AI that watches the logs, figures out what broke, and drafts a fix before we even wake up.
What it does
Auto_Ops acts like an automated SRE for your machine learning models. It connects to Arize Phoenix to monitor traffic. When things look weird (like an influx of errors or rate limits), it spins up a team of Gemini-powered agents to investigate the raw data, find the root cause, and suggest a patch on a dashboard. But crucially, it never executes the fix without a human clicking "Approve."
How we built it
We instrumented a mock FastAPI pipeline with OpenTelemetry to generate traffic and errors. Then we used Google Cloud's ADK (Agent Builder) and Gemini 2.0 to create a chain of three agents: Triage, Diagnosis, and Action. To let the agents talk to our observability stack, we wired them up to Arize Phoenix using the Model Context Protocol (MCP). Everything comes together in a slick React/Vite dashboard.
Challenges we ran into
One of the biggest headaches was dealing with raw OpenTelemetry JSON, it's incredibly dense, and getting our agents to zero in on the right metrics without getting confused took a lot of prompt engineering. We also kept accidentally blowing past our Gemini free-tier rate limits (just 5 requests a minute!) while testing, which forced us to write some clever retry logic and pacing scripts.
Accomplishments that we're proud of
We’re really proud of getting three separate AI agents to seamlessly share context and collaborate on a single problem. Bridging them to a live observability tool using MCP felt like magic. We also love our feedback loop: if a human rejects an agent's proposal, that rejection is logged as a dataset example to train future models.
What we learned
We learned that collecting data is just half the battle; the real value is making it actionable. Hooking up Gemini to Phoenix completely transformed static charts into actionable insights. We also got a deep dive into the MCP standard, which is definitely the future of tool-calling for LLMs.
What's next for Auto_Ops
We want to let the Action agent automatically open GitHub Pull Requests to fix system prompts once approved. We're also planning to add Slack integration so engineers can approve or reject patches directly from their team chat.
Log in or sign up for Devpost to join the conversation.