Inspiration
When a payment system breaks, every minute costs money, and an on-call engineer gets paged to do the same detective work every time. We wanted an AI that could do that investigation, safely enough to trust with production.
What it does
OnCall AI is an AI on-call engineer. When our live payment API on AWS breaks (bad deploy, traffic surge, slow fraud check, or database leak), CloudWatch alarms page the agent. Gemini investigates real metrics, logs, and deploy history by calling tools, without being told what broke, and proposes one fix. A human approves, the fix runs through a locked-down AWS command, and the agent verifies recovery and writes the postmortem.
How we built it
Python, FastAPI, Postgres, nginx, and Docker on EC2, with Locust simulating shoppers. CloudWatch, EventBridge, and SQS detect problems and page the agent. Gemini uses function calling and returns validated JSON. Fixes run only through an SSM document and a narrow IAM role. A Streamlit dashboard and a demo storefront show it all live.
Challenges we ran into
Overloaded and rate-limited AI models at 3 AM (solved with a model fallback chain and backup rules), a fix that didn’t actually clear the alarm (redesigned as a circuit breaker), and keeping an AI that reads logs safe from manipulation (guardrails in IAM, not the prompt).
Accomplishments that we’re proud of
Gemini diagnosed 3 of 3 faults correctly. A bad deploy went from fault to verified recovery in about 100 seconds. When a fix didn’t work, the agent caught it and escalated instead of claiming success.
What we learned
The intelligence can live in the model, but the safety has to live outside it: in permissions, human approval, and verification.
What’s next for OnCall AI
Learning from past incidents, auto-approving low-risk fixes once proven, one-click deployment into any AWS account, and Bedrock so logs never leave the account.
Built With
- amazon-web-services
- docker
- fastapi
- gemini
- python
- streamlit
