Inspiration
Getting paged at 3 a.m. is the worst time to make a production decision. You're half awake, the dashboards are noisy, and the fastest fix is usually "roll back the last deploy." AI assistants can already suggest that. The scary part is when they can also do it because someone said "yes" too quickly.
We wanted a voice incident agent you could talk to hands-free, one that does the investigation for you but structurally can't take a destructive action on vibes. Talking should get you answers. It should never, on its own, get you a rollback.
What it does
OnCall is an MCP server and operator workspace for the engineer on call.
- Investigates in parallel. It pulls error rates, recent deployments, blast radius and the runbook, then lets Amazon Nova Pro (Bedrock Converse with tool use) dig into logs or past incidents if it needs to.
- Correlates, never guesses. A deploy-vs-spike correlator points to the release that lines up with the error spike. It reports timing, not causation.
- Gates every rollback on evidence. The server refuses to even propose a rollback unless fresh metrics show the service is degraded right now. Unknown health is never treated as healthy.
- Needs a real approval. A proposed rollback is bound to one exact release, expires in 120 seconds, runs once, and needs a 6-digit code sent to your pager. "Yes", "approve" and "do it" are never approvals. Codes are stored hashed and lock after 3 wrong tries.
- Earns recovery. Verification ignores metrics from before the rollback and waits for a fresh window. A resolved alert never closes an incident by itself.
- Remembers. Conversations persist across restarts, so "what about the deploy before that?" works.
You can drive it from Alexa+ (or any MCP client), from the web workspace by voice or text, or just watch the public status page.
How we built it
The safety rules live in one place: a single safety kernel (oncall_core.py) that owns incidents, evidence, approvals, voice codes and verification. Every surface goes through it, including the web UI, the Bedrock agent and every MCP client. No client can skip it.
- MCP server (Python
mcpSDK) exposes 18 typed tools. 8 are read-only and the rest are workflow steps:propose_action,request_voice_approval,approve_with_voice_code,execute_action,check_recovery,verify_recovery,generate_reportand more. - Bedrock Nova Pro acts as the investigator. It only gets read-only tools plus
propose_action, so it can't approve or execute anything. If Bedrock is unavailable, a deterministic investigator returns the same structure of answer. - OAuth 2.1 authorization server for Alexa+ account linking, following the MCP Toolkit contract: Protected Resource Metadata (RFC 9728), PKCE
S256, aclient_credentialsdiscovery token, then authorization code + PKCE with a static client. - A real production stack to break: a payments service, a release controller with a deploy ledger and rollback webhook, a traffic generator, Prometheus and Alertmanager. Alerts hit
/api/alerts, open an incident and page the operator. - Frontend: a React workspace, an Alexa+ simulator and a status page, bundled with esbuild.
- AWS: the whole stack runs on one EC2
t4g.micro(Graviton, Ubuntu 24.04) behind Caddy with automatic HTTPS. Bedrock access uses an instance role scoped tobedrock:InvokeModelon Nova Pro, so no AWS keys live on the server. AWS Budgets alerts at 50/80/100%. Provision, ship and teardown are each one script.
Approval risk is layered. Here a code is 6 digits, valid for 120 seconds, with at most 3 attempts, so the chance of guessing one is
$$ P(\text{guess}) = \frac{3}{10^6} = 3 \times 10^{-6} $$
per proposed action, and each code works only for one release on one action.
Challenges we ran into
- Alexa+ access. The Alexa AI CLI and simulator turned out to be limited to select partners (the public npm package returns 404). We built the real, spec-compliant MCP server and OAuth flow anyway, and added a faithful Alexa+ simulator so judges can try the full voice experience.
- Auth details that break linking without any error. Alexa+ needs a static client (no dynamic registration), reads
/.well-known/oauth-protected-resourceat the root, and expects a401withoutWWW-Authenticate, which is the opposite of generic MCP behaviour. Our first fully spec-compliant server would have failed linking. - The 500 ms response budget vs. a 30-second recovery watch. We split verification into a non-blocking
check_recoverythat starts the watch in the background and reports progress on the next call. - Least-privilege Bedrock IAM. Inference profiles need both the profile ARN and the foundation-model ARN in every region they route to. Figuring out the minimal policy took trial and error.
Accomplishments that we're proud of
- An approval model that holds no matter which client is talking: web, Bedrock or Alexa+.
- "No evidence, no rollback" enforced on the server, not suggested in a prompt.
- One-command deploy to AWS with every secret generated on the box and never printed.
What we learned
- Put safety in the server, not in the prompt. LLMs are great investigators and should never be the thing holding the key.
- "Unknown" is a state of its own. Treating missing metrics as healthy is how incidents get closed too early.
- Voice UX needs async patterns. Anything slower than a sentence has to become "started, ask me again."
What's next for OnCall
- Real Alexa+ account linking and certification once toolkit access opens up.
- Multi-operator identity and on-call rotations.
- Postgres and multi-host support, plus reconciliation of interrupted actions.
- More executors (Kubernetes, ECS, feature-flag rollbacks).
Log in or sign up for Devpost to join the conversation.