Sentinel

Inspiration

Every engineer knows the worst part of an outage isn't the fix, it's the triage. Production is down, twenty PRs merged today, and the whole team is grepping commits asking "which change caused this?" That window, alert to root cause, is where the real downtime lives.

We wanted to collapse it to zero. Not a dashboard that shows you more graphs, but a system that does the reasoning a senior engineer does during an incident: correlate the crash with what recently changed, and name the exact culprit. Autonomously.

What it does

Sentinel scores every pull request for risk at merge time, watches production, and when something crashes it automatically attributes the failure to the exact PR, function, and line, with a suggested fix, before the on-call has finished reading the Slack alert.

Code Guardian scores each merged PR (touches-critical-path, size, off-hours, author history) into a $0\text{–}1$ risk score.
A self-hosted collector runs on the customer's own servers and detects crashes/restart-loops with the stack trace.
A LangGraph + Gemini pipeline runs the triage correlation and writes a grounded root-cause report.

The whole thing is live on real infrastructure: merge a risky PR → it's flagged → it crashes in prod → incident + Slack alert → root cause, all with no human in the loop.

How we built it

The core design rule was "the database is the seam." The ingestion path and the AI agent are completely separate services that only ever meet at Postgres, neither calls the other. This let two of us build in parallel against seed data with zero coupling.

Ingestion (AWS): GitHub webhook + collector POST to an API Gateway → Lambda that writes Aurora Serverless v2 over the RDS Data API, under an IAM role with no static keys. The Lambda fetches each PR's changed files from the GitHub API so risk scoring has real data.
Agent (EC2): A FastAPI + LangGraph service polls Aurora, scores PRs, and runs the RCA pipeline using Gemini 2.5 Flash on Vertex AI.
Dashboard (Vercel): A read-only Next.js app that renders whatever the agent writes, instantly.

Code Guardian's score is a transparent weighted heuristic (no black box):

$$\text{score} = \mathrm{clamp}!\Big(\sum_i w_i\, f_i,\ 0,\ 1\Big), \quad w_{\text{critical_path}} = 0.52$$

The RCA was the interesting part. The naïve version ranks suspects by risk score, but the highest-scoring PR isn't always the culprit. So we made attribution evidence-driven: the agent matches the crash trace's file:line against each candidate PR's changed files, and a file↔trace match outranks a high score. Critically, if no candidate's files match the trace, it returns low confidence and refuses to guess, a confident wrong answer is worse than "I'm not sure."

Challenges we ran into

We hit a lot of real walls, and each one made the system more honest:

AWS Bedrock was unusable, the account's per-day token quota was clamped to $0$ across every model. We pivoted to Gemini on Vertex AI via application-default credentials.
Vercel froze our fire-and-forget calls. Serverless functions are killed after the response returns, so our async Slack alert never fired. Fix: await it.
An org SCP blocked public Lambda Function URLs (created fine, returned 403 on invoke). We re-fronted the Lambda with API Gateway, same payload shape, zero handler changes.
The RDS Data API won't cast text → enum on insert like a normal driver does, so every enum write needed an explicit ::enum cast.
The collector dropped the stack trace. Node writes crash traces to stderr, but our log capture only read stdout, so the first live incident arrived with no trace, and the RCA correctly fail-safed to "no strong match." Watching our own anti-hallucination guard fire in the demo was oddly satisfying; we fixed the collector and it nailed the culprit at 98% confidence.

What we learned

Heuristics are brittle in ways you only see on real data. Our critical-path check matched checkout/ as a prefix; the real repo nested it under src/services/checkout/, so the risky PR scored near zero until we switched to segment matching.
Security is an architecture decision, not a checkbox. We moved all writes off Vercel into AWS under an IAM role and gave the dashboard a read-only Postgres role — so a leaked frontend credential can't mutate the database.
The best AI feature we shipped was teaching it to say "I don't know." Grounding attribution in evidence and failing safe is what makes the output trustworthy.