Paramedic

Inspiration

When a service breaks at 3 a.m., users see error pages, the on-call engineer starts from zero, and the fix shipped under pressure is rarely proven. Code generation is solved; the first ten minutes of an outage aren't. Paramedic is a paramedic, not a surgeon: it stabilizes, protects users, and hands a human everything they need.

What it does

• First aid by fixed rules (seconds): an alert starts a GitLab pipeline. It measures the service, classifies the incident (bad deploy, spike, attack, unknown), rolls back or turns on a fair waiting room with an honest wait estimate ("100 ahead, 8–12 min"), and opens an incident issue with the evidence.
• GitLab Duo Agent Platform flow: triage reads the evidence and commits, then routes:
  • fixer writes a regression test first and opens a fix MR;
  • scaler opens a budget-priced capacity MR;
  • guard handles attacks without scaling;
  • escalate wakes a human with what is and isn't known.
• Proof-carrying fixes: repro:gate only accepts a fix if its test fails on main and passes on the branch. With no reproduction there's no fix: the agent escalates.
• Semantic canary: the canary books real tickets and checks business invariants. A release that returns HTTP 201 but oversells seats is caught and sent back.
• Guardrails in code:
  • policy.yml sets autonomy per action, with a budget cap and a kill switch;
  • agents earn autonomy after 5 human-graded clean incidents and lose it after 1 bad one;
  • CODEOWNERS means only humans change the policy.

How we built it

• App: Python/FastAPI (TicketDrop), with a Firestore store designed for Cloud Run.
• GitLab CI/CD:
  • validate, unit tests, SAST, secret detection, dependency scanning, image build to the container registry;
  • manual drills that start real builds as servers inside CI jobs;
  • canary, promote and rollback jobs for Cloud Run (they switch on when GCP variables are set).
• GitLab features: feature flags, pipeline triggers and schedules, CODEOWNERS.
• Two Duo flows in .gitlab/duo/flows/: incident-response and verify-and-record.
• Principle: the AI proposes, rules decide.

Challenges we ran into

• Against the Firestore emulator, 25 simultaneous bookings for one show crashed lock-based transactions. We switched to optimistic writes: two instances selling the same show went from 64 s to 0.12 s, with no overselling.
• Our first wait estimate predicted ~170 s for ~2.5 s waits. We rebuilt it on Little's law: 74% of waits landed inside the predicted range (local test, 40 concurrent users).
• A pipeline rule would have silently skipped our tests on every push. We now test our own pipeline rules for each pipeline type.
• Our cloud trial couldn't be activated in time, so every drill runs real builds inside GitLab CI instead of a live Cloud Run service.

Accomplishments that we're proud of

• drill:oversell: health check green, p95 16.8 ms, every request 201, and the semantic canary still caught "oversold 7/5" and turned the job red on GitLab CI.
• drill:incident on GitLab CI measured the bad change at 51 DB calls per request vs 2 for the last good build (215 ms vs 16.2 ms p95). It classified the incident as a bad deploy, turned on the waiting-room feature flag, and opened incident issue #2 by itself.
• 73 automated tests, and a green pipeline including all three GitLab security scanners.

What we learned

Agents are most trustworthy when they don't have the last word. LLMs reason and write; deterministic checks decide. That makes it safe to let them act at 3 a.m.

What's next for Paramedic

• Go live on Cloud Run with the Cloud Monitoring alert → first-aid trigger (code is ready).
• Shared queue state across instances.
• More incident classes.

Built With

• docker
• fastapi
• firestore
• gitlab-ci
• gitlab-duo-agent-platform
• gitlab-feature-flags
• google-cloud-run
• k6
• python