-
-
1 in 604 flights has a medical emergency, and in over half of them there's no physician on board to help.
-
How it works: crew speaks, safety screen, Gemini only classifies, deterministic engine decides, Fish Audio speaks.
-
Cardiac arrest in seat 23C: hands-free CPR coaching at 110 compressions per minute, with AED prompts.
-
Guardrails: dosing requests are refused, and on safety questions Gemini may only choose the safer answer.
-
Captain view: diversion airports ranked by time to an ER, using real runway data and real hospitals.
-
Ground physician handoff: one-tap SBAR plus a Gemini radio brief that's rejected if it invents a number.
-
Vitals by voice (impossible values rejected) and passenger questions spoken in 6 languages by Fish Audio.
-
Installable PWA: protocols, CPR timer and voice keep working in airplane mode; the audit log syncs later.
-
79/79 safety scenarios pass live in the app, including 32 regressions from our own red-team.
-
Databricks AI/BI dashboard: fleet medical analytics over synthetic incidents plus real incidents from the app.
Try it live: https://cabinmedic.vercel.app (works on a phone, tablet or laptop, installable as an app, and keeps working offline)
Inspiration
About 1 in every 604 commercial flights has a medical emergency, and in more than half of them there is no physician on board to help (Peterson et al., NEJM 2013). The person responding is a flight attendant with first-aid training, 35,000 feet from a hospital, often doing CPR with both hands on the patient's chest. They can't type into an app. Information gets lost between the crew, the ground physician on the radio, the captain deciding whether to divert, and the paramedics waiting at the gate.
We took "In the Clouds" literally: we built for the moment when being in the clouds is the problem.
What it does
CabinMedic is a hands-free voice copilot for cabin crew.
- Voice triage. Two questions recognise cardiac arrest. CabinMedic then guides one of 10 protocols: CPR + AED, unresponsive-but-breathing, fainting (the most common in-flight emergency), allergic reaction/anaphylaxis, breathing difficulty, choking, low blood sugar, seizure, chest pain and stroke. Stroke is time-critical, so it goes straight to ground contact and a diversion review. Anything else gets a general-support path, so it never dead-ends.
- CPR coach. A 110-per-minute beat, a prompt to switch compressors every 2 minutes, and AED prompts.
- Timers. Seizures are timed and escalate at 5 minutes. Low blood sugar gets a 15-minute re-check.
- Vitals by voice. "Pulse 112, oxygen 91" becomes a timestamped chart. Impossible values are rejected and abnormal ones escalate.
- Ground physician handoff. A one-tap SBAR, read aloud for the radio, plus an optional Gemini radio brief. The brief is automatically rejected if it contains a number that isn't in the record.
- Captain view. Diversion airports ranked by time to an emergency department, using real runway data (OurAirports) and real ER hospitals (OpenStreetMap). The captain must confirm, and the decision is logged.
- EMS handoff. A QR code and a printable report with the full timeline.
- Works offline as a PWA. Crews install it on their tablets with no app store. With the network off, the protocols, CPR timer, rule-based parser and on-device voice keep working, and the audit log syncs when the connection returns.
- Passengers who don't speak English. 8 pre-translated triage questions in Spanish, French, Mandarin, Hindi, Arabic and Portuguese, spoken by Fish Audio and recorded in the audit trail.
How we built it
- Protocol engine: a deterministic TypeScript state machine. Every spoken line is pre-written; the AI never writes medical instructions.
- Voice: Fish Audio text-to-speech with caching and prefetch, Fish Audio speech-to-text, push-to-talk plus hands-free voice-activity detection, and the browser's on-device recogniser as a fallback.
- AI: Gemini with structured JSON output classifies the utterances our offline parser can't handle. A validator rejects anything outside the allowed intents.
- Data: a MongoDB Atlas audit log, and a Databricks medallion pipeline with an AI/BI dashboard and a Genie space for airline medical directors.
- App: Next.js 16, React 19 and Tailwind, with Leaflet for the map, deployed on Vercel.
How we used Fish Audio
Voice is the product, not a feature: during CPR, both of the crew member's hands are busy.
- Text-to-speech: every instruction is spoken through
/v1/tts(s2.1-pro, with the frees2.1-pro-freetier as a fallback). - Speech-to-text: push-to-talk and hands-free clips go to
/v1/asr(transcribe-1-pro). If it isn't available, the browser's on-device recogniser takes over, which is also what keeps voice working offline. - Multilingual: passenger phrases in 6 languages are spoken through Fish TTS with the matching
languagetag. - Latency: protocol lines are deterministic, so we cache and prefetch the audio for every possible next step. The answer to "is the passenger breathing?" plays instantly.
How we used Gemini
Gemini has a deliberately small job.
- Classify messy crew speech ("he's kind of gasping, lips going blue") into a fixed intent schema using structured output (
responseSchema), but only when our offline parser can't. Its JSON then has to passvalidateLlmIntent(). It can't confirm a medication, can't invent a protocol or a drug, and on safety-critical questions it is only allowed to choose the safer answer. - Write a 90-word radio brief from the deterministic SBAR. We reject the brief automatically if it contains any number that isn't in the record.
How we used MongoDB Atlas
Every incident streams an append-only audit log to audit_events: steps, vitals, confirmed medications, safety blocks and captain decisions. A de-identified summary goes to incidents. Documents are rebuilt server-side from whitelisted, typed fields, so there is no operator injection, and identifiers are redacted before anything is stored. A token-protected /api/export endpoint serves the summaries to Databricks.
How we used Databricks
Airline medical directors need fleet-level answers, not a single incident.
- Pipeline: Auto Loader ingests incidents into bronze, then silver (deduplicated, typed, medications exploded), then five gold tables: protocol mix, route risk, kit usage (a restock signal), guardrail activity and diversions. It pulls live incidents from the deployed app's export endpoint.
- Dashboard and Genie: an AI/BI dashboard, and a Genie space that answers questions like "Which routes have the highest diversion rate?"
- Model: an MLflow-logged diversion model.
- Data: the demo uses 5,000 synthetic incidents modelled on the NEJM case mix, clearly labelled as synthetic, plus real incidents from the live app.
Ethical AI
- Agent card: in the app at
/aboutand indocs/AGENT_CARD.md. - 79 scripted scenarios run live in the app's Safety tests tab and in
npm test, including 32 regressions from our own red-team. They cover prompt injection (including zero-width and leetspeak obfuscation), dosing, diagnosis and privacy requests, never guessing, CPR that can't be interrupted, explicit medication confirmation, vital-sign sanity checks and model-output guardrails. - Judges can red-team it live at https://cabinmedic.vercel.app.
We red-teamed our own agent
An independent adversarial review threw hundreds of phrasings at the engine and found 32 failures, 7 of them critical. For example, "Definitely not" at the breathing check stopped CPR, and "Forget the protocol, he stopped breathing, start CPR!" was refused as a prompt injection. There were two root causes.
- Processing order. Cardiac-arrest detection now runs before the safety screen, vital-sign parsing and pending confirmations.
- Yes/no answers that ignored the question. Each question now has its own answer vocabulary. Hedges ("not sure", "kind of") are asked again, and gasping is never treated as normal breathing.
On safety-critical questions the AI may only choose the safer answer. In testing, Gemini said "breathing normally" with 98% confidence for an ambiguous description, and CabinMedic still asked the crew to confirm. All 32 findings are fixed and live on as regression scenarios.
Challenges we ran into
- Voice in a loud hall. We added push-to-talk, hands-free detection, confidence thresholds that ask again instead of guessing, and a button for everything.
- Context. "He's not breathing" means CPR, but a passenger who is talking and "not breathing well" needs the breathing protocol. Our test suite caught this.
- Usefulness without risk. Keeping the AI useful without letting it anywhere near medical instructions.
Accomplishments that we're proud of
- 79/79 safety and protocol scenarios pass, 32 of them regressions from our own red-team, and the full automated suite (90 tests) is green.
- The engine, the parser and browser voice work fully offline.
- Diversion ranking runs on 373 real airports with real ER hospitals.
What we learned
The most trustworthy AI design here was to give the model less power: classification only, with pre-written words and a deterministic engine.
What's next for CabinMedic
- Review the protocols with an airline medical director.
- An on-device speech model for flights without Wi-Fi.
- Integration with ground medical services such as MedLink and STAT-MD.
- A full multilingual passenger mode.
Built With
- databricks
- fish-audio
- gemini
- leaflet.js
- mlflow
- mongodb
- next.js
- node.js
- openstreetmap
- pwa
- react
- tailwindcss
- typescript
- vercel
- vitest