Inspiration
When an ambulance is on its way, the emergency department usually learns what's coming from a rushed radio or phone call. The paramedic giving it has gloved hands, a patient to hold still, and no way to type. The numbers are the dangerous part: "90 over 60" misheard as "190 over 60" changes what the team prepares. Details that matter an hour later, like a blood thinner or an allergy, get lost between the scene and the resus bay.
ManageXR's brief for this track says AI wearables bring superpowers to frontline work. We wanted to see what that looks like for the person in medicine with the least free hands: a paramedic. Smart glasses fit the job. Voice is the only practical input, confirmation can come by ear, and the camera sees what the medic sees.
##What it does
Inbound turns a paramedic's spoken handover into a structured, sourced handover that the ER can read before the doors open.
- Talk, don't type. The medic says "inbound start", describes the patient, says "snap" to photograph an injury from their own point of view, and says "inbound stop". Commands work mid-sentence: "BP 190 over 90, snap the left forearm".
- Numbers are checked by ear. Inbound listens for blood pressure, heart rate, SpO₂ and respiratory rate. When two values conflict ("190 over 90, no, 90 over 60") or a number makes no sense, the glasses ask the medic to repeat it, then read back the value they log.
- The ER sees it live. A patient card opens on the ER dashboard the moment the medic starts, showing the transcript, vitals and photos as they arrive. After "inbound stop", AI writes an IMIST-AMBO handover with NEWS2 colour cues and a source for every statement, plus a checklist of what to ask the crew on arrival.
- The ER can ask back, hands-free. A doctor types or speaks a question. The glasses read it aloud and the medic answers by voice ("snap" adds a photo). The answer fills in the handover: new vitals are read back, and everything else is marked as coming from the crew.
- Paper when it's needed. One click turns the handover into a PDF to print or attach to the chart.
##How we built it
- Glasses and phone: a TypeScript MiniApp built on the Mentra MiniApp SDK. It runs on the medic's phone and talks to Mentra Live over Bluetooth. On-device speech recognition catches commands in partial results, so Inbound reacts before the medic finishes the sentence. A custom parser understands vitals the way medics say them ("one ninety over sixty", "sats ninety-eight percent", "no, ninety over sixty"), and the glasses' speaker does the read-back. Every upload goes through an ordered queue that retries until the server confirms it, so a dropped signal never loses a capture.
- Server: Python, FastAPI and SQLite. It joins the audio with ffmpeg and transcribes it with OpenAI's speech-to-text. GPT-4.1 with structured outputs writes the handover, citing the audio moment, photo or vital behind each statement. Vitals the medic confirmed by read-back always override the model. The server also relays the ER's questions and turns the medic's replies into handover updates.
- ER dashboard: React 19, Tailwind CSS 4 and Lucide, updating live over server-sent events. The browser generates the handover PDF with jsPDF. Dictation uses the browser's speech recognition, with a server-side transcription fallback.
- Optional scene video: the glasses upload video straight to the server through a Cloudflare tunnel. The tunnel only accepts uploads that carry a key for that capture.
##Challenges we ran into
- Venue Wi-Fi. Stanford's visitor network has a sign-in page, which left the glasses with no internet and no route to our laptop, so video uploads failed. We moved everything essential (audio, photos, transcript and vitals) onto the phone's connection and made video optional.
- Misheard numbers. Speech recognition turned "one ninety" into "one nine" and "sats" into "sass". Instead of guessing, Inbound treats a number that makes no sense as unclear and asks for it again. That's how the read-back loop became the core of the product.
- Commands hiding in conversation. "Start" and "stop" come up constantly in normal speech, and one test capture ran for 54 minutes. The commands became "inbound start" and "inbound stop".
- The glasses hearing themselves. When the glasses read the ER's question aloud, the microphone picked it up as the medic's answer. Inbound now ignores the microphone while the glasses speak, and strips their voice from replies.
- Hardware limits. Mentra Live can't take a photo while it records video, so with video on, each "snap" marks the moment and the still comes from the video. Cloud transcription also heard "snap the left forearm" as "splint the left forearm", so the phone's own transcript decides which words were commands.
##Accomplishments that we're proud of
- It works end to end on real Mentra Live glasses: the medic speaks, and the ER sees the patient within seconds.
- The read-back loop catches the exact error that makes radio handovers dangerous.
- The ER and the crew can talk hands-free, and the handover stays up to date with no typing on either side.
- The ER can trust the AI's handover: it follows the order ED staff asked for, every statement has a source, and confirmed numbers beat the model.
- 56 MiniApp tests and 33 server tests cover voice commands, vitals parsing, read-back, uploads and messaging.
##What we learned
- Designing without a screen. On display-less glasses the voice interface is the product. Confirmations have to be short, never talk over the medic, and only speak up when something needs attention.
- What ERs want from a handover. The IMIST-AMBO format and the NEWS2 early-warning score shaped the dashboard: the patient in one line, an ETA, and alerts only for what's abnormal.
- AI in medicine needs guardrails. Structured outputs, sources and human-confirmed values turn the AI's output into something a clinician can check, not just trust.
- Networks are the hardest part of wearables. The phone turned out to be the most reliable link between the glasses and the world.
##What's next for Inbound
- Scan medications and alert bracelets with the camera, adding medications, allergies and any blood thinner to the handover.
- Trauma-criteria alerts from confirmed vitals, so the ER can confirm a trauma-team activation back to the glasses.
- Hosting for real ambulances, with the server online so crews connect over cellular and the ER opens the dashboard anywhere.
- Fleet readiness for EMS services: battery, connectivity and app version for every pair of glasses, built to plug into device management platforms such as ManageXR.
- Production readiness before any real patient data: accounts, audit logs, encryption at rest, data-processing agreements and clinical validation.
Built With
- fastapi
- javascript
- mentra
- python
- shell
- typescript