Inspiration
When someone collapses, what happens in the first few minutes matters most, and the person standing closest is almost never trained. Help is on the way, but until it arrives a frightened bystander has to remember steps from a CPR class they may have taken years ago, if they took one at all.
We asked a simple question: what if your phone could see what you see, hear what you say, and calmly walk you through it?
While researching, we found the catch. The 2025 AHA guidelines rate cognitive aids for lay rescuers, mobile apps and voice prompts included, as "No Benefit." In studies, they delayed calling 911 and starting compressions by 30–70 seconds. A smart assistant that slows people down is worse than none at all. So our goal changed from "build an AI that helps" to "build an AI that can never be the reason someone waits."
What it does
HealthAR is an iPhone app (currently a simulation, run on a manikin and props) that guides a bystander through an emergency using voice, camera, and AR, then hands a one-screen report to EMS.
- Assess. Point the camera and say what's happening. Gemini Live watches and listens, and the app routes to the right flow.
- Cardiac arrest. Responsiveness and breathing checks, a 911 prompt, positioning, an AR ring for hand placement, a compression metronome, and AED steps with an AR arrow to the AED it found in the room.
- Possible heart attack, choking, broken bone. Guided flows with AR cues: the spot between the shoulder blades for back blows, above the navel for thrusts, and a pin on the injury.
- Handoff card. Built from the app's own event log, never from an AI transcript.
Everything works offline by tapping. The AI adds to the local steps and replaces none of them.
How we built it
Our core rule: the model reports, the engine decides.
The app runs two loops at different speeds:
- A fast loop on the phone. On-device pose detection and ARKit track the patient and draw every overlay in real time. The metronome, the protocol steps and the event log all live here, so they keep working with no network.
- A slow loop in the cloud. Gemini Live gets a camera frame every 5–10 seconds (more around key questions), plus the rescuer's voice. It works out what's going on and answers questions, but it can only report what it notices. A safety gate checks each report before the protocol engine decides whether anything changes.
RescueCore: a deterministic protocol engine in Swift with no network or media code. Every required step is a local, pre-rendered voice clip.RescueLive: two Gemini Live sessions, one for arrest and one for an awake patient, over WebSockets. The model doesn't speak commands. It calls tools likereport_finding, and every call goes through a SafetyGate.RescueVision: ARKit, on-device pose detection, and LiDAR depth that fits a glowing 3D torso to a lying patient. The model never supplies coordinates. Overlays come only from on-device tracking.RescueAudio: a local metronome, a speech gate (the protocol's own phrase always wins), and a watchdog for silence and runaway audio.- Sentinel: validators that check the AI's transcript as it streams. They cut off a bad line and play a spoken correction.
The safety gate enforces one rule above all others: urgency can only go up on its own. If $u_t$ is the urgency level at step $t$, then any event that comes from the model, a sensor, or a timer must leave $u_{t+1} \ge u_t$. The only thing that can lower urgency is a person tapping a button. In practice, that means the AI can push the rescuer toward CPR, but it can never talk them out of it.
Challenges we ran into
The cloud can't count compressions. Guidelines call for 100 to 120 compressions per minute, which works out to about $f = 2\,\text{Hz}$ at the top end. To follow a rhythm that fast, a camera has to sample at more than twice its frequency, so at least $f_s > 4$ frames per second. Gemini Live accepts at most one frame per second, and its replies come back one to two seconds late. That made it clear the timing had to live on the phone. Our metronome runs locally at 110 beats per minute, a click every $60/110 \approx 0.545$ seconds, and we added voice detection on the phone so the clicks don't leak into the AI's microphone.
Spoken words can't be taken back. With native speech-to-speech models, nothing can stop a hallucination before someone hears it. We stopped saying the Sentinel "prevents" errors and built it to contain them in layers: a gated speech channel, validators that cut the line and correct it out loud, and an engine that ignores anything the AI says about what to do.
The AI hears everyone. It hears the patient, the bystander, and a dispatcher on speakerphone. A patient saying "I can't breathe" while talking is not a collapse. A spoken "yes, they're breathing" must never end CPR. We wrote eval sets (triage lines, questions, the app's own lines played back at it) and measured them against gemini-3.8-live. For example, triage routing was correct on 20/20 held-out lines with a median of 2.5 s from end of speech to report.
Wording is safety. Every condition is shown as "possible X. Not a diagnosis." There is exactly one approved aspirin line, and only a person's "No" to the risk check can reach it. The module loader refuses any scenario that breaks these rules.
911. iOS has no public way for an app to place a 911 call. In this build the "911 agent" is simulated, and the demo build can't contain a real emergency number.
Accomplishments that we're proud of
- It works with the AI turned off. Pull the network and every flow still runs by tapping, with the same voice clips and the same metronome.
- Safety rules enforced by tests, not just written down. Only a human can leave CPR, the AI can never lower urgency, and the demo build can't contain a real emergency number. Each rule has a test, and the module loader refuses any scenario that breaks one.
- AR that lands on a real body. On a LiDAR iPhone, we fit a glowing 3D torso to the patient's measured shoulders and hips, so the hand-placement ring and the AED arrow sit in the room instead of floating on the screen.
- Real evals for a live voice model. We built triage, question, and barge-in test sets against Gemini Live. It routed 20 of 20 held-out triage lines correctly and answered 30 of 30 held-out questions within 5 seconds.
- One calm voice. The app's own lines are rendered in the same Gemini voice the AI speaks with, so the rescuer hears a single guide instead of two.
What we learned
- Put the LLM where it's strong. Gemini is great at understanding a messy scene and a panicked voice. It's the wrong tool for timing, geometry, or final decisions, so those stay deterministic and on the device.
- Design for when it fails. Network drops, a stale frame, a wrong guess: each one has to leave the rescuer at least as well off as having no AI at all.
- Safety wording is engineering. Words like "possible," "simulated," and "not a diagnosis" are enforced in code and checked by tests.
- Time to first compression is the metric. Every feature had to justify itself against that number.
What's next for HealthAR
Clinician review of every protocol and phrase, real 911 integration (text-to-911, RapidSOS), on-device compression-rate feedback, and tests with real bystanders, measured by how fast they start compressions.
This is a simulation. It cannot call 911, it has no clinical validation, and it is not a medical device.
Built With
- apple-vision
- arkit
- avfoundation
- blender
- gemini-live-api
- ios
- numpy
- pytorch
- realitykit
- speech
- swift
- swiftui
- websockets
- xcode
- xcodegen