Aidra

Aidra - Turning bystanders into first responders with real-time AR guidance, AI-driven pulse detection, and hands-free voice commands — so no one has to face an emergency alone.

Comment

Aidra — Hackathon Submission

Inspiration

Only 32% of cardiac arrest victims receive bystander CPR. With average EMS response times of 8–12 minutes, the gap between when an emergency happens and when professional help arrives is where lives are lost — or saved. Studies show that immediate bystander action leads to 3x higher survival rates, but most people freeze because they simply don't know what to do.

We asked ourselves: what if your phone could be the calm, knowledgeable voice walking you through it — like having a 911 dispatcher in your pocket, but one that can actually see what you see? That's why we built Aidra. We wanted to bridge the emergency response gap using the tools people already carry with them every day — turning anyone into a capable first responder.

What it does

Aidra is a mobile-first emergency response app that uses AI-powered augmented reality and a hands-free voice assistant to guide untrained bystanders through life-saving medical procedures in real time — in under 15 seconds from launch to guidance.

Point your camera at someone who needs help, and Aidra will:

  • Guided CPR — AR sternum targeting with ±2mm accuracy, real-time depth feedback (5–6cm), and a built-in rhythm metronome locked to the medically recommended 100–120 BPM. Perfect compression placement, every time.
  • Pulse Assessment — Computer vision guides precise finger placement for pulse detection across three positions (carotid, brachial, radial) with a 10-second timer-based assessment protocol, visual guidance, and audio prompts.
  • Airway Management — Step-by-step head-tilt/chin-lift guidance with AR overlays ensuring proper head-tilt angle, chin-lift positioning, and obstruction checks.
  • Seizure Safety — Real-time guidance for head protection, body positioning, and safety measures during seizure events.
  • Hands-Free Voice Interface — Stress-aware natural language processing so you never have to look away from the person you're helping. Just talk, and Respond XR listens and responds with calm, specific instructions.
  • Real-Time Feedback — Color-coded AR overlays (green = correct, yellow = adjust, red = reposition) paired with haptic cues, dynamic speed control, and a beat-match algorithm so you feel when you're doing it right.

The entire flow is designed around a simple pipeline: Emergency Occurs → AI Analyzes → AR Guidance → Life Saved.

How we built it

Aidra is built with React Native and Expo, targeting both iOS and Android from a single codebase. Here's the stack:

  • Gemini Vision API powers the core computer vision pipeline — analyzing the camera feed in real time to detect body landmarks, identify hand positions, and assess whether procedures are being performed correctly
  • Gemini Flash serves as the medical dispatcher brain, taking raw visual analysis and generating clear, actionable voice guidance
  • Claude AI handles medical reasoning and analysis, providing clinically-informed protocol selection and contextual guidance
  • Google Cloud Speech-to-Text enables voice command recognition for fully hands-free operation
  • Expo Camera captures the live feed, while Expo Speech and Expo Haptics deliver multi-sensory feedback (voice instructions + vibrations)
  • Custom AR overlay system built from scratch — we render detection boxes, target markers, and guidance indicators directly on top of the camera view using absolute positioning and real-time coordinate mapping from AI-returned normalized coordinates
  • A Medical Dispatcher architecture orchestrates the full pipeline: Camera Input → AI Vision Analysis → Medical AI Reasoning → AR Guidance Output

Challenges we ran into

  • Real-time performance was brutal. Running vision AI analysis on every camera frame would instantly overload the device and the API. We had to implement aggressive throttling (capped at ~10 FPS), lower-resolution captures, rate limiting, and frame skipping to keep the experience smooth without sacrificing the ±2mm accuracy we needed.
  • Coordinate mapping from AI to AR. Gemini returns normalized coordinates for detections, but translating those into pixel-accurate AR overlays on different screen sizes and camera aspect ratios required significant math and calibration across devices.
  • Hands-free UX in high-stress scenarios. Designing an interface that works when someone's hands are literally on a person's chest doing compressions — every interaction had to be voice-driven or automatic. We went through several iterations of the voice assistant flow to make it feel natural under stress.
  • Medical accuracy vs. speed. We had to balance giving detailed, medically sound instructions with keeping guidance short and urgent enough for a real emergency. Too much information is just as dangerous as too little.
  • Camera + audio conflicts on mobile. Managing simultaneous camera capture, audio recording (for voice commands), speech synthesis (for instructions), and haptic feedback across iOS and Android required careful audio session management and platform-specific tuning.

Accomplishments that we're proud of

  • ±2mm AR accuracy for CPR hand placement — the full pipeline from camera capture to AI analysis to on-screen guidance overlays works in real time, within seconds
  • The CPR metronome with haptic feedback and beat-match algorithm keeps users at the medically recommended 100–120 BPM compression rate, even if they've never done CPR before
  • Pulse detection across three positions (carotid, brachial, radial) with visual guidance for finger placement and timer-based assessment — something we haven't seen in any other consumer app
  • The voice assistant genuinely feels like talking to a real dispatcher — stress-aware, natural language, calm and specific
  • Under 15 seconds from app launch to active guidance — because in cardiac arrest, brain damage begins in 4–6 minutes
  • It runs on a phone. No special hardware, no headsets, no training. Just open the app and it walks you through it

What we learned

  • AI vision APIs are powerful but need careful orchestration. Raw Gemini output is impressive, but turning it into actionable, real-time AR guidance with ±2mm precision requires a thoughtful pipeline with proper error handling, fallbacks, and rate management.
  • Emergency UX is fundamentally different from regular UX. Every design decision had to be filtered through the lens of "what if the user is panicking?" — bigger buttons, fewer choices, louder feedback, no dead ends. Stress-aware design isn't a feature, it's the foundation.
  • Multi-modal feedback is essential. Voice alone isn't enough. Haptics alone isn't enough. The combination of visual AR overlays + spoken instructions + vibration feedback creates a guidance experience that actually cuts through the chaos of an emergency.
  • The 32% stat is personal. Learning that less than a third of cardiac arrest victims get bystander CPR — and that immediate action triples survival — made this feel like more than a hackathon project. The technology exists to close this gap. Someone just needed to build it.

What's next for Aidra

  • On-device AI models to eliminate API latency and enable fully offline operation — because emergencies don't wait for Wi-Fi
  • Expanded procedure library covering choking (Heimlich maneuver), bleeding control (tourniquet guidance), burn treatment, and allergic reactions
  • Integration with emergency services — automatic location sharing and real-time status updates to 911 dispatchers while the user performs first aid
  • Wearable support (Apple Watch, smart glasses) for even more hands-free operation
  • Multi-language support so the app can guide anyone, anywhere, in their native language
  • Community training mode — a practice mode with a CPR mannequin that scores your technique and helps you build muscle memory before an emergency ever happens

Built With

  • android
  • anthropic-claude-sdk
  • eas-(expo-application-services)
  • expo-audio
  • expo-av
  • expo-camera
  • expo-haptics
  • expo-router
  • expo-sensors
  • expo-speech
  • expo-speech-recognition
  • expo.io
  • google-cloud-text-to-speech
  • google-gemini-api
  • ionicons
  • ios
  • react
  • react-native
  • react-native-community/slider
  • react-native-gesture-handler
  • react-native-reanimated
  • react-native-safe-area-context
  • react-native-svg
  • react-navigation
  • typescript
  • web
Share this project:

Updates