Frontline

Real-time vitals, injury detection + voice guidance for emergencies.

Comment

🚨 Frontline

🎯 Inspiration

In an emergency, the hardest part isn’t calling for help, it’s knowing what to do right now while information is incomplete and time is limited.
We built Frontline to turn a short camera clip into instant triage + actionable guidance, like having a smart 911 operator in your pocket.

🔍 What It Does

Frontline is a touchless triage platform that performs rapid biometric + visual analysis from a 10-second video to generate a full emergency dashboard, incident report, and real-time guidance.

It delivers:

  • Instant vitals + injury detection
  • A structured ER-ready handoff report
  • Step-by-step first aid actions
  • A conversational AI 911 operator voice agent

🧠 How It Works (Pipeline)

1) User records a 10-second video 2) Gemini slices the video into ~10 key frames 3) Gemini performs image recognition / scene understanding 4) Frames are passed through Presage biometric analysis, returning vitals as JSON

  • Heart rate
  • Breathing rate
  • Focus / consciousness signal 5) Gemini combines visual findings + vitals JSON to generate a full triage dashboard:
  • Condition summary
  • Risk flags (shock, tachycardia, hyperventilation, unconsciousness)
  • Immediate actions / first aid plan 6) The app speaks the result out loud using an ElevenLabs voice agent 7) Users can talk back to the “911 operator” voice agent (ElevenLabs conversational AI) to get real-time help during the emergency 8) Users or EMS workers can replay the original video, generate an incident report file, or contact nearby hospitals

🧾 What the Dashboard Includes

  • 📊 VITALS

    • Heart Rate (HR)
    • Respiration Rate (RR)
    • Consciousness / focus status

  • 👁️ VISUAL ANALYSIS

    • Likely injury description
    • Bleeding severity
    • Patient posture/body position

  • 🧠 DIAGNOSIS

    • Combined interpretation of vitals + visuals
    • Shock / airway / bleeding risk detection

  • 🩹 FIRST AID STEPS

    • Clear prioritized steps: what to do immediately and what to avoid

  • 🔊 VOICE SUPPORT

    • Spoken instructions
    • Two-way “911 operator” conversation for follow-up questions

🛠 How We Built It

We built Frontline as a full pipeline product:

  • Frontend: React + Vite + Tailwind CSS

    • Video capture / upload
    • Dashboard UI + live triage flow

  • Vision + Reasoning: Gemini

    • Frame extraction + image recognition
    • Structured triage summary generation

  • Biometrics: Presage

    • Vitals estimation from extracted frames (returned as JSON)

  • Voice: ElevenLabs

    • Reads the triage instructions out loud
    • Conversational “911 operator” agent that responds to the user in real time

🚧 Challenges We Ran Into

  • Designing a pipeline that works reliably from a short clip (lighting, motion, camera angles)
  • Translating raw model outputs into clear, human first-aid actions
  • Building an interface that stays calm + readable under pressure
  • Making the voice agent helpful without overwhelming the user

🏆 Accomplishments We’re Proud Of

  • Built a complete end-to-end triage flow from video → dashboard → voice guidance
  • Combined multimodal reasoning (visuals + vitals) into one unified report
  • Implemented an interactive voice “911 operator” experience to guide users live

📚 What We Learned

  • In emergency UX, clarity matters more than complexity
  • Multimodal AI becomes most useful when it outputs structured, actionable steps
  • Voice interaction is a game-changer when hands and attention are limited

🚀 What’s Next

  • Improve robustness across devices + lighting conditions
  • Expand first-aid coverage with more conditions + severity scoring
  • Fine-tune the 911 operator agent for calmer, more reliable guidance

Built With

Share this project:

Updates