Inspiration

Every emergency system assumes you can talk. But what about someone hiding from an intruder, someone choking or having a stroke, someone who is deaf or non-verbal, or someone who doesn't speak the local language? In those moments a phone call becomes useless. We asked a simple question: what if your phone could call for help on your behalf, without you saying or typing a single word? That question became SilentPulse.

What it does

SilentPulse is zero-text, multimodal emergency triage.

  1. Open the app. No forms, no typing, no talking required.
  2. It captures the scene through the camera, ambient audio, and device sensors, along with your location.
  3. AI triages it. Gemini and Grok analyze what's happening, identify the likely emergency, and assign a severity level.
  4. Dispatch gets a structured report on a live dashboard showing what's wrong, how urgent it is, and where you are.

The result is an emergency call for people who can't make one.

How we built it

  • Mobile PWA (Next.js): a one-tap capture flow that works from the browser, so there's no app install during an emergency.
  • Backend / API: receives the multimodal payload, orchestrates calls to Gemini and Grok, and merges their outputs into a single structured triage report (emergency type, severity, confidence, location).
  • MongoDB: stores incidents and their triage history so dispatchers can see updates in real time.
  • Dispatcher dashboard: a live queue of incoming incidents sorted by severity, with the AI's reasoning and the captured context.

Challenges we ran into

  • Turning messy, wordless input into a decision. Blurry frames, background noise, and partial context had to become one clear, confident triage result.
  • Getting two models to agree. We had to decide how to combine Gemini's and Grok's outputs, and what to do when they disagreed.
  • Designing for panic. The UI had to work for someone who is terrified, injured, or can't look at the screen.

Accomplishments that we're proud of

  • A full end-to-end loop working in a weekend: phone → AI triage → live dispatcher dashboard.
  • A triage flow that needs zero words from the user.
  • Built a Twilio and AI text-to-speech pipeline that instantly calls emergency contacts. The system reads custom updates written by the dispatcher aloud, ensuring loved ones receive critical, human-controlled information immediately without tying up responders on the phone.

What we learned

  • How to orchestrate multiple LLMs on multimodal input and turn their output into structured, reliable data.
  • That accessibility isn't an edge case. A huge number of people can't safely speak during an emergency.
  • How to split a real-time system across a team (mobile, backend, dashboard) and integrate it fast.

What's next for SilentPulse

  • Integration with real dispatch infrastructure (e.g. text-to-911 / CAD systems).
  • On-device detection for when connectivity is poor.
  • Wearable triggers, such as a discreet tap pattern on a smartwatch.
  • Multilingual output so dispatchers get reports in their own language.

Built With

Share this project:

Updates

Submission history