Inspiration
Every emergency system assumes you can talk. But what about someone hiding from an intruder, someone choking or having a stroke, someone who is deaf or non-verbal, or someone who doesn't speak the local language? In those moments a phone call becomes useless. We asked a simple question: what if your phone could call for help on your behalf, without you saying or typing a single word? That question became SilentPulse.
What it does
SilentPulse is zero-text, multimodal emergency triage.
- Open the app. No forms, no typing, no talking required.
- It captures the scene through the camera, ambient audio, and device sensors, along with your location.
- AI triages it. Gemini and Grok analyze what's happening, identify the likely emergency, and assign a severity level.
- Dispatch gets a structured report on a live dashboard showing what's wrong, how urgent it is, and where you are.
The result is an emergency call for people who can't make one.
How we built it
- Mobile PWA (Next.js): a one-tap capture flow that works from the browser, so there's no app install during an emergency.
- Backend / API: receives the multimodal payload, orchestrates calls to Gemini and Grok, and merges their outputs into a single structured triage report (emergency type, severity, confidence, location).
- MongoDB: stores incidents and their triage history so dispatchers can see updates in real time.
- Dispatcher dashboard: a live queue of incoming incidents sorted by severity, with the AI's reasoning and the captured context.
Challenges we ran into
- Turning messy, wordless input into a decision. Blurry frames, background noise, and partial context had to become one clear, confident triage result.
- Getting two models to agree. We had to decide how to combine Gemini's and Grok's outputs, and what to do when they disagreed.
- Designing for panic. The UI had to work for someone who is terrified, injured, or can't look at the screen.
Accomplishments that we're proud of
- A full end-to-end loop working in a weekend: phone → AI triage → live dispatcher dashboard.
- A triage flow that needs zero words from the user.
- Built a Twilio and AI text-to-speech pipeline that instantly calls emergency contacts. The system reads custom updates written by the dispatcher aloud, ensuring loved ones receive critical, human-controlled information immediately without tying up responders on the phone.
What we learned
- How to orchestrate multiple LLMs on multimodal input and turn their output into structured, reliable data.
- That accessibility isn't an edge case. A huge number of people can't safely speak during an emergency.
- How to split a real-time system across a team (mobile, backend, dashboard) and integrate it fast.
What's next for SilentPulse
- Integration with real dispatch infrastructure (e.g. text-to-911 / CAD systems).
- On-device detection for when connectivity is poor.
- Wearable triggers, such as a discreet tap pattern on a smartwatch.
- Multilingual output so dispatchers get reports in their own language.