Inspiration
Emergency alarms assume everyone can hear, see, and respond in the same way. Guardian Alert was inspired by a close call involving a hard-of-hearing family member who did not hear a building fire alarm. We wanted to turn an ordinary phone into an accessible, privacy-conscious safety companion.
What it does
Guardian Alert analyzes ambient audio for dangerous events such as fire alarms, smoke alarms, breaking glass, screams, sirens, and explosions. It delivers alerts through the channels that best match the user's accessibility needs: visual, audio, vibration, and flash. It can also prepare caregiver notifications with location context.
The system uses a hybrid AI architecture. YAMNet and browser audio processing identify sound events. FunctionGemma runs locally through Cactus Compute for fast, private function calling. When the local model cannot produce a reliable structured decision, the backend can fall back to Google Gemini through the official google-genai SDK. Gemini selects structured emergency functions such as classify_emergency_sound and send_emergency_alert.
How we built it
The frontend is a React, TypeScript, Vite, and Tailwind application with accessibility onboarding, emergency contacts, live monitoring, activity history, and multimodal alert components. MediaPipe loads YAMNet in the browser for real-time sound classification.
The Flask backend exposes an /api/analyze endpoint. Its hybrid router first calls FunctionGemma through Cactus Compute. It repairs malformed JSON, validates tool calls, and retries locally. If the local route is exhausted and a GEMINI_API_KEY is configured, it calls Gemini with structured function declarations. The returned tool calls determine the alert level and whether a caregiver alert should be sent.
Challenges we ran into
The main challenge was balancing emergency-response latency with reasoning quality and privacy. Cloud-only inference adds network dependency, while small on-device models can return malformed structured output. We added JSON repair, schema validation, focused retries, and a Gemini fallback. We also designed alerts for different combinations of hearing, vision, and cognitive needs without overwhelming the user.
Accomplishments that we're proud of
- A complete prototype from microphone input to personalized alert.
- Real in-browser YAMNet classification using MediaPipe.
- An offline-first function-calling path with a genuine Gemini API fallback.
- Structured emergency tools rather than unbounded text generation.
- Accessibility-specific visual, audio, haptic, and flash alert components.
- A benchmark score of 82%, including a 0.911 F1 score and 100% on-device routing on the benchmark workload.
What we learned
A hybrid model is especially valuable for safety-critical applications: local inference handles the common fast path, while Gemini provides a more capable recovery path for ambiguous inputs. The interface is also part of the safety system: clear confidence indicators, accessible alert modes, transparent routing, and truthful fallbacks matter.
What's next
We would test with a broader library of real-world alarm environments, calibrate confidence thresholds, validate the accessibility experience with users, and connect caregiver notifications to a production messaging service. We would also use Gemini for richer incident summaries, multilingual caregiver messages, and context-aware escalation while preserving a strict rule that urgent local alerts never wait for the cloud.
Log in or sign up for Devpost to join the conversation.