HandsOff

Home Page 1
Home Page 2
Hand Gesture Detector
Test Alert Call
Emergency Contact List

Inspiration

HandsOff was inspired by the very real safety risks faced by LGBTQIA+ individuals, especially trans people and youth who experience disproportionately high rates of domestic violence and street harassment. In many of these situations, calling emergency services is not safe, possible, or timely. We were motivated by the idea that safety tools should work quietly in the background and support people without escalating danger. The international Signal for Help already exists as a silent, universally recognized distress gesture. HandsOff brings it into the digital world by translating it into immediate, automated protective action.

What it does

HandsOff is a silent, hands-free safety application that leverages real-time computer vision to detect the international Signal for Help gesture through the user's device camera. When the gesture is recognized and held for a configurable duration (default 3 seconds), the app discreetly activates protective workflows. At the first level, it triggers a simulated incoming phone call, complete with a realistic ringtone generated via the Web Audio API and AI-synthesized voice audio from ElevenLabs Text-to-Speech, giving the user a natural excuse to exit an unsafe situation. At the next level, it captures the user's GPS coordinates via the Browser Geolocation API and creates a Google Maps link. Users can manage trusted contacts, customize caller identity and AI voice, adjust gesture sensitivity, and review a full alert history log, all from a responsive, dark-mode-enabled interface.

How we built it

HandsOff is a full-stack TypeScript application built on React with Vite for the frontend and Express.js for the backend API layer. The UI is styled with Tailwind CSS and uses a consistent, accessible design system with full dark mode support via a custom ThemeProvider.

For gesture detection, we integrated Google's MediaPipe Hands solution, which runs entirely in-browser using WebAssembly and GPU-accelerated inference, meaning no video data ever leaves the user's device. A custom finite state machine manages gesture lifecycle (idle → detecting → confirmed → cooldown), requiring intentional sustained holds to prevent false positives.

The fake call system uses the Web Audio API with an OscillatorNode to generate realistic ringtone patterns, and ElevenLabs' v1 Text-to-Speech API to synthesize natural-sounding caller voice audio streamed as audio/mpeg buffers. Users can choose from premade AI voices with in-app preview playback.

Data persistence uses MongoDB Atlas with Mongoose ODM for schema modeling, storing trusted contacts, alert history, and app settings. API validation is handled with Zod schemas shared between frontend and backend.

Challenges we ran into

One of the biggest challenges was balancing detection sensitivity with reliability. The MediaPipe Hands model needed to accurately recognize the Signal for Help across varying lighting conditions, camera angles, distances, and skin tones. Tuning the confidence thresholds (0.7 detection, 0.5 tracking) and hold duration requirements required extensive iterative testing. Integrating ElevenLabs TTS also presented hurdles around API permission scoping, where we implemented a resilient fallback system with a hardcoded voice catalog when the voices_read permission isn't available. Additionally, designing safety features that de-escalate situations rather than create new risks demanded careful UX consideration at every step.

Accomplishments that we're proud of

We built a fully functional end-to-end safety tool that translates a silent real-world gesture into immediate, multi-layered protective action. The entire computer vision pipeline runs client-side with zero server-side video processing, preserving user privacy by design. We're especially proud that HandsOff centers the needs of marginalized communities, including LGBTQIA+ individuals, and reframes computer vision technology as a tool for protection rather than surveillance or control.

What we learned

This project taught us how critical it is to design technology for high-risk, real-world contexts where small design decisions, like gesture hold duration, false positive rates, or alert delivery latency, can have serious safety consequences. We gained deep hands-on experience with real-time browser-based computer vision using MediaPipe, audio synthesis with both the Web Audio API and ElevenLabs neural TTS, and building full-stack TypeScript applications. We also learned how to architect systems that prioritize trust, privacy, and inclusivity from the ground up.

What's next for HandsOff

In the future, we plan on integrating Twilio's REST API for delivering SMS messages containing Google Maps links to emergency contacts. Location sharing is powered by the Browser Geolocation API for GPS coordinates and Twilio's REST API for delivering SMS messages containing Google Maps links to emergency contacts. We also plan to improve gesture robustness across more environments and device types, expand the library of configurable discrete safety responses, match the ringtone to the user’s device ringtone to increase realism, and add multi-language SMS support. We aim to collaborate with LGBTQIA+ advocacy and safety organizations to validate our design choices and ensure that HandsOff meets the real needs of the community. Long-term, our goal is to make HandsOff a trusted, silent safety companion for anyone who needs it.