Inspiration
A Deaf person at a pharmacy or urgent-care desk often has no interpreter, and writing notes back and forth is slow when you're in pain. We wanted a way to communicate that needs only a laptop webcam.
What it does
Hermes watches someone sign and recognizes about 60 everyday and medical signs, like PAIN, MEDICINE, BATHROOM and HELP. It translates the signs into natural English with Gemini (MY HEAD PAIN NEED MEDICINE → "My head hurts, I need some medicine.") and speaks the sentence with ElevenLabs. When the hearing person replies, Hermes transcribes the reply and shows it as a caption.
How we built it
- Python + OpenCV for the camera loop and UI, with MediaPipe tracking the hands and body.
- Data: Google's GISLR dataset from Kaggle, plus signs we recorded ourselves.
- Model: each sign is described by hand shape, hand position relative to the body, and movement over time. A scikit-learn pipeline classifies it with a small group of MLPs that vote together.
- Translation: we started with Grok (xAI), then switched to Gemini because it responded faster.
- Built in Cursor, using its AI agents to move fast.
Challenges we ran into
- Phantom signs: classifying every frame turned the motion between signs into random words. We switched to detecting a whole sign first and classifying it once.
- Camera mismatch: the dataset's portrait videos didn't match our landscape webcam. Measuring positions relative to the signer's body fixed it.
- Latency: we switched to a faster LLM and tuned the timing so the conversation feels natural.
Accomplishments that we're proud of
- A two-way conversation, from signs to speech and from speech to captions, running live on one laptop with no special hardware.
- A public dataset and our own webcam recordings training the same model.
- A recognizer that says "not sure" instead of guessing words for someone.
What we learned
With sign language, where the hands are and how they move matter as much as hand shape. Deciding where each sign starts and ends matters as much as the model itself.
What's next for Hermes
- Call mode, so signers can phone a clinic or pharmacy
- Fingerspelling for names and medications
- Continuous-signing models that read whole sentences instead of one sign at a time
- Testing with Deaf signers
Built With
- asl
- chertfacetime
- computer-vision
- cursor
- elevenlabs
- gemini
- godaddy
- google-ai-studio
- grokbot
- kaggle
- machine-learning
- mediapipe
- numpy
- opencv
- pandas
- python
- scikit-learn
- speech-to-text
- text-to-speech
