Beyond Words

We wanted to explore how technology could help people who use sign language communicate with people who don’t understand it. As a beginner team, we set out to build a small prototype that recognizes a limited set of signs and turns them into speech.

Our web app uses a webcam to capture hand movements, recognizes signs we trained it on, and speaks the detected words aloud. It’s an early prototype focused on a small vocabulary.

We split the work across the interface, camera and hand tracking, sign recognition model, and speech integration. We used HTML, CSS, and JavaScript for the interface, MediaPipe for hand landmarks, and a trained model to recognize signs. We worked on connecting the recognized words to ElevenLabs for spoken output.

We learned how camera input, machine learning, and speech generation fit together. We also learned that recognizing a moving sign requires capturing a sequence of movements, rather than just one hand position. Working together taught us how to use GitHub branches and connect code developed separately.

Built With

css
elevenlabs
html
javascript
mediapipe
python
typescript

Submitted to

StormHacks 2026

Created by

I worked on the camera, MediaPipe hand tracking, and data collection for our ASL recognition system. I set up the webcam to detect hand landmarks, recorded labeled training samples for the signs “Hello,” “Hungry,” and “How,” and created a Python training script to train a recognition model from that data. I also organized the dataset and pushed everything to a separate GitHub branch so my teammates could continue with the backend and integration. This was my first time working with MediaPipe and training an ML model, so I learned a lot about computer vision, data collection, Git branches, and how different parts of a full-stack project connect together.

Puspita Juwita
I help the frontend with my friend. We used react.

andylou123
I built the frontend interface using React and connected the backend to the front end

Chris Lu
Devolped the camera functionality and backend integration using JavaScript and Python. Built the webcam capture feature and integrated MediaPipe hand-landmark tracking with a teammate-trained sign recognition model. Using Python and FastAPI, I worked on the backend to process recorded hand movements, return predictions, and connect recognized words to the ElevenLabs API for spoken audio. I also helped combine our components through GitHub branches and troubleshoot integration issues, bringing together the full flow from capturing a sign to hearing it spoken aloud.

Mariam Alsayad