Inspiration
The name HailMary comes from the movie Project Hail Mary, in which a man stranded in space develops software capable of interpreting frequencies produced by an alien he encounters. That software creates a reliable method of communication between two individuals who would otherwise struggle to understand one another.
With HailMary, we wanted to take inspiration from that same idea of translation and communication, but apply it to a real-world accessibility problem. Our goal was to create a simple communication tool for nonspeaking students who may rely on gestures or other nonverbal methods to express themselves in a classroom environment.
What it does
HailMary is an accessibility-focused web application that translates a student's hand gestures into text-based messages for their teacher.
Students can open the Student Dashboard, start gesture sharing through their webcam, and make supported hand gestures. Our computer vision system detects the gesture and translates it according to the student's personalized gesture preferences.
For example, a student could define:
Open Palm→ "I have a question."Thumbs Up→ "I understand."Closed Fist→ "I need help."Pointing Up→ "Please repeat that."
Once the gesture is recognized, the student can send the translated message directly to the Teacher Dashboard, where the teacher can see who sent the message and acknowledge it.
Our goal is not to replace AAC devices, sign language, or other accessibility tools. Instead, HailMary is designed to act as an additional lightweight communication option for quick classroom interactions.
How we built it
HailMary is split into a React frontend and a Python backend.
Frontend
We used:
- React
- Vite
- React Router
- Browser webcam access through
getUserMedia()
The frontend contains separate interfaces for students and teachers. The student's webcam is previewed directly inside the website, while the teacher receives incoming translated messages through a dashboard.
Backend
We used:
- Python
- FastAPI
- Uvicorn
The FastAPI backend handles communication between the computer vision system and the frontend.
Computer Vision
We used:
- OpenCV
- MediaPipe
- MediaPipe's Gesture Recognizer
When gesture sharing is enabled, the browser sends camera frames to the backend. MediaPipe analyzes those frames, identifies the student's gesture, and returns the detected gesture and confidence score.
The backend then matches the detected gesture with the student's selected message and sends that information back to the frontend.
The overall flow looks like:
Student Webcam
↓
Browser Camera Frame
↓
FastAPI Backend
↓
MediaPipe Gesture Recognition
↓
Detected Gesture
↓
Student's Gesture Preference
↓
Translated Message
↓
Teacher Dashboard
Log in or sign up for Devpost to join the conversation.