Cueit

Side view
Front view

Inspiration

For deaf and hard-of-hearing individuals signing isn’t an alternative, it’s their primary language and a deeply expressive way of communicating. Yet it’s often underrepresented and still treated as something that needs translation. Most solutions like interpreters, apps, and AI tools delay or interrupt natural flow of conversation. In medical emergencies, that delay can directly affect care.

Even human interpreters aren’t always immediately available in urgent medical situations. And most tools assume patients can adapt to the technology by holding a phone, facing a camera, or waiting through setup. But in reality, someone in distress often can’t do any of that.

In moments like these, communication should just happen. It should be instant and effortless, because every sign deserves to be heard!

What it does

Our AI-powered smart glasses, Cueit, mount the camera on the doctor’s frame and follow their natural line of sight. They capture sign language during normal interaction and convert it into speech in real time, without any hassle or setup for the patient. Communication begins the moment the doctor looks at the patient while the wearable translates it simultaneously. This shifts the burden away from the patient and creates a more natural and inclusive experience where communication feels effortless. Unlike traditional translators, Cueit isn’t stationary, it moves with the doctor, enabling seamless translation anywhere in real time.

How we built it

We built an AI-powered smart glasses translation system on a Rubik Pi connected via USB cable. Using a computer vision pipeline powered by MediaPipe, our system detects and tracks hand gestures in real time through a mounted webcam. The recognized signs are mapped to medical keywords and sent to Anthropic’s Claude API, which converts them into clear, medically accurate sentences. Finally, ElevenLabs’ API transforms the response into natural-sounding speech, which is played aloud directly to the doctor.

Challenges we ran into

Integrating the Rubik Pi took significant time due to higher power requirements, leading us to switch to a 40-Watt adapter
Connecting the Eleven Labs voice API was challenging, as audio output initially failed on the Rubik Pi and required adapting to syntax and environment differences from a typical setup
Hardware limitations made audio output difficult. We switched from an 8-Ohm 2W mini speaker with an external amplifier to USB-C earphones or a computer speaker for reliable audio output.

Accomplishments that we're proud of

Building a portable sign language translator that reduces the challenges and delays of traditional ASL interpretation.
Successfully integrating a vision-based pipeline that recognizes hand gestures in real time. Getting our LLM (Claude API) to interpret recognized signs into medically accurate, context-aware sentences.

What we learned

We learnt that many hospitals still lack a standardized protocol for communicating with deaf patients in emergency situations.
Even simple gestures like “yes,” “no,” and “pain” can be misinterpreted without proper context in sign language communication.
How to effectively integrate software and hardware, especially as most of the team came from a software-focused background.
How to deploy computer vision models on embedded hardware while working within real-world constraints like power limitations and audio output challenges.

What's next for Cueit

*Enable two-way communication, allowing doctors to respond back in a way that is accessible and friendly for deaf or sign language users. *Expand the gesture vocabulary and improve the sign language recognition model for higher accuracy and broader coverage. *Develop more advanced motion analysis to better capture complex and nuanced signing patterns. Optimize hardware by using more compact and efficient components that maintain performance while improving portability.