-
-
optics behind the project
-
-
demo of smart glasses!
-
conversation with sentient
Inspiration
Coming from immigrant families, we've seen our parents—and many others—struggle with language barriers in the workplace. These challenges often lead to miscommunication, missed opportunities, and added stress on the job. We wanted to build something meaningful: a tool that uses the power of generative AI to support people like them and make workplaces more inclusive and accessible.
What it does
LinguaLens can be described as smart glasses with augmented reality features that helps non-English-speaking workers understand spoken English in real time. It utilizes generative AI to transcribe, translate, and even have conversations on the spot with our bot "Sentient", displaying the translated text directly in the user’s field of view. Our best application is for immigrants in blue-collar environments such as house cleaners and construction who have essentially no English proficiency.
We also added a "Hey Sentient" feature which enables users to query ChatGPT through the smart glasses, enabling workers to access unlimited knowledge through their voice.
How we built it
The system uses an Arduino Nano as the central controller, connected to an HC-05 Bluetooth module for wireless communication with a computer. Spoken English is captured through a microphone, transcribed and translated using AI via the Google Speech Detect API and OpenAI API, and then sent back to the glasses in real time. The translated text is displayed on a small OLED screen mounted inside the glasses.
To make the text visible to the wearer, a mirror and magnifying lens setup projects the display into the user’s line of sight without obstructing normal vision. All components are housed within a custom 3D printed shell designed to be lightweight and wearable throughout a workday.
Challenges we ran into
Our biggest challenge was designing the hardware and connecting it to our software. We had to perform research on how light refraction and magnifying glasses work. Since none of our members had expereince with hardware originally, this was a really big learning moment for all of us. Then, we had to research parts, buy them, assemble them, and then configure them. This involved figuring out how to get parts cut correctly, assembled.
Accomplishments that we're proud of
We’re proud that we turned a deeply personal problem into a functional, real-world solution. LinguaLens successfully translates and displays live speech with minimal lag, using generative AI in a practical, wearable format.
How to run
You can run our software by first installing all the necessary python dependencies and uploading your API keys. Then, you just simply run "python mic_to_text.py". Unfortunately, since most of our project is hardware, you can't recreate that at home.
Log in or sign up for Devpost to join the conversation.