Inspiration

We wanted to work build around the sensing or analyse of a human signal. We quickly agreed on voice and decided to help mute people get their voice back.

What it does

Silent Speech is an app on the web (can be run on iphone, PC, Mac etc). It watches the lips of the user and uses AI to reconstruct what is said even if their is no sound coming out of the user's mouth. It allows mute people to call someone for the first time in history.

How we built it

Using a smartphone we can connect to a webpage where all the frontend is shown. Then the camera frames are sent through the cloud (cloudflare) to a local Asus Ascent PC we have (courtesy of CalHacks). On this PC we run locally all our AI models: lip reading machine learning, then LLM (Qwen) to correct the lip reading output, then emotion detection (HSEmotion) and finally speech synthesis using all of the above. Furthermore, we use Twilio to be able to stream the speech synthesis through a cellular phone line.

Challenges we ran into

Wifi and network issues were frequent. We struggled so much with it that we decided to setup our own hotspot from the Asus PC and work only offline for the data transfer between the phone and the Asus PC.

Accomplishments that we're proud of

We made a real time LIVE lip reading application. This mean we did not just build an application that takes a video clip and outputs the text later, but something that works at the same rhythm as the user, allowing real time communication for mute people.

What we learned

Different things for each members, but globally: -lip reading models -running AI locally -cloud and wifi data transfer -twilio and phone calls api

What's next for Silent Speech

Next we could train our own model on a huge database such a Youtube videos. Once we get a really accurate model, this could be deployed as a proof of concept for a potential monetized product for mute and death people (basically Duolingo like application). The project could also be open-sourced.