Lips’ Talk

Inspiration

Our group realized that although we have many people with hearing and speaking disabilities in our society, they rarely had the chance to communicate with anyone who doesn’t know hand language. It is the same sad situation the other way around. With the booming of AR technology and devices, our group started to think about using them to enhance people’s conversations with subtitles to help the people with hearing and speaking disabilities.

What it does

Our project applies lip reading technology on Hololens(and other augmented reality devices) to help people with hearing or speaking disabilities to communicate like everyone else. The application also enhances common users’ communication quality under different scenarios.

How we built it

(A diagram is on our Slides)

Challenges we ran into

Only about 30 to 45 percent of the English language can be understood through lip reading alone, so it won’t produce the correct output every time. HoloLens is a new device. To develop an app on it, our team found it challenging as the resources online are quite limited. The project involves lots of video processing and real-time file sharing.

Accomplishments that we're proud of

We debugged the codes of lip reading we found on github and edited it to fit our project We used Unity to build the image capture and transcription projection process of our project We figured out how to deploy apps onto HoloLens We were working on the connections of our individual works