Masked Captions

A handy communication tool that crosses barriers posed by both languages and masks, powered by Google Cloud

Comment

Inspiration

The pandemic has presented us with unusual challenges and one that our group was compelled to explore from firsthand struggles was communicating through masks in day-to-day interactions. One particular challenge that modern day workers face is the inability to communicate efficiently within socially distant meetings as seen through coop experience. This results in meetings being unnecessarily prolonged and crucial information being miscommunicated. This prompted the idea of utilizing a phone to translate speech to text such that individuals can have a safe conversation while being socially distant.

What it does

Masked Captions is a user-friendly, cross-platform mobile application that continuously streams speech input from a phone’s microphone and transcribes it to text on the app’s view with translation capabilities. Paired with our 3D-printed phone mount which is compatible with commercial masks, our product provides users real-time captions that are visible from a safe distance and facilitate smooth conversations.

How we built it

The app was built using Unity connected with Google Cloud’s Speech-to-Text and Translate APIs. In order to connect the unity app to an IOS mobile device, a computer running a MacOS with Xcode was utilized to create the mobile application. The UI was rapidly prototyped in Figma and implemented in Unity. The phone mount was designed using SolidWorks and 3D printed on a Creality CR-6 SE.

Challenges we ran into

An initial challenge was selecting a mobile development framework that worked best with the aforementioned APIs. After exploring a couple options, we landed on Unity and were successful in setting up and connecting to Google Cloud.

Accomplishments that we’re proud of

Although we had low hopes for any hardware components in our hack given the fully virtual nature of this event, we were glad we could incorporate 3D printed components. The ability to bring all of the features that we initially planned for in the tight timeline.

What we learned

We discovered the power and versatility of Google Cloud and Unity as development platforms. In addition, we learned how to download unity created applications on IOS mobile devices through Xcode.

What's next for Masked Captions

We're hoping Masked Captions can become a useful tool for all frontline workers who have a difficult time communicating through masks and other sound-inhibiting precautionary measures. We also see this product being useful for the hearing impaired.

Built With

Share this project:

Updates