Hello Mundo

A web app that transcribes whatever you say in a speech bubble and translates your message to another language.

Comment

As our society becomes more and more globalized and interconnected, communication between everyone and anyone is crucial, especially those who share different languages or have hearing disabilities. Our app aims to combat this issue! We aim to use Hello Mundo (Mundo meaning "World" in Spanish) to advance the process of learning different languages.

What it does

The app transcribes whatever you say into a speech bubble right next to your face and instantly translates your speech into a language of your choice. The app also has the ability to read out loud the translation if you are interested in what your speech sounds like in another language in a call-and-response format. This web app can connect people who speak different languages and connect them in a natural face-to-face conversation.

How we built it

We used the tracking.js facial recognition algorithm to track the user’s face so we know where to insert our speech bubble, which was created and positioned using JavaScript. The app also uses node.js, express.js, and Google's Speech API to constantly input the user's speech and output the transcription. We used Google's Translation API to translate the spoken text and translateJS to read out the translated transcription. The design and chat log was created using HTML and CSS.

Challenges we ran into

Since we are all freshmen with no computer vision experience, we had some difficulties with creating the speech bubble and having it follow the user. We had some trouble with the webcam framerate and the constant flickering of our speech bubbles. Towards the middle, we have some issues with implementing Google's speech API with our main code that creates the webcam and the speech bubble. After this implementation, it took a while to translate the transcription because we were new to Node.js and were trying to understand the communication between the client base and the server.

Accomplishments that we're proud of

We're proud that we were able to grasp the functionality of Node.js and other other frameworks and APIs that we used after staying up all night! Seeing the text bubble float by the user's head, watching the speech get translated almost instantly into different languages, and looking at the neat, colorful designs we came up up have made this experience well worth the lost sleep!

What we learned

We learned a lot about the basics of Node.js and the structures of Google's Speech and Translator APIs. We extended our PHP skills and refined our understanding of JavaScript, which we have been developing for several months. We are super excited to take our newfound Google Cloud API skill set and use it to make Hello Mundo better and add Google's APIs to our future projects.

What's next for Hello Mundo

In the future, we aim to allow multiple people to talk to each other, each with their own bubble, and have their conversations be translated. Ultimately, we hope we can extend this app for impactful applications, such as Chrome extensions that can translate videos that you are streaming or Skype calls. This app has the capability of bridging language and communication barriers, allowing people to interact with a more diverse pool of people and culture on the Internet. We can help educate future generations through our app's simple call-and-response way to learn a new language. With some added features, it can even help people all across the world talk and learn from one another. We truly believe that our technology, even combined with something like Google Glass or HoloLens, could revolutionize communication and make each other better members of the global community.

Share this project:

Updates