Universal Translator

The UI of our translator.

Inspiration

Ayrton was working in retail with a delivery worker who had to use google translate in order to communicate with his other coworkers. Inspired by that experience, we decided to take a shot at our own translator.

What it does

It takes in speech input of a given language, then outputs translated audio.

How we built it

The base model was that we used a three step machine learning pipeline. The first machine learning layer was used to transcribe speech into text, then we translated that speech into text of another language. The final layer took the translated text, and generated the output audio which would be played.

Due to the heavy workload demanded in these machine learning models, work was offloaded to a server over a network.

Challenges we ran into

A huge problem was handling the socket network communication, which took us a huge amount of time to fix. Another odd complexity that we didn't expect was the handling of audio files, especially in Flutter where it was sometimes difficult to keep track of the file locations.

Accomplishments that we're proud of

Overcoming the sockets is a huge one we're happy with. It's also been quite fun to put together the separate parts of our pipeline, and then run them through all together.

What we learned

Networking is scary, and sockets are out to get us

What's next for Universal Translator

While the hackathon didn't give us the time needed to handle these, if we had the time we'd have loved to register AI voices of speakers, so that the output would sound more personalised. The auto-detection of input language would have been a huge boon if we could accomplish it (though it seems especially infeasible for the scope of this hackathon). And we'd have liked to implement some noise reduction AI in order to make a speaker's words register more clearly.