Team 45_5E_translingo

Ever craved a connection beyond borders, nationalities, language? Presenting Translingo, a confident step towards a world without language barrier, powered by generative AI with a dash of human touch.

Comment

Inspiration

Our exchange programs overseas had been an invaluable opportunity that broadened our worldview, but a big regret we had was the lack of interaction with locals who spoke little English. After all, learning German, Danish, Japanese and more foreign languages is no easy feat. That was our inspiration for TransLingo, a next generation translator app that leverages the power of AI for accurate translation. However, we wanted more from TransLingo. We craved that personal connection, so with TransLingo, we create the translation in our own voice.

What it does

Our application provides a one stop platform for users traveling to countries speaking foreign languages, whether the user is looking to translate audio to a specified foreign language, converse with locals, or whether the user has questions regarding tourist attractions or looking for recommendations on what to visit near them. This application integrates state of the art artificial intelligence applications for speech to text transcription, dialogbox assistant using prompt engineering, as well as translation from any language to another language. After translation, the user has the option of playing the translated output, which would come in handy if you would like to express something in a foreign language but you do not know how to read the sentence.

How we built it

We built the project from start to finish on React and Next.js, also making use of Tailwindcss for the frontend. The backend of the project was integrated with Amazon AWS S3 Bucket, which allows user data such as login information, user-specific audio to be saved to the database and retrieved during login. We added openAI's text to speech synthesis AI model in our application. We also integrated our application with OpenAI API for various generative-AI applications such as translation, transcription and chatbot assistant.

Challenges we ran into

Integrating different components of generative AI smoothly into the application was a challenge we needed to overcome. We combed through the latest state-of-the-arts for powerful AI models that could power TransLingo, and combine their functionalities to seamlessly work with our web app. However, despite each of our team members' background in diverse branches in computer science and mathematics, we could still successfully merge most of the deep learning framework with the web app. Using AWS S3 bucket for the first time, we also had difficulty doing authentication initially. The only issue we still face is the text-to-speech synthesis. We wanted to synthesise speech in the user's voice (recorded during registration), however most existing models required expensive paid API keys. We worked on integrating open source models into the application, but there were also many issues such as inability to call the model in python on our next.js application despite dockerising the model and more. Hence, we decided to stick to simple text to speech synthesis first, without the voice cloning ability.

Accomplishments that we're proud of

Despite our very different backgrounds in the team, we are able to work to our individual strengths, work together when any issue is faced, and brainstorm solutions together. Many of us are inexperienced with frontend developing and deploying AI models, so it was a challenge to pick it up in the short timeframe and we were faced with many unexpected errors that were difficult and time consuming to resolve, but we were able to get almost everything to work without issue.

What we learned

Throughout our journey in this hackathon, we were able to hone a variety of skills from understanding and the various intricacies of frontend development, attempting to craft seamless user experiences through well designed interface, to improving user experience further by integrating state of the art artificial intelligence models. Beyond coding, we also improved various soft skills that are equally important. Communication skills, ability to collaborate, effective conveyance of ideas were all important aspects that contribute to a positive project environment that encourages problem solving and brainstorming.

What's next for Team 45_translingo

We want TransLingo to be a more complete application, with more Artificial Intelligence uses such as image generation, that will continue to improve the experience for tourists. Our next step would be to have more complete functionalities such as allowing login authentication using Google Account, allowing user to view history of previous translations. AI applications such as image generation can be used such that when tourists have queries regarding any tourist attraction, such as looking for an attraction, or trying to identify an attraction, they can speak to the chatbot which will generate relevant images for tourists. Finally, we would like an optimized version for mobile application.

Built With

Share this project:

Updates