SpeakNow

Inspiration

Our inspiration came from the language struggles many immigrant families, including ours, experience every day. From having family members who were unable to speak English to having difficulties communicating in our native tongues, we experienced firsthand how language can create barriers in moments that matter most. After watching language challenges play out again and again, it became clear that there had to be a better way to communicate. This idea is what inspired us to build our project: an AI-powered live translation agent that joins phone calls.

What it does

SpeakNow acts as a real-time translation agent by listening to two people who don’t share a common language and translating their words instantly, repeating them back in the other person’s language. Through this process, our app allows for seamless, natural conversation where both callers can talk in their own language and still fully understand each other.

How we built it

We built SpeakNow using Websocket, RealtimeAPI from OpenAI and Twilio Voice API as our main sources of API calling, using Python for our backend and phones to test everything out!

Challenges we ran into

Some challenges we ran into was figuring out how to use technologies we've never used before. For us, we've never tried using Realtime API from OpenAI or the Twilio Voice API, which are crucial for our goals of creating a live translation service to help individuals with language barriers. We had no idea how to process the speech and make it possible to be translated, but with some engineering and a lot of Googling, we came up with a flow-chart using current technologies to create a system of making live translation possible, in real time.

Accomplishments that we're proud of

We came to VTHacks 13 as a team with two first-time hackers, and together we were able to collaborate and build a finished project.
We gained valuable experience working with two new APIs and integrating both in our project.

What we learned

We learned how to make live translation possible for individuals with language barriers, while gaining valuable experience with unique APIs and technologies; things we've never tried before.

What's next for SpeakNow

We want to expand our app beyond simple translation to become a more adaptive communication tool. For users with additional language barriers, such as those who rely on hearing aids, we could provide live captions alongside translations. We also plan to refine our system so it supports not only more languages, but also diverse accents and dialects with greater accuracy.