AI Assisted Language Learning Auto Pronounciation

Inspiration

I'm a total travel junkie and former literacy program developer so with the two influences, I wanted to create a product that helps people learn the correct sounds of a language from the initial learning stages that can be applied both for literacy interventions and in real world contexts. This was inspired from having worked with at-risk and English as Second Language students for many years in New York City as well as when traveling meeting a group of women who were sex trafficked by their families in Cambodia. They told me if they could learn to speak, read and write English more proficiently, they would be able to get better jobs in tourism. I wanted to help them, but I couldn't up and move to Cambodia so I started building online language and literacy development platforms. For years, I've wanted to build out an avatar based pronunciation assistant but with providing free resources for learners and having self-funded all of the projects in up to a 8 languages, both datasets and language models weren't proficient enough and working with the type of engineers that can build such a multimodal approach is rather costly. While I've been able to attract top talent and had lots of amazing people help me on this journey, this type of project takes a considerable time and dev work. Bolt and Tavus help get us closer to that goal and with some more time, work and a proprietary dataset I've built up myself, I should be able to finally customize multilingual pronunciation and reading assistants even further to serve language learners, emergent readers and those with wanderlust, like me, who just want to communicate across cultural lines with more confidence.

What it does

It provides real time feedback on pronunciation for language learning and reading to help people learn the correct sounds from the outset of the language development journey. It uses AI models trained with phoneme recognition to help learners understand how to orientate their mouths correctly for the proper pronunciations. It gives feedback at the sound, word and sentence level for practice from the beginning stages of language learning to the conversational stages. The Tavus avatars allow the learner to continuously practice for enhanced skill development. The more you practice- the more you learn!

How we built it

Built using Bolt! Tavus, Canva and a wide mix of Hugging Face datasets and AI models.

Challenges

We had significant challenges with crashes later on in the hackathon because of server overloads. Every server seemed to crash from Bolt, Tavus, Netlify, even DevPost.. and we deployed this more than twice using duplicate copies on Netlify and the links don't appear to be working so we had to reconfigure and rebuild this app many, many times over so far. We may have added too many datasets and models to improve our feedback loop and there may be too many advanced features that we tried to add all at once for it to work correctly every time. It works accurately about 50% of the time right now.

Accomplishments that we're proud of

Getting the backend of the AI models added and trained. I've wanted to build this for a long time so I'm glad I finally got to play around with the MVP.

What we learned

That we can use Bolt Vibe Coding to embed several models and datasets and build with them. While we might have overloaded our project- still it can be done to some extent with vibe coding- which is a great starting point.

What's next for AI Assisted Language Learning Auto Pronunciation

We'll continue to improve upon what's there with proprietary datasets and nuanced conversational skill development. We'll also spend significantly more time customizing and training the Tavus avatars.

Built With

• bolt
• canva
• huggingface
• stripe
• supabase