PhonoLearn

Inspiration

Taiwanese Hokkien is a low-resource language and its tone rules are hard to learn. An effective way to learn correct pronunciation is for a native speaker to correct the learner on their pronunciation. However, not everyone has access to someone who can tutor them on pronunciation, especially considering that Hokkien is relatively low-resource, so I wanted to create an app that tutors people on pronunciation with AI. These same problems are also present in other nearby, low-resource languages (like Wu and Eastern Min), so PhonoLearn is also extensible to many other languages. This project will help to preserve underresourced languages by allowing people to learn the pronunciation of these languages easier, as well as helping people who need to learn these languages for speaking with people who may only understand low-resource languages i.e. humanitarian aid workers.

What it does

The app uses ASR to interpret the user's pronunciation and compares it to a target sentence. Then, the app does an analysis of the pronunciation and tells the user what they got wrong in their pronunciation. The analysis is then displayed through gradio, complete with color. The app is also extensible, as there is a predefined Python interface for adding new languages. To add a new language, all you need to do is to provide ASR capabilities as well as a comparison function.

Note: to allow recording functionality, make sure to set your browser's microphone permissions to "Allow" or else the website won't have access to your microphone

How we built it

• I used an ASR model from HuggingFace Hub (https://huggingface.co/emlinking/wav2vec2-large-xls-r-300m-tsm-asr-v6) that is a finetune of wav2vec2 combined with gradio

Challenges we ran into

• The HuggingFace CPUs were a bit slow, slowing down development a bit. However, in a real world deployment of this app, it would be much faster due to the ability to utilize GPUs.

Accomplishments that we're proud of

• Creating a working learning app for a low resource language

What we learned

• Wav2vec2 works well for low resource ASR

What's next for PhonoLearn

I hope to extend this app to more languages, and to also host this on a platform with better processing power and GPUs.

Built With

• gradio
• huggingface