Voice is the most natural form of interaction for humans. We use voice to communicate our needs and express ourselves to the fullest. We participate in so many conversations daily on a wide variety of topics. The goal of the project is to build voice interactions that can serve two important purposes: 1) To help practice reading and speaking to improve conversational skills in a language of choice. 2) A medium to build & play Choose-Your-Own-Adventure Games based on voice.
What it does
- LingoBuddy is a complete end-to-end platform for the creation and consumption of interactive voice-based stories. It allows an individual to create interactive & immersive stories through the story designer and share it with the world. At the same time, anybody on a mobile device can play the stories wherever they are and whenever they want.
- It provides a medium to help with language learning, as it is vital to practice a language by taking part in conversations. The ecosystem can also be utilized to create choose-your-own-adventure games for all ages. It can even be used by teachers to make any learning more interactive and fun. The possibilities for such a platform are endless!
How I built it
- The most crucial element of the entire ecosystem - speech recognition - is powered by Wit.Ai. Separate apps have been created on the Wit.Ai platform for different languages. For the mobile web app, the Wit.Ai library has been utilized to enable streaming on WebRTC & WebSockets. The LingoBuddy ecosystem can easily work with all the languages supported by the Wit.Ai platform (though currently, I have added the tokens for three languages: English, German, French). To add a new language, one can create an app on Wit.Ai, extract the Client Access Token and enter the language and the token in the language variable in the index.html file. It is as simple as that.
- For the database and hosting, we are using Firebase (though it can easily be deployed on any system).
Challenges I ran into
- The main challenge was to create a UI/UX that feels clean and functional. It took a lot of experimenting with many design iterations to get the final version ready.
- Another challenge was to work with the speech recognition bit as that is the soul of the ecosystem. Initially, the plan was to record the audio and then send the audio data to Python backend for processing through the Wit.Ai library and then return the result. But this would have degraded the experience and make it a little less friendly for the developers that may want to expand this ecosystem in the future. Luckily, I was able to work with the Wit.Ai microphone library, which worked smoothly (after a little tweaking, of course).
Accomplishments that I'm proud of
To create such a complex ecosystem that only works on web technologies!
What I learned
I have learned a lot while building this project! I have worked on a similar tech-stack before but never to this detail. Such a complex ecosystem requires the handling of many scenarios taking into the perspective of all the players. I learned the power of Wit.Ai, and it has tremendous scope to expand on.
What's next for LingoBuddy
From my experience building this project, I understood the following additions could be considered: 1) Adding text-to-speech so a person can also improve his/her listening skills. 2) Add more types of blocks to the designer to make it more engaging in many more ways.