We found there to be a gap in engaging 2nd language education resources in terms of speaking practice
What it does
we created a choose your own adventure style app that implements a speech to text algorithm to guide users through a conversation eg. coffee shop order
How we built it
Using Android Studios and the CMU Pocket-sphinx library.
Challenges we ran into
Aside from a plethora of technical difficulties, the speech recognition module is based on cmusphinx/pocketsphinx open source speech recognition toolkit. A big technical issue we faced was the fact that we originally had to speak with 100% accuracy in order to pass a single ‘stage’ of the app. We solved this issue by implementing a fuzzy strings cosine similarities-based comparator as a measure that tests the similarity between a user’s voice input and the expected phrase, and accepted inputs with 70% precision to give leeway for users while also accounting for possible difficulties in phonetic transcription, as current transcription models struggle with person-to-person differences in speaking styles (speaking speed, idiolect, intonation)
Accomplishments that we're proud of
This is all of our first hackathons, and we are proud that we could get a working prototype within 24 hours, given all of the technical difficulties we faced.
What we learned
We had a brief introduction to the world of speech recognition, and an extended welcome to the world of Android App development.
What's next for Conversationalist
In the future we would really like to further tune the acoustic model by training it on more audio data to increase its accuracy when transcribing phonemes with finer detail. Furthermore, the pocketsphinx toolkit includes libraries for foreign languages and we could implement the same tasks for other languages which is a great opportunity for people who are studying a foreign language and would like conversation practice. Lastly, there’s more than enough every day situations that we can draw inspiration from and we’d like to add more scenarios aside from the initial coffee shop encounter.