-
-
Water was guessed correctly
-
Stairs weren't guessed correctly, so it shows the right translation
Inspiration
When Alex was learning English, he found that he learnt fastest by associating words in the target language with pictures or the objects they represent. We learnt through cognitive classes that generally human learn fastest through visual stimulation. We created this app so that we, as adults could learn new language with everyday objects around us using the 'see and say' method, just like children do.
What it does
ThingoLingo teaches new languages using 'see and say' method, User points their phone or web camera to an object and say the name of the object in target language. If the name of object is wrong, the app will then let user know the right answer.
How we built it
We took advantage of existing APIs in the market, using visual AI from Google to recognize the object and return array of possible object names. We then prompt the user to say the name of the object in the language they want to learn, converting it to text in our system using Bing Speech API from Microsoft's Cognitive Services. Next, we translated the text we received to English using Google Translate API. We then wrote a comparison algorithm to check if the word exist in the array.
Challenges we ran into
One of the most challenging part is to ensure that we could integrate seamlessly with all the APIs we used and understanding what each piece of code in the API does. We also had some trouble making camera access work for all browsers, phone and web.
Accomplishments that we're proud of
Although it is also the most challenging part, we are most proud of integrating many different APIs to our system for specific functions we want. We are also proud that we manage to code this program entirely using front-end languages.
What we learned
We learned that API can be unpredictable and it can feel really helpless when the issue is with the API as there is nothing much we can do with it. We should always use reliable and better documented API when writing our program.
What's next for ThingoLingo
We would integrate more languages and improve the design, especially for web, as well as making sure that it works for both front and back camera for phone. It would also be really great if we could have a phone app version.
Log in or sign up for Devpost to join the conversation.