One of the hardest parts of learning a new language is memorizing the thousands of items and objects that exist in your life. Unlike grammar rules, there typically aren't general rules for translating nouns and even fluent speakers often have difficulty remembering the exacat name for a rarely seen object. Our project hopes to help people communicate through the objects in their lives, naturally learning the translations for "table" or "laptop" while the user is actually in the same room and even helping monolingual people easily build a basic foundation upon which to communicate with their peers in completely new languages.
What it does
Using your iphone's camera, we identify the objects that your are scanning in real time, allow the user to choose a custom language to translate the item's name into, and both speak and display the translated description of the object. For example our application can detect "A laptop sitting on a table" and in mere seconds tell the user that they can describe it as "Ein Laptop auf dem Tisch sitzen."
How I built it
We built the application in swift for ios 10, utilized microsoft's image recognition, google's translation services, and set up a java spring-boot server hosted on microsoft's azure in a tomcat virtual machine for future expansion.
Challenges I ran into
Several of the sdk's seem to be still underdevelopment and did not have all of the features that we required, so we had to juggle several related api calls to get the necessary information and experiment with local algorithms to minimize the utilization of limited api keys.
Accomplishments that I'm proud of
It works in real time and extends instant translations to not only text but actual objects too.
What I learned
How to use google's image recognition and translate apis, configuration and deployments to azure vm's , and some very interesting new words.
What's next for Mosaic
We want to finish development and deploy the application so that end users can begin to use it for language training and as a communication aid. As far as future features, we would like to add in internationalized utilities that aid in common problems when traveling. One such application would be detecting the name for the chinese yen and computing the exchange rate between yen and dollars based upon the user's source language and the predominant currency utilized by people in that language group.