debugging process 1
debugging process 2
The inspiration for TranslatAR comes from the desire to understand different languages in order to better connect with people around the world and wanting to implement AR into our application for a complete interactive experience. Learning a new language can be extremely difficult and nowadays people are having a hard time finding excitement in learning a new language. With TranslatAR, we are able to reinvigorate people's interest and passion to learning a new language, and hence connecting people around the world.
What it does
TranslatAR uses the iPhone's camera to detect objects by our own custom trained model from Microsoft's Cognitive Custom Vision API. Once the objects are detected, they will be able to be translated into the user's selected language in real time using Augmented Reality. What's unique about our AR app, is that the words are anchored to the object in space, therefore identifying and creating a "label" in a unique environment for users to learn.
How we built it
We built our application using primarily Microsoft applications. Microsoft Cognitive Services that were used includes, Microsoft translator API and custom vision model API. Other programs used included Micrsoft Azure, ARkit, and Swift.
Challenges we ran into
We had no prior experience building a mobile app using Swift before. We wanted to use Swift because of their new ARKit. As you might have predict, we ran into many challenges with understanding and programming with an unfamiliar language and being able to communicate between new partners. Challenges in Swift included embedding Microsoft API's within the code because of lack of code documentation for Swift4. Other challenges included training our API to make precise and accurate prediction over 95% of the time. We had to train more than 50 instances of an object.
Accomplishments that we're proud of
We are proud of our progress we made while utilizing a completely unfamiliar language. Implementing AR into our application was very fun to do and something we feel many users will enjoy. We are also very proud of being able to learn, have fun, and meet new people while at this hackathon!
What we learned
We learned a completely new language and were able to overcome obstacles intertwined with learning an unfamiliar topic in a short period of time. This allowed us to develop our skills in better understanding language we are unfamiliar with.
What's next for TranslatAR
We want to continue to add more languages to the application so there are less barriers in connecting and understanding different parts of the world. Future endeavors can include expanding API recognition to more objects and adding extra features such as practice phrases and speech pronunciation. With response capabilities, and better UI/UX functionality we believe that TranslatAR can truly change the way we learn.