Lingo

Inspiration

Millions of immigrants coming to the United States for better opportunities face steep learning curves picking up difficult languages, like English. We've researched that connecting with language visually and having the chance to connect new words and phrases to ones from your native language is a huge help in developing a comprehensive vocabulary. Being able to identify the things you interact with on a daily basis is a great first step.

What it does

Lingo is an augmented reality iOS app where users struggling with English can learn the language within their everyday life through the objects right in front of them. A user simply needs to open up the app and tap the screen to both see and hear the English translation of the anything they can find. We currently have implemented nine languages for non-native English speakers.

How I built it

We built Lingo using Swift and a variety of APIs including Google Translate, Apple's CoreML in addition to their augmented reality tool kit.

Challenges I ran into

Neither of us have ever worked with augmented reality or the Apple's machine learning framework. We ran into a lot of problems combining the augmented reality features and the machine learning framework to identify the objects through the camera lens. We also ran into problems with the MobileNet model we used to identify objects.

Accomplishments that I'm proud of

First and foremost, we're proud of getting our initial idea completed as we first imagined it and how our idea can make a huge impact on other's lives.

What I learned

We learned a lot about what goes into developing both augmented reality and machine learning applications. We also learned a lot about task management when deciding who should work on what.

What's next for Lingo

We want to expand Lingo and implement features like a visual vocabulary list for users to come back to words they'd like to review in the future. Repeated exposure to vocabulary is great for retention. In addition, we'd love to continue adding additional languages and translation support. The more objects Lingo can recognize, the more words we can teach the world.

Built With

• augmented-reality
• coreml
• google-translate
• swift
• vision
• xcode