Between our entire team, we speak or have limited proficiency in Russian, Hindi, Hebrew, Gujarati, Mandarin, French, Spanish, and of course, English. That is seven languages between three people, all born and raised in this country. How did we do it? We learned language while growing acclimated to our surroundings, albeit in other languages. The world around remained the same as we learned these new skills. You see, when someone holds up a red round object, there are millions of people in the world who think “manzana” before they think “apple”, but the juiciness of the fruit remains the same. That was the issue we wanted to face, how can we bring English into the lives of those who don’t understand, and vice versa as well. Similarly, we recognized that textbook learning is simply not practical, and even realistic for everyday. Our team members have also worked with students with learning disabilities, and we understood that students with learning disabilities also learn best from methods that involve the real world.

What it does

Our product takes language and the concept of learning a language and applies it to the real world. It is a practical and 21st century approach to learning by doing- the user is able to simply take a photograph of an object and they will be able to learn about in another language. FluentSee allows the computer to see and the computer to translate- so that you don’t have to. Ultimately, Fluentsee takes the real, tangible world, the world’s most understandable language, and presents it to the user using words and phrases that may seem more familiar. All this is done with the ease and convenience of a simple camera.

The user gets to see visible objects in both languages, a google search link for the words, a phonetic pronunciation for the foreign language, related words and common phrases for the object that is being detected.

How we built it

The application itself is built on Flask and then hosted locally. The server-side flask application communicates with about 13 different libraries and various APIs (including Google APIs) in order to receive a JSON with the information that will then be displayed to the user. The JSON is exported and then it is parsed by the Javascript and then it is displayed in our clean, user-friendly UI for the user to the information about the words that are being shown.

Challenges we ran into

The hardest challenge for us was the integration of all of the APIs and the communication between them If an API lacked a feature or did not have the capabilities that another had, we had to manually adjust for the rest in order to keep FluentSee one function product. Thus, not only did we have to communicate between APIs, but we had to pick up the slack for them as we fed the information into the next one, and this meant a lot of troubleshooting as even a simple syntax error would break the chain.

Accomplishments that we're proud of

At the end of the day, we are incredibly proud of the word pronunciation, because it meant that we had everything else completed. The pronunciation, for us, meant that we had substantiated the work that came before it. Before we could reach that stage, we had to establish our own product, and only then were we able to add this feature. Thus, we are proud of the cherry on top that we placed with allowing to phonetically see the language in English. It was the final touch and one that we felt was masterful in creating a complete and finished hack. Aside from that, we are incredibly proud of the new skills we learned, and the fact we were able to blend everything so seamlessly, including our object recognition, evaluation, and translation capabilities.

What we learned

The technical skills we learned included using Flask for python, which was a new framework to our team. But our growth was not just limited to what we were able to with the computer science aspects of it. We learned how to work as a team and experience team growth while creating something that was unique as well as interesting, something that will be used for a long term. We were able to combine a variety of different APIs, ML processes, and pretty UI in order to make something that we were proud of in the end and something that belonged to us completely. We also picked up quite a few non-english words along the way...

What's next for FluentSee

The next step will be to take the usefulness of FluenSee to the next level. E recognize the value that such a product could have in our society, from helping those who simply struggle with learning a new language to those with learning disabilities to even those who cannot see very well and rely on phonetic clues. We want to scale our product to make it as large as possible, in order to get to the root of why we created the product in the first place-- helping others. We understand that each culture has its own traditions and languages, and that language is a very nuanced subject. Thus, the next step will be to factor in the nuance and then scale the product as much as possible.

Share this project: