Babel

Feature#1: Snap a picture
Feature#1: Upload picture
Feature#1: Language selection
Feature#1: Vocabulary list and flashcards for photo scan
Feature#1: Flashcards and actual objects
Feature#2: Record reading
Feature#2: Reading evaluation
Feature#2: Flashcards for reading

Inspiration

COVID-19 has deeply affected primary school education, and many states have registered large drops in Reading test scores. In a broader scope, language learning is incredibly difficult, so many people give up early, saying they don’t have enough time. With these problems in mind, we developed Babel, a language learning app that is optimal for all ages and fits well into daily life.

What it does

On a subway commute and have down time? By simply snapping a picture of the scenery, Babel detects objects in the image and stores them as flashcards (btwn. native language and translated language). After snapping a picture, Babel uses Google Translation API to let you select the language that you want to learn.

Another feature of Babel is the language reading practice, which lets you randomly choose a piece of text to practice your reading ability. It then gives you a score, and generates flashcards that give you the definition, part of speech, translation of the words you missed in the reading passage.

Google API’s Used

Google Cloud Translation API, Google Natural Language API (syntax analysis), Google Vision API (Multiple Object Detection)

Feature Number 1: Flashcard Object Detection

The Flashcard Object Detection processes objects in a photo scan and transforms those objects into vocabulary flashcards of any language you choose. This allows users to learn new vocabulary anytime and anywhere.

How We Built It: Flashcard Object Detection

Using Google Vision API, we performed Multiple Object Detection on a user photograph. Babel then stores the objects in the image as vocabulary words in a backend Django database. With the Google Translation API, Babel translates the newly generated words into the user’s learning language of choice. On the front of the flashcard we have the English version of the word portrayed, and on the back of the flashcard, we have the translated version. At the top of the page, we also have a translated list of all object images scanned by the user, which includes terms from all user scans (not just the most recent scan).

Feature Number 2: Reading Accuracy Analysis

For anyone polishing their reading skills, Babel lets you read a given text aloud and generates a transcript of your reading. With this instant feedback, language learners can identify each word they missed and review them with flashcards. The missed words are highlighted red, and flashcards are generated for each of the missed words. In addition a backend algorithm generates your overall score out of 100. Another added feature we have is the flashcard coloring scheme, a darker shade of blue represents words that are more common in the English language, and a lighter shade of blue represents words that are less common in the English Language.

How We Built It: Reading Accuracy Analysis

Mozilla’s Speech Recognition API: With this API, we were able to use Javascript to render out a user recording submission. Google Natural Language(Syntax Analysis) + Google Translation API: With Google’s Natural Language API, we analyzed and generated parts of speech for each of the words, which we believe is an integral part of vocab studying. The Translation API generates a version of the vocab term in the user’s native language, which is a helpful reference to language learners. Beautiful Soup Web Scraping: With beautiful soup, we were able to web scrape and sort through the various definitions from vocabulary.com.

Complex Text Analysis & Grading: In order to compare the accuracy between two texts, the frequency of words in a dictionary was compared in an algorithm. For each word, the difference between its actual and real frequency is calculated into a weighted error score. Based on this, the user receives a grade on how well they read the text. With the incorrect words, the correct words are traced back to using the list and appropriate flashcards are generated. By using a database containing over 90,000 words, the flashcard’s color corresponds to how common it is in use in the English language.

Challenges we ran into

When making Babel, we met a lot of challenges. Since we used Mozilla’s Speech Recognition API, Google's Vision API, Google Natural Language API, and Google Translation API for the first time, we needed to learn as we built. Moreover, we struggled with getting the flashcard interface to connect with the object detection with the backend in Django.

Accomplishments that we're proud of

We are proud of integrating Mozilla’s Speech Recognition API and all the various Google APIs for the first time. We are also proud of coming up with the algorithms to analyze the frequency of usage of each word and creating a dynamic flashcard based off of it, scrapping an online dictionary, and building a user-friendly interface.

What we learned

We learned a lot about Google’s Vision API to perform Multiple Object Detection, Google Translation API, and Google Natural Language API, and web scraping. We also became more familiar with using Django framework, and using HTML, CSS, etc to format our webpage. Moreover, we were able to collaborate with VS Code’s live share to improve our efficiency.