When we look at a block of text, our brain will automatically find the logic within the texts. It may try to recognize some words and phrases to begin with. Then, the sensation moves up to read some higher-level structures like sentences and paragraphs. What if we can use the popular web language, aka Emoji, to facilitate some of this process?
What it does
It uses the phone camera and recognizes the words live-time on the screen, and automatically replaces words that can be translated into emoji. It also supports uploading pictures from photo library, and taking photos with the app.
How we built it
We created an iOS app, using Swift, Objective-C. We imported firebase libraries that allowed us to convert detected text into strings, and built machine learning functions that allows a larger word bank of text-to-emoji translation.
Challenges we ran into
We started with developing an app that recognizes a menu and displays the ratings for each item through the data of how many transactions a certain food item was purchased. After investing a lot of time, we realized the data provided was limited. We had to change directions while still trying to utilize the partially developed app that recognized text. After we implemented the function of text-recognition, it took us a good while to figure out how to process the text. We eventually thought it was best to implement a test-to-emoji library with our live time text recognition. Like all teams, we encountered hundreds of errors and hours of trying to debug.
Accomplishments that we're proud of
We are able to build a functioning app that uses the iPhone camera to translate words into emojis live through text recognition. We are particularly satisfied with our implementation of the Firebase ML Kit, provided by Google. There was a lot of reading, trial and error, and learning in the process as we learnt to incorporate these concepts into working Swift code, and eventually an entertaining app. As a group of first-time participants, we are proud to be able to incorporate all our passion and creativity into a deliverable product within 36 hours.
What we learned
When talking about incorporating complex concepts like machine learning and natural language processing, it sounds intimidating. From this process, although very challenging, we were able to manage and learn through the plethora of online resources.
What's next for TextMoji
The framework for text-recognition is a very primitive application from the ML toolkit. The stability can be improved to facilitate user experience. We hope to further develop this app so that it can be more applicable to general users and texting. Plain text reading is mundane, and when emojis are used correctly it brings words to life!