Over 65% of the population are visual learners, and the improved technology of smartphones and AI can facilitate a whole new way for translating and learning a foreign language
What it does
Oblate lets you quickly and visually translate objects around you in real-time without internet. Along with translation, It supports transliteration for better understanding of pronunciations. It has offline support for 6 major languages to and from.
How I built it
The ObLate app was built with Java/Kotlin in Android studio, using Tensorflow for the image detection. We additionally tried to use the latest CameraX library and tensorflow lite for optimized and lag-free performance. Note: Models were pre-trained and built, as we would not have enough time to train them at a hackathon.
Challenges I ran into
The project is mix of different new technologies: Augmented Reality part (CameraX library release in 2019 Google IO) and Object detection part Firebase ML Kit and Tensorflow lite. The default ML Kit object detection can only classify the detected object into product category Ex: Food, Fashion, etc. Which was not enough for the application to work since we needed specific object name Ex: Apple, Shoes, etc. For this we needed to use the tensorflow lite model which is pre-trained for experimental or POC purpose. Connecting these parts and lack of guide on the documentation (some links were missing for examples, consulted with Google developers and confirmed the missing part)
Accomplishments that I'm proud of
The app works! ObLate uses object recognition techniques and puts them into practical use! We focused on implementing the small part of the application that is object recognition and translation and once it is working we implemented the login with google feature using firebase auth SDK. Later improved the translation API and cached the translations on the device to save the fetching time (Firestore DB and Microsoft Translation API).
What I learned
We learned more about different model architectures for objects recognition including Single Shot MultiBox Detecters, R-CNNs, Faster R-CNNS, and Masking R-CNNs. The new Tensorflow lite library and Firebase ML kit.
What's next for ObLate
We would like to build and train our own models on the ImageNet database for a more fine-tuned and streamlined experience. And use the Model hosting on Firebase to push updates to the app once new trained model is available which will drastically reduce the initial application size and make it more scalable.