An offline image recognition tool and travel companion, powered by EdgeML.
An overview of our design. Also available at: https://imgur.com/utmHOlD
A Golden Lion Tamarin.
The Christ the Redeemer Monument.
A post that was automatically generated on Instagram using Hootsuite.
Share your travel diary with ONE CLICK!
Technical Notes: Runes by HOT-G was at the heart of our EdgeML implementation, allowing multiple imagenet models to be run locally on phones. Most importantly, Runes are highly performant and provide quick but surprisingly accurate results (depending on the model of course). During our rapid development, the use of Rune files was especially critical to quickly and effectively materialize our ideas from TensorFlow models to Explor extensions. For more technical details, learn more below!
The Hootsuite API enabled our travel diary function to share sentimental stories with friends and families across 20+ platforms. Specifically, the REST API was well-documented and powerfully simple to use, eliminating the complexities in supporting each social media platform individually.
Without these tools, our solution would be definitely more complex to build.
With the gradual reopening of borders around the globe, the dreams of travelling to a rainforest or a natural wonder have become a reality once again. It is an ideal time to plan your next vacation to these exotic destinations. For example, rainforests are great places for adventure, photography and wildlife watching. However, many remote areas of wilderness have poor or no internet access.
Imagine taking a photo of an unknown species you discovered in the Amazon Rainforest. You want to figure out what it is. Many ML-based apps do not work offline, as they rely on powerful, resource-intensive computers on the cloud. So why can’t you carry those Machine Learning models in your pocket? Yes, you can! Explor is powered by fast and lightweight Runes stored right on your device.
With the impressive performance of Runes, you can shoot photos of objects such as birds, animals and landmarks and get instant information about them while you are disconnected from the internet. It records memorable moments of your journey in the wild. Once you are reconnected, it lets you share your travel diaries easily with your friends and family in multiple social media channels all at once. All of these features make Explor a perfect travel companion for your next trip to the wilderness.
What it does
Explor enables you to:
- Explore the wilderness without worrying about losing mobile signal.
- Rapidly analyze live animals, plants, species, landmarks and other objects of * interest using EdgeML models (as Runes) stored on your device.
- Record your encounters with different species, landmarks and other objects in your Travel Diary.
- Post your Travel Diary to multiple social media accounts all at once.
How we built it
- We used Ionic 5 and Angular 12 to scaffold the UI with one codebase for the web/PWA, Android and iOS. We then used Capacitor to deploy it as a native mobile app.
- We supercharged the app with EdgeML using Runes.
- We also wrote an API using Golang which interfaces and serves various components running on GCP
- We deployed multiple Tensorflow machine learning models onto Edge devices, specifically the phone.
- We used the Hootsuite API to conveniently share travel diaries on 20+ social media platforms.
Accomplishments that we're proud of
- Leveraged powerful ML libraries to create a promising app.
- Pushed the limit of low-power edge devices to run EdgeML algorithms.
- Achieved our design goals given limited time.
What we learned
- Leveraged Rune to efficiently build and deploy TensorFlow Lite models.
- Used Hootsuite’s API to share memorable photos on social media.
- Used Ionic/Angular and other powerful libraries to quickly scaffold our application.
What's next for Explor
- Expand object detection functionalities through additional EdgeML plugins (powered by Rune).
- Add support for recording and sharing videos.