Inspiration
We love Hiking, but often, we don't have network connectivity while hiking, and it becomes impossible to learn about all the species of nature around us. This project aims to provide a tool for researching and learning about nature and wildlife while being in nature in real-time.
What it does
Before your adventure, mark any region on the map and instantly access species data from top natural research sources. While you're out in nature, TerraScout uses your live photos to provide both common and scientific information about plants, animals, and more—all without needing network connectivity.
How we built it
The project contains the following modules:
Data Fetching and Preprocessing: This module is responsible for aggregating data on a given geographic location from several sources dynamically and processing it to be accessed by the user as well as reference by the Computer Vision Models.
Computer Vision: We use a combination of MobileNetV2 and EfficientNetB3 to identify 1000s of species of plants and animals. First, we finetuned the MobileNetV2 model to classify and detect objects in nature. Then we use a multi-headed fine-tuned EfficientNetB3 to identify species and Scientific Names for the given images.
Backend: It fetches, processes, and provides APIs to access the data locally and performs inference on the Computer Vision Models. It also works fully locally without the internet once the data is ready.
Frontend: We created a Web UI to demonstrate the workflow of the application for the Demo. This Demo allows the User to select a region anywhere in the Sequoia National Forest area. Once selected, we can click a Picture using the Webcam to simulate a photo taken with a mobile phone in Nature. Then it communicates with the backend and provides data about the detected species.
Challenges we ran into
We ran into several challenges while working on this project. Firstly, we had to find a way to gather the data locally based on regions, After intense research, We were able to find sources that provide exports of data and BULK API calls to gather all the data we needed.
Secondly, we ran into various problems while fine-tuning the computer vision models which took the majority of our time to figure out the best way to train and deploy these models locally.
We also had challenges with integration and bugs involving several moving parts, but we were able to squash most of them by working collaboratively.
Accomplishments that we're proud of
We are proud to provide a working demo of the entire application that works locally by running all modules of the stack on the device with response times in milliseconds.
What we learned
We learned about the whole host of research labs and groups that work tirelessly to aggregate data on hundreds of thousands of species of plants and animals while fostering communities such that researchers and nature enthusiasts can support and add to the knowledge.
Working on this project provided us with knowledge about the entire stack of an AI application, as well as overcoming the hardships of running everything locally gave us a deeper understanding of the libraries and tools we use.
What's next for TerraScout
We want to build a Mobile App with a sleek UI and Improve our AI models to identify more species with better accuracy. We also want to expand the Geographic regions for which we can gather data.
Log in or sign up for Devpost to join the conversation.