Critter Capture

Snap, Discover, and Collect! Use AI to identify animals around you and build your own adorable virtual Critter Collection.

Comment

Inspiration

We were inspired by the way Pokémon Go gets people outside and exploring to collect Pokémon, so we wanted to bring that same idea to nature. With Critter Capture, you can discover real-world animals, snap a picture, and add them to your personal Critter Collection. It’s a fun way to explore the outdoors while learning more about the wildlife around you.

What it does

Critter Capture uses machine learning to recognize and identify animals, indexing them into a user-specific database with a fact sheet of each animal, including its conservation status. It’s efficient and informative, while also promoting awareness of animal endangerment and encouraging exploration and preservation of local wildlife.

How we built it

We built this project using the Django framework, Python, JavaScript, and TensorFlow. To identify animals in photos, we used a convolutional neural network with transfer learning, trained on the Animals-10 dataset. The project also uses the MediaDevices API to access the users camera, making it easy to capture and identify animals on the spot.

Challenges we ran into

  • Learning Django: None of us had much experience with Django before, so we had to learn the framework on the go. It was a bit of a challenge, but it helped us grow as developers. A particular challenge we faced was learning how to pass information between the front and backend between html pages.
  • Training the ML Model: We experimented with several different models on several different datasets many of which had problems. We originally wanted to use Zamba but the package currently doesn't build correctly on windows. We tried to use WSL/Ubuntu but setting up a new environment with all the necessary prerequisites was determined to take too long so we abandoned it. Next, we tried to use transfer learning on the Animals-90 dataset available on Kaggle. This dataset was very small, however with only 60 images per class and even with using data augmentation we were running into overfitting problems and long train times due to a lack of GPU (~1hr per epoch). Finally, we ended up using transfer learning on the Animals-10 dataset as a proof of concept as it is a much more comprehensive and simple dataset. Our final model is a finetuned version of MobileNetV3.

Accomplishments that we're proud of

  • Real-time Animal Recognition: We’re really proud of how we integrated machine learning with Django to let users instantly identify animals from photos they take.
  • Raising Awareness: We’re also proud of how the app promotes awareness of endangered species by providing interesting fact info about each animal.

What we learned

Our team had little prior experience with Django, which presented a learning curve. However, this challenge also provided us with an opportunity to expand our skill set and grow as developers.

What's Next for Critter Capture

  • Working with Conservation Groups: Next, we want to work with conservation authorities to upload data about endangered animals. This will help track where and when animals are spotted, which can support conservation efforts. We plan to do this by working with iNaturalist. iNaturalist is already used by nature enthusiasts to track animal sightings but is a more 'serious' app. Critter Capture is planned to integrate with the iNaturalist API to submit new sightings to support conservation work while providing a gamified experience.
  • Adding the Game: Right now we have only implemented the critter collection where you can view the critters you have found but we plan to make them more interactive with a full release. This is done to increase engagement and the bond people feel to their critters. Critters will be able to be added to your "Critter Island" when captured and from there you will be able to see the sprites walk around, interact with each other, and can be played with or fed with digital items. These digital items would be available in daily small quantities for free but additional and special ones can also be purchased with profits donated to conservation efforts.
  • Expanding the ML Model: We also want to improve our machine learning model to recognize more animals. The main challenge is getting a large database of labeled images for training as well as sufficient computing resources. We aim to partner with iNaturalist here again as they already have a massive database of labeled images with specific species names that could be used to train a more comprehensive model. The end goal would be for each critter you capture to identify the specific species or at least present probable options with pictures to select between in difficult cases.
  • Implementing Geolocation: In the future we hope to be able to use geolocation to include location information on the the users collection, as well as to provide information on endangered or vulnerable animals on the conservation page which is tailored to the users general area.
Share this project:

Updates