We were inspired to make this project when we walked around the city and saw a poster for a missing dog. We then did some research and realized that there are no good existing tools to help people find their missing pets, so we decided to create this project.

We also realized that this is a huge problem: 10 million cats and dogs are lost every year and 80% of them never come home.

What it does

We help pet owners find their lost dogs by combining crowd-sourced data with machine learning and geospatial search.

Shepherd allows anybody to upload a picture of a stray dog they encounter. This is something that is very easy to do and does not require a large time investment on the user. Then, a pet owner can easily search for images of their pet nearby by uploading a picture they have.

A combination of the location data and contact information of the person who originally uploaded the picture can then help an owner find their pet as quickly as possible.

How we built it

At its core, Shepherd is powered by a machine learning model that generates "fingerprints" (embeddings) of dogs based on images. These fingerprints are then indexed by our database, MongoDB, and used to find pictures of similar dogs quickly and accurately.

The next piece of this is geospatial search. By using device location, we can see where people uploaded pictures and then find pictures of similar dogs nearby.

Finally, our responsive user interface allows people to quickly and easily find their dog from any device.

How the ML model works

The model is a deep neural network using a ResNet architecture. It uses pre-trained ImageNet weights, but is then modified and fine-tuned on dog data. This results in a network that generates embeddings from images. These embeddings are then stored in MongoDB along with other metadata and the image URL. We then query this database for a similar image by finding the nearest neighbors in the embedding space.

Challenges we ran into

  • Subtle MongoDB bug that blocked us for several hours when finding the nearest neighbors.
  • We started with MongoDB Atlas, but it did not allow us to MapReduce and was too slow. We spent a lot of time trying to find a work around for this.

Accomplishments that we're proud of

  • The accuracy of our machine learning model
  • The speed of our geospatial search and image-comparison algorithms
  • The beautiful user interface

What we learned

  • We learned how to use MongoDB MapReduce
  • How big of a problem this actually is for pet owners without dog trackers/microchips

What's next for Shepherd

There are a number of exciting next steps we could take. One of these would be to integrate with existing pet shelter APIs to help users find lost pets (without microchips) picked up by a shelter. Another exciting step would be to generalize our model beyond just dogs and make it work for other types of pets as well. We could also add a tip system for people to reward the people that helped them find their pet. Finally, we could add a notification system that will alert a pet owner whenever a picture of a similar pet is uploaded.

Share this project: