We were inspired by dog lovers like us - curious and fascinated whenever we see a cool dog, but also eternally perplexed by the sheer number of dog breeds that exist.

What it does

What's that Doggo enables anyone to upload a picture of a dog, and then identifies exactly what breed the dog is!

How we built it

The front-end of the app was built using React.js.

Using an open-source labeled dataset of over 20,000 dog pictures and 120 breeds, we trained a deep learning model hosted on AWS Sagemaker's notebook cluster instances, and utilizing keras, sklearn, and tensorflow. We trained two convolutional neural networks with two different architectures and then combined those outputs into a logistic regression classifier to create a final multiclass classification model. We then deployed this model on AWS Sagemaker to expose an endpoint for our front-end to send requests to and gather responses.

We used AWS for most of our infrastructure; we created an API endpoint for the front-end to use; this enabled the web app to send PUT requests to the endpoint, which triggered a Lambda function that put the input picture onto S3. This then triggered our deployed model on SageMaker, which ran the model, gathered a prediction, and pushed a .json output onto S3. The web app then gathered the output and displayed the prediction and confidence interval to the user.


We ran into some challenges setting up our AWS stack, including S3 buckets permissions, API gateway endpoints, as well as SageMaker deployments. Moreover, training our model was difficult given the size of the dataset we were using; we would often run out of memory on our clusters. We were thus forced to compress our images before training to overcome that barrier. The data pipelines setup between AWS and our front-end was also a challenge.


We're proud of the architecture of our convolutional neural networks and our data preprocessing, as we were able to compress the data significantly (299 x 299) and still achieve a classification accuracy of 89.8% and a log-loss of 0.30.

What we learned

We learned a lot about the general AWS setup procedures, including ACLs and IAM, and we learned specifically how to utilize SageMaker to deploy a trained Machine Learning model. We also learned more about various techniques and architectures to optimize the performance of a CNN.


While experimenting on training photos, we realized that photos of humans would still accomplish some classification, albeit with a low confidence for all breeds of dogs. Rather than returning a "The system does not believe this is a dog, please submit a different photo" error, we took advantage of the moment -- now, you can use "What's that Doggo?" to identify the dog that you most look like. Or your friends, family, and even boss!

What's next for What's that Doggo?

After adding support for canine crossbreeds, we believe the only logical next step would be to aggressively classify the feline species.

Share this project: