Scales and Tails

Transforming Species Identification and Expression Analysis with AI

Comment

Our Video Link: https://docs.google.com/document/d/1c31wvLRhNeIyQjo26DnlgH9DoxrxGi_WGcbck4QjLFY/edit?usp=sharing Includes presentation and Demo.

Inspiration

One of our goals is to address one of the greatest difficulties for wildlife researchers. Researchers have difficulties collecting accurate field data for various endangered species. So, by implementing our convolutional neural network on a set of Fish species native to a region (in this case Northern Europe), we can put field data collection in the hands of the general public and fishermen. Through our application, a fisherman who catches a fish that they do not recognize can take a picture of their fish, upload it to our site, which will then identify the fish, giving them the appropriate information to take correct action with their catch to protect the local fishery. In the future, we will build this out so catches can be logged and located on a map, to help in tracking fish migration changes in regions badly affected by climate change.

Another one of our goals for this project is to develop a software designed to assist the visually impaired by providing them with the ability to experience aspects of the world that people with normal vision often take for granted, such as understanding the facial expressions of cats or identifying different species of fish while engaging in their hobbies like fishing. This software aims to bridge the gap, allowing those with visual impairments to better connect with their environment and enjoy these experiences through innovative technology.

What it does

Our project consists of two parts: a fish species indentifier and a cat emotions identifier. Both of these projects focus on using a CNN network to analyze user-uploaded photos.

If a photo of a fish is sent to the fish identifier demo, then the CNN will determine the fish species based 23 different species of common game fish found in the Netherlands and will print the ouput to the browser.

If a photo of a cat is sent to the cat identifier demo, then the CNN will determine the fish species based on 6 emotions: sad, angry, happy, normal, surprised, scared and will print the ouput to the browser.

How we built it

We built 2 separate ML using CNN, one for the fish identifier and one for the cat emotion identifier. We then built a website using React.js and Bootstrap that allowed users to upload photos and to display the results of the neural networks.

Challenges we ran into

Setting up the correct libraries and environments were difficult due to many incompatable versions of certain libraries and compontents. Many of the features and components of different versions of Tensorflow and Keras are incompatible with eachother, and much of our time was spent troubleshooting issues like reconfiguring the validation data separation through cv2 and numpy because the pillow framework didn't match our Keras implementation.

As far as broader data-related issues go, we were processing very abstract images, which came with a myriad of problems. Different cat breeds were easier to flag than actual facial expression differences due to the nature of CNN pooling, and the dataset had to be adjusted to provide more consistency. We were surprised to see that lowering the resolution of the images also helped solving this issue. as it reduced some of the issues with noise that led to inconsistency between the accuracy and val_accuracy of the model.

In the fish identification section, we ran into problems again with the dataset leading to the flagging of traits besides the fish, like how a hand would fill up a large portion of the screen on smaller fish and be identified as a more prominent feature than the fish itself. This, along with file size constraints, were solved by tweaking the image size, batch size, and epoch size of our model until we saw decent results. Also, for some fish like Bream and Dace in families that contain multiple species, we opted to make them one broad category to allow for better differentiation in the model.

Accomplishments that we're proud of

Our tedious work on the fish identifier led to roughly 70% raw accuracy and 47% val accuracy, which, considering the complexity of the data and number of categories (23 species), greatly exceeded our expectations. It also took 20+ minutes to train this set each time, so we were very happy to see this result!

The cat emotion model showed 89% raw accuracy and 33% val accuracy which, while not anywhere near perfect, showed strong enough statistical significance that it could be used functionally in more complex applications with a bit more tweaking. It is also really silly, which we are proud of. And it can tell that grumpy cat is grumpy, which is definitely results.

We also just worked great as a team! We met yesterday for the first time and come from 2 different schools (UM Dearborn and Ann Arbor), so it was really cool to collaborate and create such a cool deliverable with such kind people from our sister school.

What we learned

Machine learning is hard. We introduced ourselves to 2 new ML frameworks, got a better understanding of the integration of backend and frontend in React, learned how to set up local environments for training our models, and basically everything else you see in our project. There was no component in our project that all of us had used before, so it was really a great learning experience for everyone. Also Owen learned to write down his sudo password this time.

What's next for Scales and Tails

For the fish program, we would like to integrate a backend database with coordinates, timestamps, and other relevant information from user input so that fishermen could provide research grade information in using the app. Additionally, we would like collect data to train it on the set of fish local to michigan, and see how well we could apply it to help endangered species in our local community.

For the cat emotion program, we would like to connect it to video! Considering the results, our program could perform quite effectively on an optimized live-video response system that would make decisions based on pooled results of multiple images, and we think it would be awesome if everyone could take a video of their cat and tell them when they are happy. This would also have great applications for visually impaired people interacting with animals in the home, and with adaptive text to speech features could be a really great service.

Built With

Share this project:

Updates