Successfully detected snickers
Successfully detected skittles
Successfully detected twix

Inspiration

In the past, we have created an app that helps blind people navigate rooms. It is common for apps to help with navigation, since that is usually the direct result of blindness, but other important aspect like nutrition becomes overlooked. Especially during a time where candy is everywhere (Halloween!) we want to create a way to make sure that blind people can identify types of candy.

What it does

Our app uses a device's native camera to try to identify candies shown. The device will give a confidence of what it think that type of candy is, and if it passes a certain confidence threshold, it will audibly speak out the type of candy so that blind people can interpret it.

How we built it

We used python for the machine learning mode and backend. Starting off with the model, we collected data first and consolidated everything into a single folder. Then, we trained the model and had to test several iterations of different types of models to choose the best one. After tuning the model, we implemented some thresholds for real world applications and inserted into a frontend.

Challenges we ran into

One of the problems was choosing the right model for the AI. At first we used pytorch's linear model, which is poor for the type of data we were training on. Since the data we collected included color and was high dimesnions, we had to use a convelutional neural network. Ultimately, training the network on the data we collected is still not enough to create an accurate model. We finally landed on using a premade model, where we training on our data and reaching proper accuracies.

Accomplishments that we're proud of

We are proud that the model we training based on the Resnet18 is able to identify the testing data with 100% accuracy in some test. Getting this model to work with open CV is also a big milestone, where it was able to accurately identify 3 different types of candies with a properly tuned confidence threshold.

What we learned

We learned how to use Resnet18 and pytorch with custom data that we collected ourselves. This is useful since it lets us be able to train other AIs in the future and is not limited to the datasets that are available online.