Inspiration
As college students living away from home with nobody to help us make sure food doesn't go bad, we tend to forget about the groceries we bought and leave vegetables in fridge drawers for weeks, just waiting to spoil. We wanted to make something that would help us be sure that the food we buy doesn't go to waste.
What it does
The user can hold their receipt up to the webcam, and the app will scan the receipt and output a file with a list of recipes they can make with the ingredients they have.
How we built it
We split the project into 3 key parts. First, we utilized computer vision for image to text processing. The text was then parsed and filtered to output a list of ingredients. Second, we constructed a dataset that maps a list of ingredients to recipes using pandas data frames. Third, we developed an algorithm that takes the outputted list of ingredients and runs it against the dataset to determine which recipes can be made.
Challenges we ran into
Our algorithm was performing poorly, so we began to brainstorm what was going on. We realized that if there was a recipe that took 7 ingredients to make but our receipt showed we had 4 of them, our algorithm would give this recipe a score of 4. Whereas if there was a recipe that took 3 ingredients and we had all of them, our algorithm would give this recipe a score of 3. We realized we needed to normalize these scores by dividing by the total number of ingredients per recipe. We patched the bug so that the score for the first recipe would be 4/7 and the score for the second recipe would be 1. We started to get much better performance.
Accomplishments that we're proud of
We were excited to scan the receipt and parse the words directly after we ran the code. Then we added the capabilities to open a camera on our laptop to take a picture of the receipt.
What we learned
We learned that it was a lot easier to create this program than we thought. We learned how to use OpenCV in order to open the camera on our computer. We also used Pillow in order to output our image, which we didn't know how to do before.
What's next for Receipt to Recipes
We want to use ML in order to scan ingredients in the fridge that we can then use to output recipes using that data. Also, stores usually shorten their items on the receipt (for example, "chicken" might be written as "chikn" on the receipt), so we can use ML to find out the most likely ingredient it can be.
Log in or sign up for Devpost to join the conversation.