Inspiration

As frequent users of the dining hall, we have noticed that the dining halls have a consistent rotation for meals that are served. However, a look at the compost bins after a meal will show that some food items are certainly less well-received than other items.

What it does

Uses computer vision to estimate the amount of waste on each person's plate before they throw it out. Displays summarized information on a dashboard, and uses AI powered recommendations, which also consider dietary restrictions and nutrition, to suggest changes for the dining hall to implement.

How we built it

We used NextJS and FastAPI for the dashboard. The computer vision system uses an iPhone camera that streams over an IP address to a computer that can process the images. We fine tuned a YOLOv8 segmenting model to detect food on the place, and then wrote equations to estimate how much waste is on each plate from the computer vision output. The computer vision pipeline then stores information in a SpacetimeDB which the dashboard can then pull from. We are using GrokBot for a screen reader to improve accessibility for our project.

Challenges we ran into

One issue was the frame rate and latency of the computer vision system that was originally constantly running the YOLOv8 segmenting model. This made the system extremely unreliable as it would miss the plates that were coming through. Our solution to this was to remove the YOLOv8 model from the stream, and take snapshots whenever a new plate is presented and push that image into a queue for the laptop to process in the background. This works because we do not need the data immediately; rather, we just need it by the end of the day so that the dining hall staff can look over it and see what changes need to be made.

Accomplishments that we're proud of

We were able to build a system that is as lightweight as possible (minimal effort needed on the dining hall's end) that also delivers quality results and actionable items.