Pantrific

Helping you take control of your health, one meal at a time.

Comment

Inspiration

Have you ever seen people who want to lose weight but can't stick to a diet? Have you seen people with nutritional deficiencies who cannot find the right food to eat? Or have you just seen people who cannot be bothered finding good recipes? Don't worry because we are all in the same boat!

What it does

Pantrific is a distributed set of tools that provides on-demand detection of pantry and food items in various locations around your house. Our frontend provides a mobile interface to view pantry items, set nutritional goals and note deficiencies, and visualise meals that match these parameters. Our app also supports multiple pantries and the tracking of macros - either through "consuming" a meal that has been presented or manually logging values.

How we built it

Object detection

50 iterations of training were applied to a yolo26 object detection model. These training images included common pantry and fridge items. This inference runs on a Raspberry Pi 5 in a custom build housing within the fridge. It uses a webcam to take pictures of the interior of the fridge.

Backend

We designed a REST compliant Fastify API to manage user data. It is accessed by the Raspberry Pi to upload pantry data, and user clients on the frontend to configure their preferences.

Frontend

Expo, a cross-platform app framework for JavaScript was used to implement the frontend. To improve the user experience, optimistic data fetching/mutation was used with TanStackc's React Query. We also added vector icons and chose to use clean yellow-themed UI design.

Pi Housing

We used CAD to design a housing to contain the raspberry pi. A pouch was added for a NFC chip, so users can tap their phone to open the app as well as openings for ethernet and webcam cables.

Challenges we ran into

  • Making a model that accurately identified objects in a fridge was very difficult. It was found that assembling and curating our own dataset was the best way to overcome this, as premade datasets were often bloated and partially irrelevant to our target use case.
  • Finding a suitable API to fetch meals from was difficult, as there were strict rate limits. We eventually settled on Spoonacular as it had a very acceptible starter tier for its API.

Accomplishments that we're proud of

  • Creating a custom-made object detection model that can accurately identify items in a pantry/fridge
  • Creating a polished website with showcases to advertise our product

What we learned

  • How to make object detection models
  • How to make a frontend with react native and Expo

What's next for Pantrific

  • OAuth providers for authentication on the frontend
  • Improved user security
  • Simplify npm scripts
  • Improve meal fetching efficiency
  • Consider claude/gemini to identify pantry items as they are far more reliable for identifying a more diverse range of pantry/fridge items.
  • Make a better logo and integrate it properly in the frontend
  • Improve web accessibility
Share this project:

Updates