what's in my fridge ?

Our story

Have you ever stared at food in your fridge, unsure if it’s still safe to eat? We all know how easy it is to toss something out when we’re not sure—especially with those confusing “best by” dates. Food waste is a massive issue, and a lot of it happens because we don’t have a clear way to judge whether something is still good. That’s when we thought: what if we could build an app that uses AI to help people make smarter food choices and reduce waste?

Spark of Inspiration

The idea sparked from a simple realization: people waste food because they don’t always know if it’s safe to eat or not. “Best by” dates are helpful, but they don't give the full picture. We wanted to create an app that analyzes food images to determine if the food is still good or should be thrown away, cutting down on waste and saving people money in the process. This would make it easier for people to make informed, sustainable choices about the food in their homes.

What it does

Our app helps users reduce food waste by analyzing photos of their food to determine if it’s still safe to eat. Instead of relying on confusing "best by" dates, users simply upload a picture of the food they’re unsure about. The app uses OpenAI’s API to analyze the image and look for signs of spoilage, like mold, discoloration, or fungus. Based on this analysis, the app provides a verdict: "Safe to Eat" or "Throw Away." This helps users make more informed decisions, reduce waste, and save money by ensuring they don’t toss out perfectly good food.

How we built it

1. Frontend: Interactive User Interface

We used Next.js to build a clean, responsive user interface where users can easily interact with the app. The main features allow users to: i. Upload grocery pictures and have the app list the items in them. ii. Add new grocery items by either uploading a photo or using their webcam. iii. View an analysis of individual items with spoilage detection and recommended usage timelines. The interface is designed to be intuitive and user-friendly, making it easy for anyone to quickly upload images and get detailed feedback on their groceries.

2. Backend: Handling Image Uploads and Processing

We used Python Flask for the backend to handle image uploads and process them. When a user uploads a picture, the Flask server interacts with the OpenAI API to:

i. Detect Items in the Image: The app extracts grocery items from the uploaded image using the OpenAI API’s powerful object recognition features. The app then lists all identified items for the user. ii. Estimate the Usage Window: Based on the items detected, the app also shows an average number of days before the items need to be used or will likely spoil, helping users plan better. iii. Add New Items: Users can add new items either by uploading additional pictures or using their webcam to capture items on the go.

3. AI-Powered Analysis with OpenAI API

We integrated OpenAI’s API to process and analyze the images. The AI helps:

i. Identify Food Items: It identifies and lists the grocery items in the image by analyzing the visual content. ii. Predict the Shelf Life: Based on each item, the app estimates an average number of days before the food will likely spoil. This helps users know when they need to consume it to avoid waste. iii. Analyze Individual Items for Spoilage: The app can analyze specific items individually and give a verdict on whether they’re safe to eat or should be discarded, using spoilage detection algorithms.

4. User-Friendly Features for Adding New Items

We designed the app to allow users to add new grocery items at any time: i. Photo Upload: Users can simply upload a picture of any new item. ii. Webcam Integration: For convenience, the app supports webcam integration, allowing users to instantly capture new grocery items in real-time. The combination of image upload and webcam support makes the app flexible and easy to use, catering to a variety of user needs.

5. Database: Storing Data

We used MongoDB to store the uploaded images and data about the grocery items, including their names, expiry dates and user information.

6. Testing with Postman

To ensure the app functions as expected, we used Postman to test the API endpoints and ensure proper communication between the frontend, backend, and OpenAI API. This step helped us troubleshoot and fine-tune the app’s performance.

Challenges we ran into

One of the main challenges we ran into was to implement the part in MongoDB where the "best by" date of the product has approached, however the product is still good and the time to remove that object from the database needs to be extended. That main issue was to make it work with the notification that is provided one day before the expires.

Accomplishments that we're proud of

We are proud of efficiently integrating OpenAI's API to create agents to accurately identify if the food is safe to eat, avoiding unnecessary wastage.

What we learned

We learned that when used smartly and efficiently, AI tools could really help transform how we think about sustainability by assisting in minor tasks like this, the average wastage produced by each person could be reduced significantly.

What's next for what's in my fridge ?

The next step of What's in my fridge is to scale the product, increase accuracy by updating the database with the best by dates that comes with the items by just scanning their barcode to more accurately give an estimate on how much longer the items can be used for. Also making the UI more user friendly and improving the notification service.

1. User Notifications

We can easily set up email notifications to remind users when their food is about to spoil or suggest using up certain items. Using WebSockets or Cron jobs for push notifications could also be a great way to engage users in real time, keeping them informed about the status of their groceries.

2. Mobile App Development

While building a mobile app might take time, it’s totally achievable using frameworks like React Native or Flutter, which would allow us to quickly build apps for both iOS and Android. This would also allow users to interact with the app even more easily.

3. Sustainability Dashboard

We could develop a dashboard that shows users how much food they've saved from being wasted. By simply tracking expiry dates and usage, this feature is highly doable and would provide users with tangible results to keep them motivated.

Demo Video Link: https://drive.google.com/file/d/1D6k4Bo6Kug-_JHrxlCy3F1hfHHnZUaOr/view?usp=sharing