ShelfWise

Inspiration

The inspiration for creating ShelfWise came from the daily challenges college students face. With busy schedules juggling school, work, and extracurriculars, it’s easy to lose track of groceries, leading to spoiled food, wasted money, and more frequent (and often costly) dining out. ShelfWise makes pantry management fast, convenient, and budget-friendly. With just a quick receipt scan or a prompt to the ShelfMate smart assistant, students can effortlessly track food items, monitor expiration dates, and receive timely notifications about expiring food. This way, they can avoid waste, save money, and enjoy more home-cooked meals—no extra hassle required.

What it does

ShelfWise uses computer vision to scan receipts, automatically adding food items and their expiration dates to a user’s virtual pantry. This streamlines the process, making it easy to keep track of what’s in stock. The app then generates reminders for upcoming expirations, helping users avoid food waste. Additionally, ShelfWise can suggest recipes tailored to the contents of the pantry, offering options across various cuisines, difficulty levels, and preparation times. For even greater convenience, users can manage their pantry through natural conversation with the ShelfMate smart assistant, making it simple to update items, check expirations, and find recipe ideas hands-free.

How we built it

We built ShelfWise by combining a variety of powerful technologies to create a seamless, user-friendly app for pantry management. Python, OpenCV, and Tesseract form the backbone of the receipt-scanning feature, allowing us to leverage computer vision to detect and extract food item details with high accuracy. FastAPI provides a fast, reliable framework for developing our API, ensuring efficient communication between the frontend and backend components.

For data management, PostgreSQL serves as our robust database, while Prisma acts as an ORM, making database queries and updates quick and intuitive. On the frontend, we used React and TypeScript to create a dynamic, responsive user interface that enhances user experience, making the app easy to navigate. NextJS powers the backend logic, coordinating data flow between services and supporting complex interactions with the database and machine learning models.

To enhance user interaction with natural language, the ShelfMate smart assistant is powered by OpenAI’s language model, fine-tuned with custom prompts to generate data for the app based on user inputs. This tailored prompt design allows users to manage their pantry effortlessly through conversational prompts, enabling the model to respond intelligently by adding items, checking expiration dates, and suggesting recipes. This combination of technologies creates a cohesive, interactive experience—from scanning receipts to generating reminders and recipe suggestions—making pantry management seamless and efficient for busy users.

Challenges we ran into

We encountered several challenges while developing ShelfWise, from managing errors in AI prompt handling to integrating diverse data types for API endpoints. Navigating the technical details of data integration was complex, especially as we worked to ensure seamless interactions between our API, database, and front end. We also faced the challenge of developing efficient preprocessing algorithms for receipt scanning, which required careful tuning to ensure accuracy.

Alongside these technical hurdles, we were also balancing our academic responsibilities, including exams, which added pressure. Learning new technologies—such as FastAPI, Prisma, and the intricacies of language model prompts—was an additional challenge, but we approached it as a team. By working together, we supported each other in learning new concepts and troubleshooting issues, sharing our knowledge of various tools and technologies. This collaborative approach helped us make steady progress and turn obstacles into valuable learning experiences.

Accomplishments that we're proud of

We’re proud of how ShelfWise has come together as a fully functional app, working end-to-end with an efficient, cohesive design. From OCR-powered receipt scanning to NLP-driven pantry management and a user-friendly frontend, each piece integrates seamlessly to create an intuitive, impactful experience.

More than just a technical accomplishment, we’re especially proud that ShelfWise can help students and people around the world keep track of their groceries, reduce unnecessary food expenses, and combat food waste. By providing an easy way to manage food inventories and stay mindful of expiration dates, ShelfWise has the potential to make a positive impact both financially and environmentally, empowering users to make the most of their resources.

What we learned

In the early stages of development, we struggled with data type integration for API handoffs. Since we were working with different technologies—FastAPI, Prisma, PostgreSQL, and React—ensuring that data was properly formatted and seamlessly passed between the frontend and backend was a challenge. We faced issues with data types not matching, leading to unexpected behavior and bugs.

However, as we continued working through the project, we became more familiar with the nuances of each tool and learned to better handle data type conversions and validations. Through constant testing, debugging, and refining our understanding of each technology, we became better at managing these handoffs. Over time, we developed a more structured approach to ensure accurate data transfer between the components. This experience not only helped us improve our technical skills but also reinforced the importance of thorough planning and testing when working with complex systems.

What's next for ShelfWise

We plan to enhance core features to further improve the user experience and expand the app’s capabilities. We plan to implement learned preferences for recipe generation. By analyzing user behavior, including favorite cuisines, ingredients, and cooking times, we will enable the app to suggest more personalized recipes tailored to individual tastes and dietary needs. We also aim to make recipe generation faster. This will make the recipe suggestions even more relevant and convenient for users.

We also aim to refine the computer vision (CV) aspect, improving its accuracy and efficiency for receipt scanning and food item recognition. We’ll focus on enhancing the model’s ability to handle various receipt formats and better detect expiration dates, ensuring that the app can adapt to a wider range of user inputs.

In addition, we aim to improve the natural language processing (NLP) capabilities of the ShelfMate smart assistant. By fine-tuning the assistant’s understanding of user queries and responses, we can make pantry management even more intuitive and seamless, with more natural and accurate conversational interactions.

Lastly, we’re excited to continue refining and enhancing ShelfWise as standalone native mobile apps. With the app already available on mobile platforms, we’re focused on making it even more accessible and user-friendly for users on the go. This allows them to easily manage their pantry, scan receipts, receive expiration reminders, and explore recipes directly from their phones. By optimizing the mobile experience, we aim to reach a larger audience and further our mission to reduce food waste and save users money.