Inspiration
The inspiration for Squash came from the need to streamline and simplify how we track groceries and manage food inventory at home. Often, people struggle to keep track of what they have, leading to unnecessary purchases or food waste. We envisioned a solution that leverages technology—specifically speech and image recognition—to automate the process of adding items to inventory, making grocery management quick, intuitive, and effortless. By combining speech-to-text and OCR (optical character recognition) technologies, we set out to build a tool that would not only help track groceries but also contribute to reducing food waste. We provide users with recipes for any meal of the day to use their groceries, and food banks near them to donate if they have a surplus that they won't be able to use.
What it does
Squash is a smart food inventory management app designed to make grocery tracking as simple as talking or snapping a picture. Users can add food items to their inventory by either speaking into the app or taking a picture of a grocery receipt. The app uses speech recognition to convert spoken words into text and Optical Character Recognition (OCR) to scan receipts and extract information about the items purchased. Squash then categorizes these items (e.g., breakfast, lunch, dinner) to create recipes. It also stores them in a food inventory list, making it easy to know what you have at home at any given time.
How we built it
We built Squash using a combination of modern technologies for both the frontend and backend:
Frontend: We used React Native for the mobile application, leveraging its versatility across both iOS and Android platforms. For the voice input functionality, we integrated Expo's audio recording feature, which allows users to speak into the app. We then used a custom UI to display categorized food items, designed with intuitive navigation for different meal types (breakfast, lunch, dinner).
Backend: The backend is powered by Flask, a Python web framework. We implemented API endpoints to handle both speech-to-text and OCR processing. For the speech transcription, we convert audio recordings into base64, which is sent to the backend for processing using a machine learning model. The OCR functionality extracts text from receipt images and automatically categorizes the items based on predefined food categories.
We used many of Gemini's LLMs to generate responses and insights with our data.
Challenges we ran into
One of the biggest challenges we encountered was managing the accuracy of both the speech-to-text and OCR processes. The speech recognition models sometimes struggled to correctly identify grocery items, especially when multiple items were listed in quick succession. Similarly, parsing text from receipts proved difficult due to varying formats and print quality, which affected the accuracy of OCR. Another technical hurdle was handling large audio and image files efficiently without causing delays in processing or overloading the app.
Accomplishments that we're proud of
We’re proud of successfully integrating two powerful technologies into a seamless user experience and catchy brand. Building a functional app that can transcribe speech and scan grocery receipts, and then categorize and store that information into an organized inventory, is an achievement we’re incredibly proud of. This solves a very practical and real world problem.
What we learned
Throughout the development of Squash, we learned the intricacies of working with real-time audio and image processing. We deepened our understanding of APIs, especially how to handle errors and edge cases when interacting with services. Additionally, we gained valuable experience in optimizing mobile app performance, particularly when dealing with large files and asynchronous processes. Most importantly, we learned the importance of user feedback in refining our approach, leading to improvements in the user experience and overall app efficiency.
What's next for Squash
In the future, we aim to further refine Squash’s accuracy by training custom models that are optimized for grocery-related data. We plan to integrate generating shopping lists when supplies are low. We also want to expand the app to include meal planning and calorie tracking features, providing a holistic food management tool. Additionally, a major goal is to make Squash compatible with smart home devices like Amazon Alexa and Google Home, allowing for even greater integration into users' daily lives.
Log in or sign up for Devpost to join the conversation.