Inspiration
As Uni students, our relationship with cooking and being able to feed ourself is rough. From eating instant ramen to "cooking" frozen dumplings, learning how to make food while balancing school and co-op hunting has been difficult. That's why we came up with rat-AI-touille, your own Recipe Enhancement & Meal Yielder (REMY).
What it does
Rat-AI-touille is a hardware AI cooking assistant that watches your cooking through a camera and provides real-time commentary and guidance. It analyzes the video feed to understand what’s happening in the kitchen and can give tips, reminders, or suggestions. Users can also press a push-to-talk button and ask the AI questions while cooking.
How we built it
We used an ESP32-CAM to capture video from the cooking area and stream frames to a laptop for processing. When the user starts the application, they provide a dish or ingredient, and the AI generates a recipe for it. As the user cooks, the laptop runs an AI pipeline that analyzes the video feed and guides them through the recipe step-by-step. A push-to-talk button allows the user to speak to the AI through the laptop microphone, enabling voice interaction while cooking. The system then responds with helpful commentary, answers questions, and suggests the next steps in the recipe.
Challenges we ran into
Our greatest challenge was the tight deadline, considering that we started the project at 5 PM 😭😭. Getting the video feed from the ESP32-CAM to reliably stream to our laptop also took significant troubleshooting. We ran into several hardware issues during development, and the hardware itself had limitations. For example, the ESP32 setup didn’t include a speaker or microphone, so we had to rely on the laptop’s speaker and microphone for audio output and push-to-talk voice input.
Accomplishments that we're proud of
We’re proud that we were able to integrate hardware, computer vision, and AI into a working interactive system within a short time. Getting the ESP32 camera streaming, enabling voice interaction with push-to-talk, and producing meaningful AI commentary made the project feel like a real cooking assistant.
What we learned
We learned a lot about integrating hardware with AI systems, including streaming video from microcontrollers, designing AI pipelines for real-time interaction, and managing multimodal inputs like video and voice. We also learned how to optimize workflows to reduce unnecessary API calls by refining our code logic and keep the system efficient.
What's next for Rat-AI-touille
Next, we want to incorporate more hardware into the system and develop it into a fully independent device that doesn’t rely on external prebuilt hardware like a laptop. We also want to improve the AI by adding more personality and character so the assistant feels more like a fun, interactive cooking companion.
Log in or sign up for Devpost to join the conversation.