Inspiration
Our team was inspired by the beloved movie Ratatouille, a film that shaped our childhoods and sparked our love for cooking. As university students, we’ve faced the challenges of balancing cooking with busy academic schedules and knew there had to be a way to make it easier and more fun. That’s why we decided to bring Remy, the genius chef rat, from the screen into reality! With our project, anyone can cook—no matter how busy or inexperienced you are in the kitchen. Remy will be your guide, helping you whip up delicious dishes with ease.
What it does
Remy is your personal kitchen guide, sitting right on your head—just like in Ratatouille. Simply say, “What’s up, Remy?” to initiate a conversation. Not sure what to cook? Using his "eyes" and "ears," Remy will fetch a recipe based on what you say, and guide you through every step. He’ll even give playful tugs on your hair to steer you toward the right ingredients and tools in your kitchen—just like in the movie! Feeling lost in the sauce? Ask Remy anything, and he’ll answer instantly. No more scrolling through recipes on your phone with greasy hands—Remy’s got your back, helping you chef it up with ease and fun.
How we built it
We built Remy using a multithreaded system on a Flask backend, running a total of five threads to manage real-time tasks. One thread constantly listens for audio input, which is captured using a mobile phone acting as both the microphone and camera due to hardware limitations with the Raspberry Pi. The audio is transcribed into text using speech-to-text APIs, and another thread handles text input, processing it through a fine-tuned OpenAI model optimized for cooking instructions. A separate thread replays responses using text-to-speech, while the fifth thread captures footage from the phone’s camera for analysis and uploading.
For the physical interaction, we used a Raspberry Pi to control servo motors that power Remy’s animatronic movements. Although the original plan included complex servo movements, we simplified them due to hardware failures. Remy can still perform basic motions, guiding users in the kitchen.
Additionally, we developed a dynamically updating frontend using React and Socket.io, allowing users to track the images captured by Remy, view the questions they’ve asked, and see the answers Remy provides in real time. We also leveraged TensorFlow machine learning models to help Remy identify ingredients for recipes, such as distinguishing between different fruits.
Challenges we ran into
Our project faced several challenges, particularly with hardware and integration. Initially, we planned to use a microphone and camera directly connected to the Raspberry Pi, but we couldn’t procure a working setup in time. As a result, we shifted to using a mobile phone for audio and visual input, which required adapting our system to handle external hardware. Additionally, we encountered issues with the servo motors used to animate Remy. During testing, multiple servo motors failed, and with limited hardware resources available, we had to simplify Remy's movements to basic gestures. We also had to learn sewing techniques to embed the servo motors into Remy’s skeleton, dealing with snapping strings and other complications in the process.
On the software side, we had to manage the complexity of multithreading across five separate tasks on a Flask backend, ensuring that the real-time audio processing, transcription, and camera input didn’t cause delays or crashes. Balancing these threads while keeping performance smooth was a challenge, especially when processing multiple inputs simultaneously.
We also had to fine-tune machine learning models using OpenAI and TensorFlow to help Remy correctly identify ingredients in real time. This required a lot of iteration, as the models needed to accurately distinguish between similar-looking ingredients, such as different fruits, which took considerable time and effort.
Lastly, integrating all of these components into a coherent, user-friendly experience was a significant challenge. Developing a dynamically updating frontend using React and Socket.io, while syncing it with real-time data from the backend, took careful coordination. Despite these obstacles, we were able to successfully deliver a functional, interactive kitchen assistant.
Accomplishments that we're proud of
We are incredibly proud to have presented a coherent, functional solution made out of multiple machine learning models, showcasing our abilities of problem solving and working under pressure. When faced with complications, such as failed hardware and problems with our trained models, we were able to pivot efficiently and find solutions on the fly. Another achievement was successfully integrating all the hardware components into a cohesive system, overcoming the complexities of putting everything together.
What we learned
Throughout this project, we gained valuable insights into integrating diverse technologies and handling real-world constraints. We learned the intricacies of multithreading on a Flask backend, managing multiple simultaneous tasks while maintaining real-time performance. Working with machine learning models in TensorFlow taught us how to fine-tune models for specific applications, such as ingredient recognition. We also discovered the importance of adapting to hardware limitations and learned practical skills like sewing to integrate servo motors into animatronics. This project reinforced our problem-solving abilities and highlighted the significance of flexibility and innovation when facing unexpected challenges.
What's next for Remy the Rat
Looking ahead, we plan to enhance Remy's capabilities with additional features and improvements. We aim to refine the animatronics by integrating more advanced servo motor movements and exploring alternative materials to improve durability and functionality. Expanding Remy’s knowledge base with more comprehensive recipe databases and multilingual support will make him even more versatile in the kitchen. We also envision incorporating more sophisticated computer vision algorithms to better identify and interact with ingredients. Additionally, we want to explore potential integrations with smart kitchen devices for a more seamless and efficient cooking experience. Our goal is to continue evolving Remy into an even more helpful and interactive kitchen assistant.
Built With
- gpt
- multithreading
- python
- raspberry-pi
- react
- tensorflow
- websockets
Log in or sign up for Devpost to join the conversation.