Remy

An AI physical cooking guide that sees what you see and talks you through your recipe.

Comment

Inspiration

Cooking is stressful - you're constantly switching between the recipe and using different appliances. We wanted to build a hands-free cooking coach that actually watches what you are doing, almost like having a chef instructor in the kitchen with you. We named it Remy, inspired by the rat who guides the chef in Ratatouille.

What it does

Remy is essentially a real-time AI cooking assistant. You can either describe the dish or paste a URL of a recipe, and it will generate step-by-step instructions while scanning for allergies and offering substitutions. Once you start cooking, your phone camera will stream a live feed that our client analyzes every second - visually verifying whether each step is complete before you can move on to the next step. You can also talk to Remy while you cook and it will respond in real time.

How we built it

  • Frontend: Next.js | Tailwind | Figma | React
  • Backend: Python | openAI | RealtimeAPI | FastAPI | OpenCV | BeautifulSoup

Challenges we ran into

  1. Getting the phone camera visible to openCV.
  2. Voice activity detection threshold that can avoid background noise that can possibly cause false transcripts.
  3. Managing the video worker, audio stream, transcription, and GPT simultaneously and efficiently.
  4. Getting openAI to understand video input using action frames.
  5. Sending a live video feed analysis to the front end.

Accomplishments that we're proud of

  1. Our app is finally reliable in determining a step is done from a live camera feed after lots of prompt engineering :)
  2. Fully hands-free communication while cooking.
  3. Incorporating allergen filter + ingredient substitute flow works from both described recipes as well as URLs.

What we learned

  1. Building real-time AIs that can still communicate with the user.
  2. Few-shot prompting our client so we get reliable outputs (needs to be very specific and strict regarding format).

What's next for Remy

  1. Instead of using external camera capture like Camo, it can work straight from the phone's browser.
  2. A mobile app for the full kitchen experience.
  3. integrate Remy into Meta Glasses.

Built With

Share this project:

Updates