Cook Over

• 

  adjust ingredient 1

• 

  adjust ingredient 3

• 

  adjust ingredient 2

• 

  add timer 2

• 

  add timer 1

• 

  recognize a feedback from user to complete a previous step 1

• 

  recognize a feedback from user to complete a previous step 2

• 

  ask any clarifying question 1

• 

  ask any clarifying question 2

Inspiration

• When cooking, your hand is wet, you are surrounded by ingredients and equipments, you are using gloves, so on and so on.
• It is so annoying to keep going back to your phone to look at the recipe to find what to do next, how much water to use, what if I put too much sugar.

What it does

• Hands-free, voice-controlled recipe navigation.
• Read and understand recipes from text and URL then adjust serving/ingredients to your liking.
• Intelligent task parallelization to save time when cooking: automatically analyze recipe to create an optimized cooking order (e.g. while water is boiling, go chop vegetables).
• Answer any question related to instruction such as "What is al dente?", "I added too much salt now what?", "How should my food look like?", "How do I know if the meat is cooked?"
• Real-time cooking guidance: guide user through the cooking process step by step, keep track of the conversation to know what the users need help with and provide appropriate suggestions.

How we built it

ElevenLabs for voice interface and Mistral for understanding request and generating response

Challenges we ran into

• Latency
• Hard to parse recipe
• Hard to ochestrate the workflow

Accomplishments that we're proud of

Natural conversation interface to assist with any question while cooking

What we learned

• So hard to create an optimal step by step instruction to save time when cooking

What's next for Cook Over

• Mobile app

Built With

• elevenlabs
• mistral
• react