Inspiration

Cooking can be stressful when you’re new to it, especially when you’re unsure what ingredients you have or what to do next. We wanted a hands-free companion that helps turn “What can I make with this?” into a meal and guides you along the way.

What it does

Peter Cookin uses the camera to spot food and cooking ingredients, including packaged items, sauces, seasonings, and produce. Ask for meal ideas or point the camera at your dish for a progress check and a practical next step.

Typing with messy hands while cooking is inconvenient, so we use Voiskey for voice input. It lets users speak naturally to Peter instead of returning to the keyboard. Gemini reviews potential ingredient labels, reads visible package text when possible, suggests recipes, and comments on cooking progress from camera images. Its visual guidance is helpful, but it can’t confirm food safety or internal doneness.

How we built it

We built the app with Google AI Studio using Python and a local desktop interface. OpenCV handles camera capture and image processing, while Ultralytics YOLOE and ByteTrack propose and track objects. Gemini’s API reviews those candidates, refines labels, reads package text, and provides recipe and progress guidance. Voiskey supports voice input, ElevenLabs reads Gemini’s replies aloud, and Apple Vision OCR helps read English package labels on macOS.

Challenges we ran into

Object detection sometimes missed ingredients or confused similar-looking packages. We added Gemini as a second-pass visual reviewer to refine the detector’s labels; it reviews results at runtime rather than retraining YOLOE’s model weights. We also had an API issue that stopped text-to-speech from playing, which took troubleshooting across the response and audio pipeline.

Accomplishments that we're proud of

We brought camera-based ingredient detection, Gemini’s visual review, recipe suggestions, cooking progress checks, voice input, and spoken replies into one local app. We’re proud that Peter Cookin can help users go from a countertop full of ingredients to a useful next step.

What we learned

A detector’s confidence score doesn’t guarantee a correct label, and a good system needs ways to review and correct its guesses. We also learned that voice interaction matters in the kitchen: the assistant needs to fit around the cooking, not make the user stop and type.

What's next for Peter Cookin'

We want to make the experience more hands-free with a wake phrase and automatic speech capture, improve detection for less common ingredients, and build an efficient cooking plan that coordinates prep with cooking time. Future additions could include clearer allergen alerts and an end-of-session nutrition summary based on ingredient amounts.