CookPilot

Inspiration

As busy university students, deciding what to cook always ate up time we didn't have — staring into the fridge, then a grocery run for stuff we already owned. Once cooking started, phones made it worse: recipe on screen, hands covered in food, constantly wiping the screen to check the next step. We wanted to remove friction on both ends — deciding what to cook, and cooking without touching a phone.

What it does

CookPilot turns your fridge into a recipe you can cook right now, hands-free.

Smart ingredient detection - scan your fridge via smart glasses or iPhone to see what you have.
Personalised recipes - tailored to dietary needs and pantry staples, with missing ingredients linked directly to Woolworths for quick ordering.
Hands-free cooking - ask "Hey Chef, what's next?" or "Are my eggs cooked?" and get a spoken answer, no screen needed.
Learns over time - remembers feedback from past sessions (like a step that ran long) to personalise future guidance.

How we built it

Swift & SwiftUI for the iOS app.
Meta Wearables Device Access Toolkit to stream live camera feed from the smart glasses.
AVFoundation for camera, microphone, and audio playback.
OpenAI GPT-4.1 Mini for ingredient detection, recipe generation, and cooking Q&A.
OpenAI Whisper for speech-to-text voice commands.
OpenAI TTS (Nova) to speak instructions and responses back.

Pipeline: camera frame → GPT-5 Mini reasoning → recipe rendered in large, glanceable steps → Whisper/TTS handling the voice loop while cooking.

Challenges we ran into

Getting a stable, reliable camera stream from the Meta glasses.
Building a voice loop that doesn't pick up its own responses or kitchen noise.
Cutting latency so answers feel instant while standing over a hot pan.

Accomplishments that we're proud of

A genuinely hands-free loop : scan, decide, cook, ask questions, get spoken answers, working reliably in a real kitchen, across software and three AI services.

What we learned

Working with wearable camera streaming in real, imperfect conditions (steam, motion, low light).
Designing voice interactions that feel natural, not like a command line.
How much latency matters once a real person is waiting on an answer.