Are You Done Yet?

Why Should I Care?

Everyone eats, and most of us cook badly without ever finding out why.

Cooking has a broken feedback loop, because you follow a recipe, taste the result, and find it somehow flat or soggy or just not what the picture looked like, yet across the forty small decisions you made along the way, nothing tells you which one actually cost you the dish.

There are many things people regret not knowing at 18, and this is one of them, though it has far less to do with knife skills than with the unglamorous fundamentals nobody ever says out loud. None of that is difficult to learn, but all of it is invisible, and a video tutorial can explain it a hundred times without ever being able to watch you.

What Does It Do?

Counter Scan: Looks at your ingredients before you begin, then counts and categorizes everything it can see so the rest of the session has something real to work from.

Recipe Library: Lets you search by name, ingredient, or tag, and shows on every dish both what you already have and what you are still short of.

Recommend a Dish: The AI model customizes a personal recipe for you and only ever suggests a dish you can genuinely make then and there.

Guided Walkthrough: The interface is able to guide you through the recipe step by step in AR, creating guidelines and recommendations for the actions you perform on food such as cutting, mixing, saucing, etc.

Ask the Chef: LLM answers what you are missing at the press of a controller button, reasoning entirely from what you actually did based on a trained dataset of fundamental cooking mistakes

Session Debrief: Summarizes your score, what you can improve in your cooking performance, and logs it to analyze your cooking history long-term.

How We Built It

The app runs on a Meta Quest 3S in WebXR through IWSDK with a MongoDB database to maintain a personal dynamic catalog of recipes and ingredients each user favors.

Everything you do is recorded as an action, and the state of the bowl is derived from that log rather than observed, which is what allows sixteen distinct faults to fall out of it, from under-salting and wet greens through dressing too early, over-tossing, and never once tasting the thing you are about to serve.

OpenAI is never shown pixels at any point, only the measured facts as text, so its job is narrowed to deciding which problem matters most and how a chef would actually phrase it. We then grounded the whole error model against CaptainCook4D, a dataset of 384 recorded cooking sessions carrying roughly 2,400 human-annotated errors, and selected our three recipes from it by ranking the dataset on data volume against simplicity.

Challenges

Accomplishments

We built a coach whose feedback is derived rather than guessed, which means it is structurally incapable of telling you something it never observed.

Our fault model, which we had written from scratch out of intuition, turned out to map almost exactly onto a published dataset of real annotated cooking errors.

What We Learned

Developing for VR meant working within an entirely different developer ecosystem. Although our team was experienced with hardware integration and OpenCV-based object detection, porting our systems and AI architecture to Meta's platform required us to learn its SDKs, build pipeline, and on-device testing workflow, where every iteration had to be deployed and validated on the headset itself.

What's Next?

We want to open up more of the dataset, since we shipped only 3 of its 24 recipes and the remaining ones already have recorded sessions and annotated errors sitting behind them.

Heat is the natural next discipline, because our timing model already treats overcooking and undercooking as two ends of a single mechanic rather than as separate problems, and often varies between appliances, thickness of cut, and pan handling.

Memory across sessions would let the chef say "you under-salt every time" at the end instead of meeting you actively on every run, which is worth considerably more than any single session's advice.