SnapCook

Inspiration

Our team was inspired to create this AR-powered smart cooking assistant after realizing how much time we spend juggling recipe books, timers, and various devices while cooking. We wanted to simplify the process and make it more enjoyable, so we combined our love for food with cutting-edge tech. By using AR on the Snapchat Spectacles, we could float essential cooking tools right in front of the user, allowing for a hands-free and more interactive experience. This project ties directly into wellness and health by encouraging home cooking, which is often a healthier alternative to takeout or processed foods. With easy-to-follow, visual recipe cards and timers right in view, we hope to help more people cook from scratch without the pain of a cluttered culinary environment, leading to more mindful eating habits and a greater connection to what’s on their plate.

What it does

With virtual recipe cards, you’ll have easy-to-follow steps right in front of you, floating in the air (and optionally, following you around) so you don’t need to keep checking a screen. With ingredient and recipe cards that offer all the interactivity without any of the associated maintenance and cleaning, we've made trying new dishes easier than ever before. The application also includes virtual, interactable timers that you can set and adjust with a simple pinch of your fingers, so no more scrambling for your phone. Furthermore, our integrated voice commands let you search for recipes or add timers while you cook, taking your experience to an even more hands-free and hassle-free level.

How we built it

We built this smart cooking assistant using Snapchat Lens Studio for Spectacles, which offers an intuitive developer experience for everything from designing and positioning the floating recipe cards, ingredient lists, and timers to thorough joint and location hand tracking in the AR space. This is all powered by TypeScript scripts that are built on top of Snap's developer toolkit. On the backend, a custom API was developed that integrates directly with GPT-4o as well as the google search API to retrieve, generate, and format recipes. To make the experience even more intuitive, we created and trained a YOLO model to detect where pots and pans are in the scene, meant for easy creation and positioning of timers. (Unfortunately, this model was too heavy to run in real time on the AR glasses and caused significant lag, so we scrapped it for our final prototype). For hands-free control, we used Lens Studio's AI VoiceModule for speech-to-text, offering access to intuitive voice commands for searching recipes and setting timers.

Challenges we ran into

There were a few roadblocks we had on our way to the final product. For one, the majority of the team did not have prior Unity/Lens Studio experience, so there was a bit of a learning curve that stagnated how fast we could add UI elements, enable interactivity, etc. Furthermore, due to how recently the Spectacles were released, there were a few features that didn't play well with Lens Studio and this was something we had to work around. Another hurdle we faced was that Lens Studio did not support nodejs development, which was actually the necessitating factor for developing a backend integration for the app. It was only through this that we were able to fetch accurately formatted, relevant data to display recipes and ingredients. Lastly, we encountered issues while trying to utilize our trained YOLO models on the specs. Not only did the Spectacles lack support for anything but pb and onnx models, but, during development, we found the model was making the experience laggy and unexciting.

Accomplishments that we're proud of

All said and done, during the span of only 16 hours, we were able to develop not only the vision we had in mind for the Spectacles project but also added more complex features to our existing design. To be able to work collaboratively on such a new technology and produce a really interesting application was very fulfilling and helped us improve our development and iterative processes overall.

What we learned

During the hackathon, we learned a ton about working with new digital tools, especially Snapchat Lens Studio and the Spectacles platform. None of our team had experience with AR development before, so diving into how to build, position, and program UI elements in 3D space was something of a learning curve. We also got hands-on practice with machine learning by training a YOLO model, which pushed us to think about how to balance performance with functionality in AR. On top of that, integrating voice commands and working with LLM-based APIs taught us the importance of backend systems in creating seamless, interactive experiences. Overall, it felt like a great "crash course" in trying new technologies and learning how to adapt when things didn’t go as planned.

Built With

• lens-studio
• nextjs
• python
• typescript
• vercel