VirtualChef

An augmented reality chef that can assist in every step of the cooking process with live AI feedback.

Comment

Inspiration

In today’s fast-paced world, many people find it challenging to cook meals at home, often lacking the confidence or skills to prepare diverse dishes. Additionally, traditional cooking methods like following recipes or watching video tutorials can be cumbersome, requiring constant pausing and rewinding. Virtual Chef addresses these challenges by leveraging augmented reality (AR) and artificial intelligence (AI) to create an interactive, personalized cooking experience.

Virtual Chef transforms your kitchen into an immersive learning environment, empowering users of all skill levels to cook confidently. The app utilizes the following:

  • AR.js and A-Frame for animating step-by-step cooking instructions in augmented reality, allowing users to visualize each stage of the recipe directly in their kitchen environment.
  • Hugging Face's Inference API with the meta-llama/Llama-3.2-11B-Vision-Instruct model for real-time image analysis and feedback
  • Flask and Python for a robust backend API, handling recipe management and instruction processing
  • React for a responsive and interactive front-end user interface
  • Blender for creating high-quality 3D models of the virtual chef and kitchen implements
  • HTML and JavaScript for core web functionality and augmented reality integration
  • Computer vision techniques for real-time assessment of user actions and ingredient preparation

Our "step validation" system was inspired by computer vision techniques used in quality control processes in manufacturing. It ensures that each cooking action meets the required standard before progressing.

What it does

Virtual Chef is an AI-powered augmented reality cooking assistant designed to guide users through the cooking process by:

  • Projecting animated 3D cooking steps into the user's kitchen environment using AR technology, allowing users to follow along with each stage of the recipe.
  • Providing step-by-step visual instructions for various recipes.
  • Analyzing user actions through computer vision to ensure the correct execution of each step.
  • Offering immediate feedback and suggestions for improvement based on AI image analysis.
  • Supporting multiple recipes with detailed, interactive guidance.
  • Allowing users to progress through recipes at their own pace with hands-free interaction.

How we built it

  1. AR Environment: Implemented AR.js and A-Frame to create an immersive augmented reality kitchen, projecting a virtual chef into the user's real-world space.
  2. 3D Modeling: Animated a realistic 3D model of a virtual demonstration using Blender, ensuring seamless integration with the AR environment for cooking demonstrations.
  3. Step Validation System: Developed a computer vision-based validation system using the Hugging Face Inference API with the meta-llama/Llama-3.2-11B-Vision-Instruct model. This system analyzes images of the user's actions in real time, comparing them to the expected outcomes for each recipe step.
  4. Backend Architecture: Built a Flask-based Python backend to manage recipes, process instructions, and handle API requests. This server facilitates communication between the front end, AR components, and AI services.
  5. Frontend Development: Created an intuitive React-based user interface, allowing seamless interaction with the virtual chef, recipe progression, and display of real-time feedback.

Challenges we ran into

  • AR Animation Implementation: Creating and integrating AR animations with markers, ensuring smooth and realistic movements of cooking steps in the user's environment.
  • Backend-Frontend Integration: Seamlessly connecting the Flask-based backend with the React frontend, particularly in handling real-time data flow for recipe steps and user feedback.
  • AI Response Formatting: Crafting precise prompts to ensure the AI (using the Hugging Face Inference API) consistently responds in the specific format required for our step validation system.
  • Real-time Performance: Optimizing the application to handle simultaneous AR rendering, image processing, and AI analysis without lag or performance issues on various devices.

Accomplishments that we're proud of

  • Learning AR Technology: Successfully implemented advanced AR features with limited documentation, showcasing our ability to innovate in emerging technologies.
  • AI Integration: Seamlessly integrated Hugging Face's Inference API and the Llama model, enabling sophisticated real-time image analysis and feedback generation.
  • Immersive Cooking Experience: Created a truly interactive and engaging virtual cooking experience that responds to user actions in real time.

What we learned

  • How to integrate and utilize the Llama model through the Hugging Face Inference API for real-time image analysis and task evaluation.
  • Techniques for creating and animating AR models, specifically using AR.js and A-Frame for web-based augmented reality experiences.
  • The process of developing and implementing marker-based AR animations to create an interactive virtual chef.
  • How to build a robust backend using Flask, including setting up API endpoints and managing server-side logic.
  • Development of responsive and interactive front-end interfaces using React.js.
  • Effective integration of backend services with a React front-end for seamless data flow and user experience.

What's next for VirtualChef

  • Seamless AR Integration: We plan to fully mesh the augmented reality components with the main app, creating a more cohesive and immersive cooking experience.
  • Expanding Recipe Database: Our next step involves adding a wider variety of recipes to cater to different cuisines, dietary requirements, and skill levels.
  • AR Device Compatibility: We aim to extend our AR capabilities to dedicated AR devices like Meta Quest, enabling a more immersive hands-free cooking experience.
  • AI Performance Optimization: We're working on improving the speed and proficiency of our AI responses, ensuring near-instantaneous feedback and guidance during the cooking process.
Share this project:

Updates