Inspiration
The inspiration for UnBurnt came from two members of our team who are international students. After moving away from home, they found themselves cooking on their own for the first time and quickly realized how difficult it can be without guidance. They shared how frustrating it is to follow recipes that have vague instructions, especially when there is no one around to show you what the steps should actually look like. They express that without real-time feedback or someone being there to correct you, it could lead to small mistakes, which often lead to burned food or wasted ingredients. Hearing these experiences made us realize that many people face the same challenge when learning to cook independently, which inspired us to build a solution that provides real-time guidance and support while cooking.
What it does
UnBurnt is a mobile cooking assistant powered by Amazon Nova Lite and Nova Sonic. The app uses the user’s phone camera to observe what they are cooking in real time and provides step-by-step guidance and feedback throughout the process. By analyzing the cooking progress, the AI can detect mistakes and offer immediate suggestions to help users correct them before the dish is ruined. If a user gets stuck or needs clarification, they can simply speak to the built-in AI assistant, which provides hands-free guidance through voice interaction. Users can browse from a set of pre-saved recipes or generate entirely new recipes using the AI based on the ingredients they have available. When users find a recipe they enjoy, they can also save it for future use.
How we built it
We built UnBurnt as a full-stack AI-powered mobile cooking assistant that combines real-time computer vision, voice interaction, and dynamic recipe generation. The mobile app was developed using React Native with Expo and TypeScript, which allowed us to create a cross-platform interface with screens for recipe browsing, cooking guidance, login, and user accounts. On the backend, we used Python running on AWS Lambda with AWS SAM and API Gateway to process requests such as recipe generation, step expansion, camera frame analysis, and authentication. For AI capabilities, we integrated Amazon Nova Lite multimodal models to analyze camera frames sent from the app every few seconds and provide real-time coaching or detect cooking mistakes. The backend evaluates the image together with the current recipe step and returns contextual guidance that is displayed as overlays on the cooking screen. In addition to vision, we implemented a real-time voice assistant using WebSockets and Amazon Nova 2 Sonic, allowing users to speak hands-free while cooking; the system streams audio to the server, transcribes speech, and returns spoken and text responses that can also trigger actions such as moving to the next recipe step. We structured the frontend using modular folders for screens, components, hooks, and Zustand state stores, while the backend separates routing, AI services, and authentication logic. Secure Google OAuth authentication is handled by the backend, which verifies the Google token and issues a signed JWT stored in SecureStore on the device, while guest mode allows users to access the core cooking features without signing in. Together, these components create a system where the mobile app captures camera and voice input, the backend processes it with AI models, and the user receives real-time cooking guidance, mistake detection, and conversational assistance throughout the cooking process.
Challenges we ran into
Authentication Across Environments: Bridging Google OAuth between a React Native Expo frontend and an AWS Lambda backend was a major hurdle. We had to bypass standard Expo auth patterns to have the backend handle the Google token verification, issue its own signed JWT, and securely persist that session in the device's SecureStore.
Cloud vs. Local Synchronization: Testing mobile hardware features (like the camera) against a cloud backend created intense debugging loops. For example, a missing variable in our Python rules engine caused a 500 Internal Server Error. Fixing it required managing an Ngrok tunnel for the frontend while simultaneously rebuilding and deploying the backend to AWS SAM, all while navigating Python version conflicts in our local deployment environments.
AI Latency and Cold Starts: Processing high-resolution camera frames with Amazon Nova Lite every few seconds is computationally heavy. We initially hit hard 30000ms timeout crashes because the AWS Lambda cold starts were exceeding our API Gateway and Axios limits. We had to optimize the request payload, adjust timeout configurations across the stack, and implement fallback UI states so the app wouldn't crash while waiting for the AI.
Real-Time Voice Activity Detection (VAD): Building the bidirectional voice assistant with Nova 2 Sonic meant managing a complex WebSocket state machine. We had to build a custom audio pre-roll buffer that constantly monitors mic RMS to detect speech above −28 dBFS, seamlessly stream PCM chunks to the server, and gracefully handle 900ms silence cut-offs without breaking the conversational loop.
Tuning the Multimodal AI for the Kitchen: Initially, the camera struggled to accurately detect nuanced cooking mistakes (like incorrect heat levels or bad stirring techniques). Amazon Nova Lite needed more context than just a single image. We solved this by heavily engineering our prompts and ensuring the backend passed the exact current_step and a running array of mistakes_so_far into the model with every frame to ground its logic. We also had to account for real-world variables like motion blur, varied lighting, and hands blocking the pan.
Synchronizing Two Independent AI Lanes: We had two massive streams of data happening at once: an HTTP "Vision Lane" sending camera frames every 1.5 seconds, and a WebSocket "Voice Lane" streaming live PCM audio. Keeping our frontend Zustand state perfectly synced was a nightmare. We had to write strict logic to ensure a voice command like "next step" wouldn't clash with an incoming visual mistake alert being triggered at the exact same millisecond.
Accomplishments that we're proud of
Seamlessly Integrating Two AI Modalities: We are incredibly proud of getting Amazon Nova Lite and Nova 2 Sonic running in parallel. Managing an HTTP "Vision Lane" sending frames every 1.5 seconds alongside a live WebSocket "Voice Lane" for audio—without crashing the mobile app or losing Zustand state sync—was a massive technical win.
Custom Voice Activity Detection (VAD): Instead of relying on a pre-built voice SDK, we built our own VAD state machine in React Native. Successfully handling pre-roll buffers, tracking −28 dBFS speech thresholds, and engineering precise 900ms silence cut-offs to create a natural, hands-free conversation feels like a huge achievement.
Context-Aware Mistake Detection: We didn't just build a wrapper around an image analyzer; we built a system that actually understands the state of the kitchen. Passing the current_step and mistakes_so_far into the AI so it can accurately coach the user, issue priority alerts, and adapt the recipe checklist dynamically is a feature we are extremely proud of.
A Robust, Custom Auth Flow: Successfully bridging Google OAuth between a React Native Expo app and our own AWS backend was incredibly satisfying. Bypassing the standard Expo auth to have our Python server verify the token, issue a custom JWT, and persist it in SecureStore gave us a production-grade authentication system.
Full-Stack Deployment Under Pressure: Beyond the code itself, we are proud of our dev-ops execution. From debugging 500 Internal Server Errors on the fly to managing Ngrok tunnels and successfully shipping our Python backend via AWS SAM under a strict hackathon time limit, we proved our ability to deliver a live, cloud-hosted architecture.
What we learned
Multimodal AI Requires Deep Context: We learned that simply passing raw images to an AI isn't enough. To get Amazon Nova Lite to accurately understand nuanced cooking mistakes, we had to ground its vision analysis with rigorous state context, passing the current recipe step and historical mistakes alongside every frame.
The Nuances of Audio Processing: Building a custom Voice Activity Detection (VAD) system taught us a massive amount about raw audio handling. We learned how to manage PCM buffers, calculate RMS for decibel thresholds (−28 dBFS), and balance pre-roll audio to create a seamless, natural conversation with Nova 2 Sonic.
Parallel State Management is Unforgiving: We learned just how quickly race conditions can happen when managing two independent AI streams. Keeping our React Native Zustand store perfectly synchronized between an HTTP vision lane and a WebSocket voice lane forced us to write incredibly strict, bulletproof state logic.
Full-Stack Cloud Architecture: Moving from local development to AWS taught us the harsh realities of cloud deployment. We learned how to configure AWS SAM, manage IAM permissions and temporary credentials, handle Lambda cold-start latencies, and route traffic through API Gateways and Ngrok tunnels.
Custom Auth is Complex but Worth It: We learned the intricacies of the OAuth 2.0 flow. Bypassing standard Expo helpers to manually verify Google tokens on our Python backend, issue our own signed JWTs, and securely store them gave us a much deeper understanding of secure, production-level authentication.
What’s next for UnBurnt
Looking ahead, we want UnBurnt to become one of the leading AI cooking assistants for anyone who struggles with cooking independently. Our goal is to continue pushing the limits of what AI can do in the kitchen by making the experience even more interactive, personalized, and accessible. In the future, we plan to improve the vision system so it can recognize more cooking techniques, ingredients, and kitchen tools to provide even more accurate real-time guidance. We also want to expand recipe generation so the AI can adapt recipes based on dietary preferences, available ingredients, and skill level. Additionally, we aim to enhance the voice assistant to support deeper conversations and step-by-step coaching, making it feel like a real sous chef beside you. Long term, we hope to integrate features like smart kitchen device support, meal planning, and grocery recommendations so that UnBurnt can assist users through the entire cooking journey from planning a meal to successfully preparing it.
Log in or sign up for Devpost to join the conversation.