Chanoch Grocery

Inspiration

The spark for this project came from observing a daily struggle in my own home. Given the current economic situation and recent grocery price hikes, I would often watch my wife painstakingly open multiple grocery flyers and websites just to find items that were reasonably priced or on sale. To manage our family budget, she also frequently relied on chat agents like ChatGPT to help create weekly meal plans.

One evening, just as we were about to go to bed, I turned to her and said, "How about I create an app for you to help with our grocery shopping and meal planning, to save us both time and money?" It felt like fate when, the very next morning, I saw a post by Ayo on LinkedIn announcing the Gemini Live Agent Challenge. I knew immediately that this was the perfect opportunity and the exact technology needed to bring my idea to life.

What it does

Chanoch Grocery is a voice-first, multimodal AI assistant designed to act as your personal grocery shopper and meal planner.

• Dynamic, Location-Aware Store Filtering: The app doesn't just search blindly. It uses your device's GPS coordinates or postal code to dynamically filter its internal database of grocery chains, ensuring the agent only searches for deals at stores that physically exist in your specific region or neighbourhood.
• Cross-Store Deal Hunting: The core of the app allows users to search for grocery items across their favourite local grocery stores. The agent autonomously compares prices, finds active sales, and calculates the best deals based on the user's location and postal code.
• Intelligent Meal Planning: The agent generates multi-day meal plans tailored to the family's budget, dietary preferences, and allergies. You can ask the agent to swap out a specific dinner or add a snack to a specific day using just your voice.
• Multimodal Vision: By integrating device camera access (front/back) and screen-sharing, the agent can "see" what you see. You can point your camera at your pantry to take inventory or share your screen while looking at a digital grocery item.
• Seamless Multilingual Support: The app features automatic language detection. If you speak to it in Spanish, it understands and responds in Spanish. Furthermore, the agent dynamically switches the entire application's UI language on the fly to match the conversation (currently supporting English, Mandarin, Hindi, Spanish, French, German, and more).
• Voice-Driven UI Control: The AI can physically control the application from navigating between tabs, scrolling the screen, highlighting specific objects on the screen, and managing your digital grocery list.

How we built it

Under the hood, Chanoch Grocery is powered by GenAI SDK, React, Tailwind CSS, Typescript, and the Gemini Live API (gemini-2.5-flash-native-audio-preview-12-2025) and the latest Gemini flash model (gemini-3.1-flash-lite-preview).

Because I don't have the expertise of a professional UI/UX designer, I knew I had to rely on established design systems. I did some deep digging into Google's Material Design 3 (M3) guidelines. I also took heavy inspiration from the interface and fluid animations of my own Google Pixel 6 phone to create a clean, modern, and truly immersive design that feels native to the user.

I utilized extensive function calling (tool use) to bridge the gap between the Gemini model's reasoning and the application's frontend. This gave the agent full control over the app's rich feature set, allowing it to execute complex cross-store searches, update React state, and manage media streams based entirely on conversational context.

What we learned

Building Chanoch Grocery was a deep dive into the Gemini Live API. We learned how to transform a conversational AI into a proactive, multimodal shopping assistant:

• Crucial Real-Time Reasoning: The Live Agent's true power is its ability to instantly synthesize multimodal inputs. When a user points their camera at a shelf and asks for the best deal, the agent simultaneously analyzes the live video frame, cross-references injected context (like live GPS and health profiles), and reasons about the optimal choice before acting.
• Mastering Live Function Calling: We learned how to bridge continuous audio/video streams with discrete app logic. Equipping the agent with custom tools (onAddItem, onSearchSales) allows it to autonomously manage grocery lists and hunt for local deals completely hands-free.
• Multi-Step Budget Optimization: We leveraged Gemini's advanced reasoning for financial constraints. By setting the model's ThinkingLevel to HIGH and providing search tools, the agent executes complex logic chains: fetching real-time prices, calculating meal plan costs, and suggesting cheaper alternatives if a budget is exceeded.
• Voice-Driven UI "Magic": We discovered how effortlessly the Live API can physically pilot the application. By exposing frontend React functions as tools, the agent can scroll pages, navigate tabs, and even use an onHighlightObject tool to render a "digital laser pointer" directly over specific items on the user's live camera feed.
• Contextual Multilingualism: We learned how to bridge AI language comprehension with traditional app localization (i18n). By allowing the agent to detect the user's spoken language, it can autonomously trigger the onSetAppLanguage tool to switch the app's UI to the correct pre-translated language dictionary (supporting English, Spanish, French, Hindi, and more) on the fly.

Challenges we ran into

Building a real-time, multimodal application presented several unique technical hurdles:

• **When users requested a single meal for a specific future day (e.g., "Plan lunch for Tuesday"), the AI successfully generated the requested meal but unexpectedly filled the preceding days (Sunday, Monday) with random, unrequested meals.
• Conflicting AI Constraints and "Filler" Data Generation: Batch-Processing Meal Plan Ingredients**: One of the biggest challenges was figuring out how to efficiently add an entire list of ingredients from a generated Meal Plan to the user's grocery list. If a meal required 12 ingredients, asking the agent to search for the best local deal for each item sequentially caused massive latency, leaving the user waiting in silence. Conversely, firing off 12 simultaneous web searches crashed the browser or hit API rate limits. To solve this, we had to implement a custom concurrency-limited Promise pool (batching requests in groups of 3) within the onSearchAndAddMultipleItems tool. This allowed the agent to rapidly hunt for deals across multiple stores for an entire recipe without bottlenecking the system.
• Latency and Synchronization: Ensuring the agent's voice responses felt natural and conversational required optimizing the audio buffer queues and managing the WebSocket connection efficiently to minimize latency, especially when the agent was "thinking" while performing background tool calls.
• Context Window Management: Passing real-time location data, current shopping lists, health profiles, and visual frames to the model simultaneously required strict payload management to keep the agent contextually aware without overloading the connection or exceeding token limits.

Accomplishments that we're proud of

• Building a Real Solution: I am incredibly proud to have built a practical tool that directly solves a real-world problem for my family and so many families out there, saving actual time and money.
• The "Magic" of UI Language Switching: Successfully implementing the feature where the UI automatically translates itself based on the language the user speaks to the agent feels like a truly next-generation user experience.
• Design Execution: Despite lacking a formal UI/UX background, successfully translating Material Design 3 principles into a fluid, Pixel-inspired interface that feels polished and immersive.
• Complex Tool Orchestration: Getting the Gemini Live API to flawlessly execute multi-step tool calls like searching three different stores, comparing prices, and adding the cheapest option to a list was a massive technical win.

What's next for Chanoch Grocery

• Direct Cart Integration: Partnering with or utilizing APIs from major grocery chains to allow users to export their optimized Chanoch Grocery list directly into a store's digital checkout cart.
• Historical Price Tracking: Implementing a database to track the price history of favorite items over time, allowing the agent to advise users on whether a current "sale" is actually a good deal compared to historical data.
• Expanded Language and Accessibility Features: Adding support for more languages and integrating deeper accessibility features for visually impaired users who could benefit immensely from a voice-first grocery assistant.
• Community Meal Plans: Allowing users to share their budget-optimized meal plans and grocery lists with a broader community.

Built With

• canvas-api
• css
• express-v5
• framer-motion
• gemini-3.1-flash-(gemini-3.1-flash-preview)
• gemini-3.1-flash-lite-(gemini-3.1-flash-lite-preview)
• gemini-live-api-(gemini-2.5-flash-native-audio-preview-12-2025)
• geolocation-api
• google
• google-cloud
• google-cloud-run
• google-cloud-secret-manager
• google-genai-sdk-(@google/genai)
• google-search-grounding
• i18next
• javascript
• localstorage
• lucide-react
• material
• navigator.mediadevices-(webrtc)
• node.js
• react-19
• tailwind-css-v4
• typescript
• vite
• web-audio-api
• webrtc
• websocket