Physical Shopping Companion

Your AI shopping companion that sees, listens, and helps you shop smarter in real life.

Comment

Inspiration

Shopping in a physical store can sometimes be challenging—especially if you're in a new place, unfamiliar with the language, or feel uncomfortable asking staff for help. Many people still love the experience of physically seeing and touching products before buying, but they often miss the convenience and information available while shopping online.

This inspired us to create a shopping companion that brings the intelligence of online shopping into the physical store. Instead of searching through apps or websites while shopping, users can simply use voice and camera to interact with products, manage their shopping list, check prices, and get recommendations -all in one place.

Our goal is to make in-store shopping smarter, more accessible, and more comfortable for everyone.

What it does

Our app acts as an AI-powered shopping companion for physical stores. It helps shoppers create and manage their shopping cart using voice, text, and camera input. The assistant can automatically detect items placed in the trolley, check them off the shopping list, and keep track of the total spending to stay within budget.

Users can ask questions about products by simply showing them to the camera or using voice, and the assistant can provide product insights, ingredient information, and comparisons. It can also suggest recipes, shopping combinations, and event-based shopping lists.

Additionally, the app can scan products and check online prices to help users find better deals, making in-store shopping smarter, faster, and more informed.

How we built it

Our system is designed as a real-time multimodal shopping assistant that processes voice, text, and visual inputs while maintaining a live shopping list.

Frontend

  • Built using React Native to support both Android and iOS with a single codebase.
  • The mobile app captures voice, text, and camera input and streams it to the backend in real time.
  • Displays a live shopping cart interface that updates items, quantities, and total cost dynamically as the user shops.

Backend

  • Implemented using FastAPI to support asynchronous APIs and real-time streaming.
  • Agent orchestration is built using Google Agent Development Kit (ADK) to manage reasoning, tool usage, and multimodal interactions.
  • Deployed on Google Cloud Run for scalable container-based deployment.

AI and Streaming

  • Real-time conversation is powered by the Gemini Live API, enabling bidirectional streaming between the mobile client and backend.
  • Text-based agent reasoning uses Gemini Flash series models for fast responses and lower latency.
  • Continuous voice interaction allows users to naturally add items to their shopping cart while moving through a store.

Architecture Highlights

  • Bidirectional audio and text streaming
  • Agent-based intent interpretation
  • Real-time cart updates
  • Cloud-native scalable backend

Challenges we ran into

1. ADK API version constraints

The ADK currently hardcodes the v1alpha API version, which created compatibility issues when integrating with newer Vertex AI streaming capabilities. Some of the live streaming models are not fully supported with this alpha version.

2. Streaming latency

Maintaining low latency for bidirectional audio streaming between the mobile app and backend was challenging. Processing audio, network communication, and agent reasoning introduced delays that needed optimization.

3. Accent and language interpretation

Voice input sometimes caused accent-based misinterpretation, where English spoken with regional accents was detected as local languages. This required adjustments in prompting and handling fallback logic.

4. Live API compatibility limitations

While integrating Gemini Live bidirectional streaming, we encountered limitations combining:

  • ADK agent orchestration
  • Alpha API versions
  • Live streaming models

This required custom handling of streaming sessions and partial workarounds outside the default ADK abstractions.

Accomplishments that we're proud of

Overall, we’re proud that we were able to bridge the gap between online intelligence and physical shopping, creating a seamless in-store assistant experience.

What we learned

1. Building real-time multimodal systems is complex

Integrating voice, text, and visual inputs into a single interactive workflow requires careful handling of streaming, synchronization, and state management. Real-time user experiences depend heavily on minimizing latency across every layer of the stack.

2. Bidirectional streaming changes how applications are designed

Using live streaming APIs requires designing systems that can handle continuous input and output instead of traditional request-response APIs. This changes how backend services, event handling, and user interfaces are structured.

3. Agent orchestration simplifies complex interactions

Using an agent-based architecture helped structure reasoning, tool usage, and task execution. It allowed us to focus more on the application logic rather than building custom orchestration pipelines from scratch.

4. Speech interpretation is highly sensitive to accents and environments

Voice-based interfaces must handle accent variation, background noise, and pronunciation differences. Even small variations in speech can lead to misinterpretation, which makes fallback mechanisms and contextual reasoning important.

5. Cloud-native deployment improves scalability and experimentation

Deploying the backend in containers on serverless infrastructure made it easier to iterate quickly, scale the service, and manage real-time workloads without worrying about infrastructure management.

6. Rapid prototyping with modern AI tools accelerates development

Modern AI models and developer tooling make it possible to build complex AI-powered applications quickly, but understanding their limitations and integration challenges is equally important.

What's Next for Physical Shopping Companion

We plan to continue expanding the assistant to make physical shopping more intelligent, personalized, and seamless.

Persistent Memory and Personalization

The assistant will develop long-term memory to understand user preferences across shopping sessions. It will remember frequently purchased items, dietary restrictions, preferred brands, and typical spending habits. This will allow the assistant to automatically suggest shopping lists, personalized deals, and faster product recommendations based on the user’s shopping history.

Store Integrations and Smart Checkout

We aim to integrate directly with store systems and APIs so the assistant can interact with real-time store inventories and billing systems. For example, when a user is shopping inside stores such as Walmart or Target, the app could automatically generate a live cart, apply available coupons, and complete checkout directly within the app.

This would allow users to scan items, track spending in real time, and pay without waiting in traditional checkout lines.

Smart Deal Discovery and Price Comparison

The assistant will expand its web search capabilities to compare prices across nearby stores and online platforms. If a product is cheaper nearby or available with a discount, the assistant can notify the user instantly and suggest better alternatives.

Premium users could unlock advanced deal discovery, automatic coupon detection, and cross-store price comparisons.

Premium Features and Subscription Tier

We plan to introduce a premium tier that unlocks enhanced capabilities such as:

  • Advanced price comparison across stores
  • Automatic coupon detection and application
  • Detailed product insights and ingredient analysis
  • AI-powered meal planning that converts recipes directly into shopping lists

This subscription model will help support the infrastructure required for real-time product recognition and store integrations.

In-Store Navigation and Product Discovery

Future versions will include indoor navigation to help users quickly locate products inside large stores. By combining store maps with camera recognition, the assistant can guide users to the exact aisle where an item is located.

Smart Recipe and Event Planning

The assistant will evolve into a planning companion capable of generating shopping lists for events such as family dinners, parties, or holidays. Users will be able to describe an event, and the assistant will automatically build a complete shopping list with recommended quantities and alternatives.

Retail Partnerships and Ecosystem Expansion

In the long term, we aim to partner with retailers and brands to create a connected retail ecosystem where shoppers, stores, and products interact seamlessly. This could include loyalty program integration, personalized promotions, and smarter in-store product recommendations.

Share this project:

Updates

Submission history