SmartCart AI x Alexa+

Talk to your kitchen. Add to your pantry, check what you have, and get recipes by voice, backed by a real app and real data.

Comment

Inspiration

Grocery planning involves a lot of small chores: checking what's in the pantry, deciding what to cook, and figuring out where to shop. These are natural things to ask out loud, but a voice assistant becomes much more useful when it can interact with the data in a real application.

We wanted to explore what an Alexa+-style experience could look like when it goes beyond answering questions and actually reads and updates a user's pantry, helps with recipes, and compares nearby grocery stores.

What it does

SmartCart AI is an AI-powered grocery assistant with a new voice-first Alexa+-style experience. Users can speak or type requests such as "Add two avocados to my pantry," "What's in my pantry?", "What can I cook tonight?" or "What's the recipe for tomato pasta?"

  • Pantry by voice: add, read and remove pantry items. The live pantry list on the page refreshes after each change.
  • Nearby store comparison: Find grocery stores near the location saved in the user's profile and compare itemized prices when supporting data is available.
  • Recipe: Save AI-generated or custom recipes and ask the assistant to read a recipe aloud.
  • Honest prices: Show where each price came from—a receipt, cached store price, or estimate—and which product it matched. Estimates do not count toward store totals, and the assistant does not declare a cheapest store unless the available data supports that conclusion.
  • Spoken replies: Amazon Polly reads responses aloud. Users can mute speech or type requests instead of speaking.

This is a simulated Alexa+ experience running in a browser. It is not a certified Alexa skill on an Echo device.

What we built during the hackathon

The core SmartCart AI grocery application existed before this hackathon. Our main addition was a voice-first, Alexa+-style experience powered by an AWS-hosted AI agent.

We added an Amazon Bedrock Nova Lite agent on AWS ECS Express Mode, Amazon Polly for spoken responses, and an MCP client that connects the agent to SmartCart AI's grocery tools. We also built a browser-based voice interface with typed input, visible tool activity, and a pantry list that refreshes after changes.

This brings together Amazon's AI services and our existing Google Cloud and Firebase infrastructure in a working cross-cloud architecture.

How we built it

  • Voice page: a React app on Firebase Hosting. It uses the Web Speech API for microphone input, with a typed fallback, and Amazon Polly for speech.
  • Agent on AWS: a service on Amazon ECS (Express Mode) that calls Amazon Bedrock (Nova Lite) as its model. It acts as an MCP client, verifies the user's Firebase token, and never lets the model choose the user ID.
  • MCP server on Google Cloud Run: five tools (get_pantry_items, add_pantry_item, remove_pantry_item, list_recipes, compare_stores) served over Streamable HTTP. Every call needs a valid Firebase token, and no tool accepts a user ID from the caller.
  • Data: Firestore and Firebase Auth hold the pantry, profile, recipes and receipts.
  • Store lookup: the saved profile location is geocoded with Nominatim, with the US Census geocoder as a fallback. Nearby stores come from Overpass across several mirrors, with a Nominatim brand search as a last resort. Chain stores are ranked ahead of small independents. Prices are taken from receipts first, then cached store prices, then labeled estimates.
  • Cost and safety controls: per-user daily limits, a minimal IAM role for the agent (Bedrock InvokeModel on Nova Lite and Polly SynthesizeSpeech only), budget alerts, and a single agent task.

Challenges we ran into

  • Public services rejecting cloud traffic. Nominatim returned HTTP 429 and Overpass returned HTTP 406 for requests coming from shared cloud addresses. We added retries, a second geocoder, extra mirrors, time limits and a distinct message for each failure, so "the lookup service is unavailable" is never confused with "no stores were found".
  • A silent removal bug. "Remove the avocados" reported success but removed nothing, because matching ignored plural names. We fixed the matching and made the tool return an honest "not found" result.
  • Trustworthy prices. Early totals were wildly inflated by bulk-size and mismatched product matches. We added per-item price ceilings, separated estimates from real prices, and stopped ranking stores on estimates.
  • Two clouds. The agent runs on AWS and the tools run on Google Cloud, so we kept the number of tool calls per turn low to limit latency.
  • Keeping every tool tied to the user. We moved user identity out of the tool arguments entirely, so it comes only from the verified token.

Accomplishments that we're proud of

  • A working cross-cloud design in which every tool call is tied to a verified user token.
  • A store comparison that reports what it does not know instead of inventing a winner.
  • A voice page that changes real data and reflects the change on screen.
  • Running the whole project toward a $0 out-of-pocket target, using free tiers, usage limits and budget alerts.

What we learned

A voice assistant is only as trustworthy as the tools behind it. Most of our time went into authentication, clear failure messages and data quality, not into the model itself. We also learned to design for failure first: free public services will reject you sometimes, and the user should always be told plainly what happened.

What's next for SmartCart AI x Alexa+

  • Receipt upload, so more prices come from real purchases at specific stores.
  • A real Alexa skill and Echo device integration.
  • Smarter product matching, so estimates become reliable store prices.
  • Nutrition and substitution suggestions on the voice path.

Built With

Share this project:

Updates

Submission history