Brocery

upload text from group chat
list of things oarsed from group chat
Aisle sort
Checkout and split payment
leaderboard
Live rendering
Live rendering

Inspiration

My bro really wanted Capri Sun, but the saddest part is I forgot it :((

What it does

Brocery is the grocery-shopping bro who actually remembers everything.

Group grocery shopping starts in a chaotic chat: one bro wants oat milk, another has an allergy, someone says “anything is fine,” and somebody else changes their mind five messages later. Brocery turns that chaos into one shared shopping experience.

Upload a WhatsApp chat export and Brocery extracts requests, preferences, dietary restrictions, budgets, and allowed replacements. It builds a clean aisle-by-aisle list, then uses AR and product recognition in-store to help the shopper find the right item, check it against the request, and add it to the shared basket.

It is not just a grocery list—it is the bro who keeps the group coordinated from “can you grab this?” to “you got it, bro.”

How we built it

We built Brocery as a React and Vite web app with a Python and FastAPI backend. Zustand manages trip and cart state, while Pydantic defines structured contracts between the frontend, backend, and AI services.

The flow begins with a WhatsApp .txt export. A local Python preprocessing pipeline removes irrelevant messages while preserving requests, corrections, budgets, dietary restrictions, and substitution rules. Gemini 3.1 Flash-Lite and Meta Muse turn that context into a structured grocery contract, which we map to a product catalog for an aisle-by-aisle route.

For the in-store experience, we built an AR pipeline with WebXR, Three.js, WebGL, MediaPipe Tasks Vision, browser camera APIs, Meta SAM, and Gemini vision. Brocery recognizes products in the live camera feed and reconnects them to the original request, so the shopper sees useful context such as “unsweetened · under $5 · any oat brand works,” rather than only a product label.

We also designed agentic-commerce guardrails around each shopping decision. The original request, budget, restrictions, allowed substitutions, product considered, and final selection remain connected through checkout. This is a foundation for a future integration with Visa’s agentic-payment infrastructure, where the payment decision can be transparent and grounded in the physical shopping context.

Our stack includes React, Vite, Tailwind CSS, Zustand, WebXR, Three.js, WebGL, MediaPipe, Meta SAM, Meta Muse, Gemini, Python, FastAPI, Pydantic, HTTPX, Pillow, REST APIs, and Cloudflare Tunnel.

Group chat → AI contract → aisle route → AR recognition → requirement verification → guarded checkout.

Challenges we ran into

AR was completely new territory for us. We had to learn how to combine a live camera feed, product detection, anchored overlays, and responsive UI without making the experience laggy or distracting.

Image recognition during live video rendering was a major challenge. Running recognition too often made the camera slow and unstable, so we switched to a faster model/workflow for live product identification. We also separated real-time visual tracking from deeper product and ingredient checks, keeping Brocery responsive while still providing useful context.

Chat parsing was another challenge. Sending an entire unfiltered conversation to a model was too slow—an early five-text test took nearly two minutes. We built local preprocessing that identifies shopping-relevant messages and preserves nearby context before sending it to the model. This let us process a much larger 600-message WhatsApp chat in about 20 seconds.

We also spent significant time researching Visa payments and how Brocery could support agentic commerce responsibly. The challenge was not simply smoother checkout; it was designing guardrails that preserve shopper intent, rather than letting an agent make an unexplained purchase.

Basically, we had to make Brocery smart enough to understand the chaos, fast enough that your bro is not waiting in the cereal aisle, and trustworthy enough to keep the shopper in control.

Accomplishments that we’re proud of

Turning a real WhatsApp group chat into a structured, reviewable grocery plan.
Reducing long-chat parsing from nearly two minutes for an early five-message test to about 20 seconds for a 600-message chat.
Building our first-ever AR experience from scratch, including live camera input, product recognition, anchored tags, and gesture interactions.
Connecting an original group-chat request to the in-store product, rather than showing a generic product label.
Building one flow from chat import to aisle route, AR shopping, shared basket, and checkout summary.
Designing a thoughtful foundation for future Visa agentic-commerce guardrails.
Making sure nobody has to miss their Capri Sun again.

What we learned

We learned that the best shopping assistant is not one that makes decisions silently—it is one that keeps people in control. AI can identify requests and organize information, but the shopper should still approve the final list and make judgment calls in the aisle.

We learned that speed matters as much as intelligence. Filtering and structuring chat data locally before calling a model made Brocery practical for long, messy group conversations.

Because this was our first AR project, we also learned how the AR stack fits together: WebXR, live camera feeds, 3D rendering, anchored overlays, browser-based computer vision, and responsive interaction design. A good AR experience has to be useful in the moment, not just visually impressive.

Most importantly, we learned that grocery shopping is an act of care. Good technology should make it easier to show up for your bros.

What’s next for Brocery

Next, we want to bring Brocery to Meta smart glasses and make the grocery run hands-free.

Your bros should not have to download another app, build another list, or answer questions while you shop. They already said what they need in the group chat. Brocery understands their preferences, budgets, restrictions, and acceptable substitutions—and carries that context into the store.

Through Meta glasses, Brocery could recognize the product you are looking at and surface guidance in real time:

“Unsweetened only · under $5 · any oat brand works.”

Then, we want to partner with retailers and connect Brocery to store data: live inventory, aisle locations, product details, prices, and available substitutions. Brocery could guide the shopper to the right aisle and recommend items that are actually available in that store.

Group chat → Brocery understands → Meta glasses guide → retailer data confirms → shopper grabs → Visa checkout.

Your bros asked once. Brocery handles the rest.