Kart

A shopping cart that actually knows you and the store. Kart sees what you grab, handles checkout, guides you by voice, finds personalized deals, prevents overbuying, and makes every trip seamless.

Comment

Inspiration

To be completely honest, Kart started as a bit of a joke. During our initial hackathon brainstorming, we thought it would be hilarious to build something involving an actual shopping cart, partly because acquiring a full-sized shopping cart would be a challenge in itself. (We were basically trying to clip farm)

But the more we thought about it, the more the idea made sense. The shopping cart is arguably the most intuitive interface in retail: everyone already knows how to use one, yet the experience has barely changed in decades. With advances in AI, computer vision, and real-time data processing, we saw an opportunity to completely reimagine what a shopping cart could do.

We also kept coming back to our own frustrating shopping experiences. Wandering around a grocery store trying to find one specific item, getting stuck behind an incredibly slow checkout line when we're already in a hurry, and somehow leaving with a grocery bill significantly higher than expected. These are small inconveniences individually, but they add up quickly.

And apparently, we're far from alone. According to Progressive Grocer's 2024 Grocery Tech Trends Study, 33% of shoppers cite difficulty finding products as an in-store pain point, while 35% cite the checkout experience. A global IBM study surveying nearly 20,000 consumers found that while 73% prefer physical stores, only 9% are satisfied with their in-store shopping experience.

There's also the issue of unnecessary spending and waste. ReFED's 2026 U.S. Food Waste Report estimates that 29% of the American food supply went unsold or uneaten in 2024, representing $380 billion in surplus food. While this extends far beyond grocery shopping, overbuying and poor planning contribute to the problem.

We wanted to address these inefficiencies together, rather than building another standalone shopping list app, checkout kiosk, or AI chatbot.

One of our biggest inspirations was actually Uniqlo's self-checkout system. We thought it was incredible how you could place a basket full of clothes onto a checkout station and have everything recognized automatically. The catch? Uniqlo uses RFID tags embedded in its merchandise. That works beautifully for clothing, but expecting every grocery store to individually tag every product, especially fresh produce, isn't particularly realistic.

So we decided to recreate that frictionless experience using computer vision instead, while taking it much further with an AI-native assistant that understands both the shopper and the store.

What it does

Kart is an AI-native shopping cart that sees what you buy, knows what you need, and understands the store you're shopping in.

At its core, Kart combines automatic item recognition, intelligent shopping assistance, personalized recommendations, store navigation, and checkout into one connected experience.

Imagine walking into a grocery store and telling Kart, "I'm making pasta for my family of four tonight."

Kart uses the store's actual product catalog to build your shopping list, determine appropriate quantities, and guide you through the aisles in an efficient order. Already have garlic bread at home? Just tell Kart, and it'll remove it from your list.

As you place items into the basket, Kart's onboard computer vision system identifies them automatically and updates your running bill. No barcode scanning, no manually logging products, and no need to unload everything again at checkout.

But Kart doesn't just track what you're buying. It understands why you're buying it.

If you grab spaghetti, Kart might let you know that Parmesan is $1 off a few aisles over. If you add a second box of pasta when your recipe only calls for one, it'll gently ask whether you actually need the extra. And if you change your mind, simply remove the item from the basket, and your bill updates accordingly.

Kart can also remember preferences and future shopping plans for opted-in members. Tell it about an upcoming party, and it can save a list for your next trip.

Importantly, we're not just optimizing for the consumer. We also built a store-facing AI merchant console where retailers can update inventory, prices, aisle locations, and promotions conversationally. Those changes flow directly into active shopping carts, allowing stores to surface relevant deals at the moment customers are making purchasing decisions.

The result is a shopping experience that can save customers time and money while helping stores improve operations, promote products more effectively, and drive relevant additional sales.

How we built it

Before building any of this, we had one rather important logistical problem: we didn't actually own a shopping cart.

We spent a surprising amount of time scouring Facebook Marketplace, eBay, and increasingly suspicious corners of the internet trying to find one. Thankfully, we managed to secure a real shopping cart on day one. Arguably our first major engineering milestone.

With the cart acquired, we began assembling the hardware and software systems.

On the hardware side, we used a Raspberry Pi 4, USB webcams, a portable battery, and custom mounting hardware to turn an ordinary cart into a computer-vision-enabled shopping system.

Our vision pipeline uses OpenCV to detect changes inside the basket. Rather than continuously running an expensive object detection model, the system watches for motion, waits for items to settle, and compares frames to determine what was added or removed.

For product recognition, we built a hybrid system combining a lightweight local image gallery with Claude. We enroll products using a handful of reference images, allowing the Pi to identify familiar objects locally. When confidence is low, the system sends a cropped image to Claude, which identifies the product using the store's actual catalog as a constrained set of possibilities.

Every recognized object is mapped to a real product record, complete with its price, inventory, and aisle location.

We used Supabase and PostgreSQL as our shared backend, handling cart contents, pricing, promotions, routing, shopping plans, and receipts. Supabase Realtime keeps everything synchronized with our native SwiftUI iPad application, which displays the live bill, shopping list, directions, and AI conversations.

For Kart's voice, we integrated Cartesia for real-time speech interaction and Claude for conversational reasoning and tool use. This allows Kart to understand natural language, query live store information, modify shopping plans, and react proactively to events like new items, active discounts, or excessive quantities.

We also built a separate FastAPI-based merchant dashboard, powered by Claude, where store owners can make changes in natural language. These changes are previewed for confirmation before being applied to the database and propagated to active carts.

Altogether, Kart coordinates six interconnected runtimes spanning the Raspberry Pi, iPad, backend, voice agent, AI services, and merchant console. Getting all of these components to communicate reliably was a project in itself.

Challenges we ran into

There were a lot. Probably more than we anticipated when we thought putting AI on a shopping cart would be funny.

The biggest challenge was the sheer scale and interconnectedness of the system. Unlike a purely software-based hackathon project, Kart had to coordinate physical hardware, computer vision, real-time databases, multiple AI agents, a native iPad app, and a live shopping interface.

And everything had to agree on what was actually happening.

For example, what happens when someone puts an item into the basket, removes it, changes their shopping list, and then adds it back? What if a store owner changes a price halfway through someone's trip? What if the internet disconnects after a product has been recognized but before the database confirms the update?

We had to account for dozens of these edge cases, including duplicate detections, unreliable connections, changing shopping plans, inventory mismatches, and conversational ambiguity. We implemented persistent event queues, unique event IDs, centralized pricing logic, and database transactions to ensure that different components didn't contradict one another or accidentally charge for the same item twice.

Computer vision was another enormous challenge.

We initially explored conventional CNN-based object recognition and training our own detector, but collecting and labeling enough images for a changing grocery catalog simply wasn't practical within a hackathon.

Our enrollment-based approach was much more achievable, but it introduced its own problems. Camera angles, shadows, changing lighting conditions, moving hands, overlapping items, and even slight movements of the basket liner could confuse the system.

We spent hours manually enrolling products, inspecting raw camera frames, adjusting detection regions and thresholds, and replaying recorded footage to understand what the model was actually seeing. It was a very hands-on, human-in-the-loop process, with lots of trial and error.

At one point, a single item could be interpreted as two separate placements. At another, two touching products could be mistaken for one. We had to refine how Kart tracked individual objects and introduce confidence thresholds so uncertain detections would trigger a quick shopper confirmation rather than an incorrect charge.

And then there was the demo.

We were determined to film Kart working in an actual store, not just in a controlled room with some groceries on a table.

Unfortunately, by the time the system was ready for filming, it was around 2 AM on Sunday, the day of submission. Virtually everything was closed.

Our only realistic option was a tiny convenience store near Berkeley. Somehow, we convinced the owner to let us bring our modified shopping cart inside and film.

The store was cramped, we had to stage our products around its shelves, and we constantly waited for foot traffic to clear before recording. Meanwhile, a steady stream of post-partygoers, some a little more intoxicated than others, walked past us and offered their unsolicited (and not always particularly encouraging) opinions about Kart.

So there we were, at 2 AM, awkwardly maneuvering a shopping cart covered in cameras and electronics around a convenience store while trying to get our entire AI system working on camera.

Not exactly the glamorous startup experience we had imagined.

But we eventually pulled it off, and we're incredibly glad we did. Seeing Kart respond to products being placed into the basket, guide us through a real store, and update its interface in real time was far more convincing than any simulated demo could have been.

What we learned

Our biggest takeaway was that building an AI-native product is much more than connecting an LLM to an interface.

The difficult part is grounding that intelligence in the physical world: making sure the agent knows what's actually in the basket, understands what the store sells, reacts to changing information, and behaves sensibly when something goes wrong.

We also learned just how much complexity emerges when software has to interact with real hardware and real people. A perfectly functioning backend doesn't mean much if a camera misidentifies a product, and an impressive AI agent isn't particularly useful if its recommendations don't reflect actual inventory.

Most importantly, this project reinforced our belief that some of the most interesting applications of AI aren't necessarily new interfaces. Sometimes, they're familiar, everyday objects that can be fundamentally reimagined.

Kart is still a hackathon prototype. Our payment terminal is simulated, and the vision system needs more work to handle crowded baskets, stacked products, and changing store environments reliably. But getting the full experience working together on a real shopping cart was a huge first step.

What started as an excuse to acquire a shopping cart for some funny hackathon content became a much more ambitious attempt to rethink physical retail.

Because ultimately, grocery shopping shouldn't require you to think about finding products, tracking prices, checking out, or managing your budget. You should just be able to shop, and let Kart handle the rest.

Built With

Share this project:

Updates

Submission history