ClipCart

Checkout Demo
Chrome Extension
Analysis 1
Agent Activity Live

Inspiration

Have you ever been scrolling Instagram and thought, “Holy shit, I want that jacket”?

Then you open the comments, find five people asking the same question, and realize nobody has an answer. You take a screenshot, try describing it to a search engine, and end up comparing vaguely similar products across twenty tabs.

That gap became the inspiration for ClipCart.

Ariana Grande described the ideal experience: “I see it, I like it, I want it, I got it.” I wanted to build the missing steps between seeing something and knowing where to find it.

The bigger question was what happens afterward. Could the experience remember what I liked, respect my budget, and make the next search more useful? ClipCart became my attempt to connect visual discovery, shopping research, personal style and transaction evidence.

What it does

ClipCart turns something you see in a video into a shopping research workflow.

Capture what catches your eye. A Chrome extension lets you capture a visible frame, crop the piece you want, and open it in ClipCart with its source page. You can also upload photos or videos directly.

Understand the item. Perceptron mk1.5 identifies pieces and extracts visible attributes, readable text and bounding boxes. For uploaded videos, Find in video helps locate an item from a description and jumps to a frame where it appears.

Shop with constraints. Add a budget, size and preferences. ClipCart retrieves catalog listings and separates useful candidates from results that need more checking.

Dispatch a research agent. Grok Bot receives the item description, shopper constraints and a temporary crop link. It researches retailer and resale pages, then returns structured offers through an authenticated callback. Offers carry their source, price evidence and uncertainty.

Build a personal wardrobe. Save items, hide unwanted results, confirm purchases and inspect inferred preferences. Saved items, opened links and purchases remain distinct. You can correct or remove preferences instead of accepting an invisible personalization system.

Explore checkout and receipts. A separate controlled merchant flow supports Solana devnet payments and compressed NFT receipts. It demonstrates payment verification and buyer-owned receipt evidence using test funds, without claiming to fulfill external retailer listings.

How I built it

ClipCart combines a React and TypeScript web app, a Chrome Manifest V3 extension and a Python FastAPI backend.

The media pipeline validates and decodes uploads, handles orientation, samples timestamped frames and produces crops. Perceptron observations use a consistent normalized coordinate format so detection, selection and overlays agree.

The shopping pipeline combines SoleScout catalog retrieval with asynchronous Grok Bot research. A webhook dispatches each research job. The agent receives an expiring signed crop URL and submits results using one-time callback authorization. The backend validates the response, stores accepted offers and exposes job progress to the browser.

The verification layer applies category and shopping constraints independently of the agent. An agent’s exact-match claim cannot establish verified identity. Unpriced offers never become confirmed budget matches or lowest-price winners.

MongoDB Atlas stores sessions, observations, research jobs, preferences, wardrobe events and owner-scoped retailer links. I verified owner isolation and restart persistence against Atlas.

The memory layer turns explicit preferences and supported interaction history into a compact profile. Personalization can reorder results while preserving hard constraints.

I also benchmarked hosted Gemma 3 12B on a Lambda A10 for preference interpretation. The current demo uses the rule-based reader because it achieved higher precision in my small evaluation. Model proposals pass validation and require confirmation before becoming saved preferences.

The Solana prototype verifies the payment’s network, success, recipient, amount, reference, buyer and order window. Its receipt worker uses Bubblegum V2, durable attempts and concurrency controls, with DAS used to verify compressed asset ownership.

Challenges I ran into

Connecting the whole experience. The extension, browser session, media pipeline, research agent and wardrobe all needed consistent ownership and contracts. I fixed a fresh-session race that could leave an upload pointing at another newly created owner’s session.

Making agent research trustworthy. Product pages expose uneven information. A title might identify a jacket while saying nothing about fabric or size availability. I built explicit uncertainty states rather than treating missing evidence as a match.

Moving beyond catalog coverage. SoleScout provided a working starting point, but its coverage and available seller information were limited. The research agent expanded discovery across independent pages.

Keeping provenance intact. A stale unpacked extension initially omitted source metadata. Tracing the exact folder Chrome loaded revealed the mismatch, and updating it restored the intended handoff in regression tests.

Handling durable transactions. Real MongoDB exposed a unique-index collision between unpaid orders. Concurrent receipt requests also exposed a duplicate-mint race. Fixing those required partial indexes, atomic claims and signed-attempt persistence.

Accomplishments that I'm proud of

Demonstrated a real extension capture, live Perceptron analysis, shopping search and retailer redirect.
Completed live Grok research in which the agent fetched the actual crop, returned an authenticated callback and persisted five offers in Atlas.
Independently checked two returned product pages against their reported titles and prices.
Demonstrated saved items, shopper-confirmed purchases and an inferred brand preference.
Verified a real devnet payment and compressed receipt ownership on-chain.
Reached 458 passing backend tests on the combined release, alongside generated-contract consistency and a clean web build.

What I learned

The hardest part of agentic commerce is preserving meaning between steps.

A click is not a purchase. A cheap alternative is not necessarily the same product. A model’s confidence is not evidence. Those distinctions belong in the data model and interface.

I also learned that personalization needs controls. A useful system should explain what it remembers and let the shopper change it.

Finally, model selection should follow measurement. Testing Gemma against a simpler parser helped me choose reliability for the current experience rather than adding a model merely because I could.

What's next for ClipCart

I want to reduce research latency, expand retailer coverage and improve product-image comparison.

Other priorities include broader live validation for watches and jewelry, wardrobe actions for research offers, stronger stock and size evidence, and a more polished checkout-to-receipt experience.

Longer term, ClipCart could connect visual inspiration to affiliate-supported discovery across stores, while carrying the shopper’s preferences forward.

The goal is simple: you should be able to start shopping with something you saw, even when you don’t know what it’s called.