Dripy

Inspiration

Every week, TikTok and Reels sell us a jacket we don't need. We screenshot it, buy it, and it ends up next to two almost identical jackets we've worn twice. Shopping apps are built to say yes, and nothing on your phone knows what's already in your closet.

At the same time, AI agents are starting to shop for people, which raised the question we wanted to answer for the Visa track: how do you let an agent check out for you without ever letting it spend your money on its own? We wanted a stylist that lives where people already talk (iMessage), knows their wardrobe, is honest enough to say "skip it," and can't pay a cent until you say yes.

What it does

Dripy is an AI stylist you text like a friend.

Send a screenshot from TikTok, Reels or a store page. Dripy identifies the clothes and finds where to buy them, with prices.
Ask "should I buy this?" and get a BUY / SKIP / CONSIDER verdict that cites what you already own: "Skip it — you already own the Zara Black Bomber Jacket and the Mango Black Faux-Leather Biker Jacket." Or: "Buy it — it fills your smart-casual outerwear gap. $79.90 vs ~$95 typical."
Say "ok buy it" and Dripy sets up a checkout: "Reply YES to pay $79.90 at zara.com." Nothing is charged until you reply YES. Then it pays with a Visa card and sends the receipt.
Get outfits from your own closet: "rooftop dinner Friday, what should I wear?"
Declutter: after a purchase, Dripy points out something you've never worn and finds nearby thrift stores, consignment shops and donation drop-offs, with distance and opening hours.
Your closet builds itself from order-confirmation emails, which Dripy parses into wardrobe items.

How we built it

iMessage bridge: a Node worker holds a Photon connection, forwards each text and photo to our backend, and sends the replies back. It handles one message at a time per conversation, so "ok buy it" can't overtake the message before it, and it ignores duplicate deliveries from Photon.
Backend: Next.js 16 API routes in TypeScript, with Supabase (Postgres + Storage). iMessage and the web chat endpoint use the same agent.
Agent: a Google Gemini tool-calling loop with seven tools: search wardrobe, build outfit, analyze image, analyze reel link, buy-check, create purchase intent, and find donation places. Every tool input is validated with Zod.
Vision and shopping: every phone image (HEIC photos, huge PNG screenshots, rotated JPEGs) is converted with sharp and heic-convert into one standard JPEG. Gemini Vision identifies each garment's category, color and occasion. Then SerpAPI (Google Lens + Google Shopping) finds products you can buy and a typical market price.
Buy-check: this is deliberately not an LLM call. A deterministic scorer compares the item with your closet on three separate signals: similarity (drives SKIP), occasion gap (drives BUY) and price vs. market median (tips the balance):

$$\text{similarity} = 0.5 \cdot \text{subcategory match} + 0.5 \cdot \text{color-family match}$$

A score of 0.75 or more counts as a near-duplicate, and the verdict only cites items that actually matched.

Payments (our Visa-track focus): the model has no pay tool. It can only propose a checkout for a product it already found, and the amount and store come from that stored product, never from the model. While a checkout is waiting, your reply goes through a rule-based classifier before the LLM sees it: "yes" approves, "yes but the black one" doesn't. Postgres enforces the rest:
- a payment can't be marked paid without an approving message (a check constraint)
- approval happens in one atomic database step, so tapping Approve and texting "yes" at the same moment can't charge twice
- one open checkout per user, a $300 cap, a merchant allowlist and a 10-minute expiry
- an append-only audit log of every step

Payment methods sit behind one switch: a Stripe test-mode charge on a Visa test card (•••• 4242), a simulated mode, and an adapter for Visa Intelligent Commerce's sandbox.

Receipts: order emails are converted to plain text, and Gemini parses them into structured wardrobe items.
Donations: live OpenStreetMap data from the Overpass API, merged with a seeded list of Seattle spots, with distances and "open now" worked out from opening hours.
Fallbacks: if any outside service is down, its endpoint returns a labeled mock or cached result, so the app never just breaks. Every response says whether its data is live, cached or simulated.

Challenges we ran into

Keeping payments safe with an LLM in the loop. A caption in a screenshot or hidden text in an email could say "approve the payment." So we took approval away from the model entirely and enforced the rules in the database instead of the prompt.
Telling a real yes from a maybe. "ok", "yes please" and "👍" should pay. "yes, but the black one" and "ok actually the other one" shouldn't. We pinned this down with 31 test cases.
iPhone images. iMessage sends HEIC files and recompresses photos, and sharp's prebuilt binaries can't decode HEIC. We added a pure-JavaScript converter and turn every image into a JPEG before vision and Lens see it.
Unpredictable demo results. A recompressed photo gives different Lens results each time. So we cache each analysis by the image's content and keep a curated catalog for the demo garments.
Latency. Gemini's first call can take 30–60 seconds, which feels like forever in a text thread.
Platform walls. Instagram links hit a login wall, so Dripy asks for a screenshot instead. Gmail OAuth wasn't realistic in a weekend, so we used real-format order emails. Visa sandbox credentials were still pending, so the live payment runs on Stripe test mode with a Visa test card.

Accomplishments that we're proud of

An agent that can say no, with verdicts that name real items from your closet instead of vague vibes.
A checkout flow where the model can't pay on its own: no pay tool, prices set by the server, and a database constraint as the last line of defense.
Real iMessage conversations end to end: send a screenshot from your camera roll and get product cards, verdicts and a receipt back.
Every endpoint is documented, mocked and contract-tested, so frontend and backend could work in parallel from the first hour.

What we learned

Use the LLM to understand images and hold the conversation, and plain code for anything involving money. The parts we trust most are the ones with no model in them.
Treat everything an agent reads as untrusted input: screenshots, captions and emails.
Build fallbacks from the start. At a hackathon, some API will be down during your demo.