Project Story

About the project

Inspiration

When people get older and their vision weakens, shopping stops being something they can do alone. A trip to the pharmacy turns into asking for help. We were inspired by a neighbor who struggled with her vision and wanted to shop for herself again, without asking someone to read labels and prices for her. We wanted to give her that independence back.

The problem is bigger than one neighbor. The CDC estimates that about 7.1 million people in the United States live with vision loss that glasses can't fully correct, and about 706,000 of them are blind (CDC VEHSS, 2024 modeled estimates). Health savings add a second hurdle. HSAs now hold about \$174 billion across 41.7 million accounts (Devenir, year-end 2025), and an EBRI-based analysis found that 47% of FSA holders forfeited money in 2023, an average of \$422 each (Money, March 2025). Working out which card to use for which item, while you can't read the shelf tag, is a hard problem to face alone.

What it does

ReWind lets someone point their phone at a product and hear what it is, what it costs, and which card to pay with. Everyday items go on a Visa. Eligible health items go on the FSA first, then the HSA, then the Visa. At checkout it reads back what goes on each card, and it never reads out a card balance.

How we built it

Frontend. An Expo (React Native) app built around the camera. A live camera view, a solid card with the product's details at the top, and a caption for every spoken line. During onboarding the user scans their payment cards with the camera. We keep only the card type and the last four digits, and the photo is never stored.
Backend. A FastAPI server on Render. One scan request takes the camera frame through every step and returns the spoken answer, so the phone makes one call per item.
Vision and eligibility. Claude vision reads the frame and returns the product name, brand, size, shelf price, and whether it's a medication. Before the demo, we used Apify to scrape product listings from HSA/FSA stores, mostly fsastore.com, and loaded them into a Moss index. The listing decides eligibility. An item that isn't in the listings is treated as not eligible and goes to the Visa. If Moss can't be reached, items are also treated as not eligible, so an outage can never make something look eligible.
Prices. The price comes from the shelf tag. If the tag can't be read, it comes from the store listing. If there is no listing either, Claude searches the web and the app says the price is an estimate. The demo items use a fixed price so every run is the same.
Card routing. Plain code, never a language model, because a money decision can't be wrong. If a card can't cover its items, the app offers to move them to the next card and asks for a yes.
Checkout and data. Supabase Postgres stores the cards, cart, scans and payments. Checkout is one all-or-nothing transaction that is safe to retry. Every "yes" is tied to one question, works once, and expires, so nothing is added or paid by accident. Capital One's Nessie holds the demo accounts and records each payment, and the invoice is built from the saved payment. All data is mock and payments are simulated.
Voice. Cartesia Ink turns speech into text and Cartesia Sonic turns replies into audio. The server splits each reply into sentences, so the first one is ready quickly and the rest are made while it plays. If Cartesia is down, the phone speaks with its own voice.

Challenges we ran into

Deciding what counts as eligible. We started with IRS category rules, then replaced them with real store listings, scraped with Apify and searched with Moss, as the only source. The tradeoff is that a product has to be in those listings for us to confirm it. Anything we can't confirm goes to the Visa, never to a health card.
Nessie doesn't move money. We planned for Nessie to hold the card balances. When we tested it, an account's balance never changed after creation. Deposits, withdrawals and updates were all accepted and then ignored. So Nessie holds the demo accounts and records each payment, and our own database keeps the live balances.
Getting payments exactly right. A wrong charge is the worst failure for this app. A retried request is recognized and charged once. A payment only goes through for the split the user heard. If the cart or a balance changed in between, the app asks again instead of charging.
Designing for someone who can't see the screen. Every confirmation and every error has to be spoken clearly. We decided early that balances are never read aloud, because saying one in a store exposes it to anyone nearby. When a card can't cover everything, the app doesn't say "declined". It offers to move the items to the next card and asks for a simple yes.

What we learned

The hard part of building for accessibility isn't adding voice. It's making sure someone who can't see the screen never needs to. We learned that when we can't confirm an item, it's better not to guess: the app names it, gives the price, and puts it on the Visa, never on a health card. And we learned that for everyday items, not bringing up HSA at all works better than saying "not HSA eligible", because health-account terms only confuse people when the item is a bag of chips.

We also learned how much people like our neighbor lose when they can't shop on their own. It's not just about groceries. It's about dignity. ReWind doesn't just read labels. It helps her walk out of the store knowing what she paid, which card she paid with, and that her HSA or FSA was used where it applies.

What's next for ReWind

Test with organizations that serve blind and low-vision communities, with real users
Link real HSA and FSA accounts through an aggregator such as Plaid, with the user's consent
Move insurance from mock profiles to live formulary lookups through pharmacy benefit managers
Reimbursement help: when an eligible item was paid on another card, prepare the itemized receipt so the user can file a claim with their HSA or FSA provider
Work with pharmacies and retailers that want accessible checkout for their customers