Inspiration

At home I kept running out of things at the worst moment: the last of the milk, the last toilet paper, the dish soap halfway through the dishes. You only find out when you reach for it, and by then it's too late. I wanted something that knows what's about to run out and deals with it first.

What it does

Replenish keeps track of the things your household uses up. You add what you buy by snapping a photo of a receipt or typing it in. It learns how fast each item goes, works out when it will run out, and says how sure it is. When something runs low, it finds it online and asks you first, or buys it for you within limits you set. Auto-buy is off by default and is switched on per product.

How we built it

  • Next.js, React, Tailwind and Postgres (Neon).
  • Receipts: Google Cloud Vision reads the photo. My own code rebuilds the printed rows from the words' positions, since receipts are curled and tilted. A dictionary then turns "GV WHL MLK GL" into Milk. You review the result before anything is saved.
  • Predictions: plain, tested maths. The usual purchase is the median of recent ones, recent gaps count more than old ones, and two unusual gaps in a row count as a new habit.
  • Finding the product: rules filter out listings that are out of stock, used or the wrong kind of thing. Gemini then reviews the shortlist and explains its pick, but it can't override any limit.
  • Paying: PayPal charges the user (Vault for the saved account, Orders for the charge), Zinc buys the item, and PayPal refunds whatever wasn't used. If the purchase fails, everything is refunded. The charge is a ceiling:

$$\text{charge} = 1.3 \times \text{price} + \$1 + \max(0.07 \times \text{price},\ \$1.25)$$

Each step is saved before the next, and every outside call carries the order ID, so a retry can never charge or buy twice.

Challenges we ran into

  • Receipts are messy. Prices land one row off, and abbreviations are cryptic. I added a regression test for every real receipt that broke it.
  • Charge, then refund. PayPal documents an immediate capture for a saved account, not authorise-then-void, so I built the pipeline around refunds.
  • Auto-buy kept teaching itself the wrong lesson. Orders arrive early on purpose, and learning from them made each order earlier than the last. Those gaps are now left out of the rate.
  • People are wrong too. A typed stock count can be a typo, so the app asks "are you sure?" and trusts what it measures more than what it was told.
  • Safety with money. There are per-order, monthly and fee limits, and test money can never be mixed with real money.

Accomplishments that we're proud of

  • It works end to end: a receipt photo becomes a prediction, then a found product, then a charge, a purchase and a refund of the unused part.
  • Auto-buy is built to be cautious. It asks whenever any limit is in doubt, shows every fee before it spends anything, and never learns from its own orders.
  • It's deployed and running at https://replenish.jeffawe.dev, with a demo account for judges.

What we learned

Assume anything can be wrong, including what the user tells you. Never block them. Ask fewer questions as the app learns.

What's next for Replenish

More ways to add purchases, such as email order confirmations. Perishables will be ask-only rather than auto-bought. After that, PayPal webhooks and returns.

Everything runs in PayPal's sandbox, so no real money moves.

Built With

Share this project:

Updates

Submission history