Inspiration
Every bank app has the same blind spot. It knows you spent $84.12 at Amazon; it has no idea that was a week of groceries and a phone case. It knows nothing about the $12 you handed a food truck. So the category charts are wrong, cash is invisible, and "where did my money go?" never gets a real answer. We're students who live on exactly this kind of spending. We wanted a ledger that knows what's inside every charge, and we wanted capturing it to take two taps, not two minutes.
What it does
Double-tap the back of your iPhone over a receipt. Penny photographs it, reads every line item, finds the matching charge in your bank account, and your dashboard updates while you're still putting your wallet away.
- Itemized spending.
AMAZON $84.12becomes $61.14 of groceries and $22.98 of shopping, with tax split in proportion. Category charts finally reflect what you bought, not where you bought it. - Matching that explains itself.
CHIPOTLE 2741 AMHERST $27.40is linked to your $23.40 Chipotle receipt, recognizing the $4 tip and the messy bank name. Every match shows its confidence and why. - Cash, finally counted. A receipt with no bank charge is recorded as cash: money your bank has never seen.
- Subscriptions, detected. Recurring charges are found automatically, with the monthly total and the next expected date.
- Ask AI you can trust. "How much did I spend on food this month?" gets an exact answer, and the tools it used are shown under every reply.
How we built it
Capture. An Apple Shortcut bound to Back Tap takes the photo, resizes it and posts it to our FastAPI backend. Nothing to install.
Extraction. Gemini turns the photo into structured JSON: merchant, date, totals and every line item, each with a category from a fixed list.
Bank. Capital One's Nessie API is our bank. We seeded 60 days of realistic purchases (groceries, coffee, Uber, four subscriptions, one big Best Buy) and sync them into MongoDB Atlas.
Matching is deterministic code, not an LLM, so every decision is explainable and unit-tested. Candidates are unmatched charges within ±2 days. The amount must match to the cent, or be up to 30% higher at restaurants (the tip). Merchant names are normalized and fuzzy-matched with RapidFuzz. Then:
$$\text{confidence} = 0.5\cdot\text{merchant} + 0.3\cdot\text{amount} + 0.2\cdot\text{date}$$
The highest candidate wins. Below 0.65 it stays cash, because a wrong match is worse than an honest "cash".
Ask AI is an agent with tools, not a chatbot with a data dump. Gemini never sees raw transactions and never does arithmetic. It chooses among six tools (spending by category, category breakdown, top merchants, subscriptions, cash vs card, receipt item search). Each tool is a Python function running a real MongoDB query through the same code as the dashboard, so the AI and the charts cannot disagree. Gemini chains calls on its own: asked "how much cash?", it checked this month, then widened the range to be sure. When it once misspelled an argument (Categories), it read the error and called again correctly.
Frontend. Next.js, Tailwind and Recharts, with "show the reasoning" panels for every match, subscription and AI answer.
Hosting. The backend runs on Vultr as a systemd service behind Caddy for automatic HTTPS. The iPhone and the dashboard talk to one public endpoint.
The demo, by design
Four receipts, each proving one thing:
| Receipt | Bank charge | Shows |
|---|---|---|
| Amazon, $84.12 | AMAZON MKTPLACE $84.12 |
one charge split into two categories |
| Chipotle, $23.40 | CHIPOTLE 2741 AMHERST $27.40 |
tip tolerance and messy bank names |
| Stop & Shop, $47.85 | STOP & SHOP #0412 $47.85, posted next day |
charges that post late |
| Taco truck, $12.00 | none | cash the bank never sees |
Challenges we ran into
- Nessie drops cents. $27.40 came back as $27, which would break exact matching. We store whole cents and convert in one place.
- One store, two names. After a match, "Amazon" and "AMAZON MKTPLACE" counted as different merchants. We now group by the bank's name and display the receipt's.
- Gemini rate limits mid-test. We fall back across Flash models so a busy one doesn't fail a scan.
- Three people, one contract. We wrote the API spec first and built in parallel. The branches merged with zero conflicts.
Accomplishments that we're proud of
- The whole loop works live, in seconds, from a phone tap to an updated dashboard.
- Every number Penny shows, including AI answers, comes from deterministic, tested code: 31 tests, 10 endpoints, 4 external services.
- Matching handles tips, late posting and messy names, and shows its reasoning.
- It was our first time building with AI coding tools, and we shipped and deployed.
What we learned
- Let the LLM decide what to compute and Python decide how. That's the difference between an AI answer you can check and one you have to trust.
- A written contract between teammates is worth more than any meeting.
- Deploying properly (HTTPS, SSH keys, a service that restarts itself) is within reach in a weekend.
What's next for Penny
- Real bank connections instead of a mock bank.
- Splitting shared receipts between friends.
- Flagging unusual purchases against your own history.
- Budgets per category built on item-level data, not merchant guesses.