EarmarkPay

Purpose-locked payments for families: PayPal authorizes and holds the money, Gemini AI checks each bill for purpose and fraud, and only the verified amount is captured and paid out.

Comment

Inspiration

Millions of people send money home every month for something specific: a parent's diabetes medicine, a sibling's school fees, a family's groceries. Once the money is sent, the sender has no way of knowing it was used for that. The only options are to trust blindly or to call and ask. Donors who help individual families face the same problem. We wanted to let people send money for a purpose, not on a promise.

What it does

EarmarkPay lets a sender set money aside for a purpose and release it only when a verified bill proves it was used for that purpose.

  1. Rules in plain language: the sender writes "Only medicine and groceries. No alcohol. Ask me before paying any bill over $100." Gemini turns this into a structured policy.
  2. PayPal holds the money: the sender approves a PayPal authorization. The money is reserved, not sent.
  3. The recipient photographs a bill: Gemini's vision model reads the merchant, date, every line item, tax and total, and labels each item's category.
  4. Item-level payment: a pharmacy bill with medicine and beer pays for the medicine only.
  5. PayPal pays exactly what's verified: the app captures that amount from the authorization and sends it to the recipient with PayPal Payouts.
  6. Close anytime: the remaining authorization is voided, so unused money never leaves the sender's account.

It also has an AI agent (Gemini function calling) that can create earmarks, find bills waiting for review, approve or reject them, and close earmarks. It writes spending reports in the family's language (Hindi, Spanish, Tagalog and more), and all activity is tracked in AG Grid ledgers.

Fraud protection

Bill photos can be faked, so every bill passes a series of checks, and each bill gets a 0–100 risk score:

  • Duplicates: a SHA-256 hash of the file, plus merchant + date + total, catch reused bills.
  • Arithmetic, checked by code, not by the AI: a bill passes only if $$\left|\sum_i p_i + \text{tax} - \text{discount} - \text{total}\right| \le \max(1,\ 0.03 \cdot \text{total})$$
  • Indian GST numbers are validated with their government mod-36 check digit and state code.
  • Bill-splitting (several bills just under the review limit) and too many bills in a day are flagged.
  • A recipient trust score switches off auto-pay after repeated rejections.
  • The AI flags visible editing, such as a total in a different font from the rest of the bill.

Risky or large bills always wait for the sender's approval, and every decision is explained check by check.

How we built it

  • PayPal: Orders v2 with intent: AUTHORIZE to hold funds, the JS SDK (via @paypal/react-paypal-js) for approval, Payments v2 partial capture and void on the authorization, and Payouts v1 to pay the recipient. Everything runs in the PayPal sandbox.
  • AI (Google Gemini): vision + JSON mode to read bills, JSON mode to parse rules, function calling for the agent, and text generation for multilingual reports.
  • Design decision: the AI reads the bill, but deterministic code decides what happens to the money. A rule engine turns each check into pass / warn / fail, then approve / review / reject, so every payment is explainable and the AI can't move money on its own.
  • Stack: Node.js + Express backend, React 18 + Vite + Tailwind CSS v4 frontend, AG Grid Community for ledgers, deployed on Render. The app also runs fully in a demo mode with no keys, so anyone can try it.

Challenges we ran into

  • PayPal payouts from India: our default sandbox app was linked to an India business account, which can't send payouts. We created a US sandbox business account and linked a new app to it.
  • Overloaded free AI models: some Gemini models were unavailable to new keys or returned "high demand" errors. We added automatic retries, a fallback chain of models, and timeouts, so a busy moment doesn't break the payment flow.
  • False alarms: the AI first flagged perfectly normal digital bills as "templates." We rewrote the prompt to require concrete evidence of tampering, and moved arithmetic checks out of the AI into code.
  • Partial captures: releasing money bill by bill means capturing an authorization in pieces, then voiding the rest. Getting the amounts, final-capture flags and void timing right took care.

Accomplishments that we're proud of

  • A real end-to-end money flow on PayPal: hold, verify, capture exactly, pay out, void the rest.
  • The AI correctly reads real-looking bills, excludes off-purpose items, and catches an edited total, including spotting the different font.
  • Fraud checks that are explainable, and a design where the worst case is losing one bill, never the whole amount.

What we learned

  • PayPal's authorize → capture → void flow is a strong fit for conditional payments: holding money is often more useful than moving it.
  • LLMs are excellent at reading messy real-world documents, but money decisions should stay in deterministic, auditable code.
  • Honest threat modelling matters: we documented what we can't catch yet, instead of claiming fraud-proof.

What's next for EarmarkPay

Our goal is to make fake bills impossible by making the payment itself the proof:

  1. Pay shops directly, locked by category: money goes only to approved pharmacies, groceries or schools, verified by card-network merchant category codes.
  2. Partner merchants on PayPal send itemized orders from their own checkout systems.
  3. Government-signed e-invoices: verify India's GST e-invoice QR codes, which can't be forged.
  4. E-prescription matching for medicine.
  5. Recurring monthly earmarks, WhatsApp bill upload, and PayPal webhooks for real-time payout confirmation.

Built With

Share this project:

Updates

Submission history