Inspiration

We run paid renovation jobs in Switzerland. The most expensive problem is not fraud, it is drift: a stage is called "done" while a few items on the list are quietly skipped, and nobody notices until fixing it means tearing out finished work.

What it does

  • The owner funds a renovation stage with PayPal. The money is authorized and held, not paid.
  • Each stage has a checklist of line items. The crew closes every item with a phone photo.
  • Claude checks each photo against what "done" should look like for that item. It answers done, not done or unclear, with a reason and clear instructions for retaking the photo.
  • If anything is missing, the money stays on hold and the crew gets a task, for example "Re-upload photo for: Remove old tiles (floor)".
  • When everything is proven, an AI payment agent prepares the payout. The owner signs, PayPal captures the hold and a Payout sends the money to the crew.
  • Every photo, verdict, signature and PayPal ID is saved to a tamper-evident timeline.
  • All jobs, names and photos in the demo are fictional. PayPal runs in Sandbox.

How we built it

PayPal

  • Orders v2 with intent: AUTHORIZE (hold the money, don't charge) through the PayPal JS SDK buttons
  • Payments v2 authorize, capture and void
  • Payouts to the crew's PayPal

Everything runs in Sandbox. Amounts always come from the server, and capture uses an idempotent PayPal-Request-Id.

AI

  • Claude Opus 5.5 (claude-opus-5-5) with vision and structured JSON output: verdict, confidence, reason, what_to_reshoot.
  • A second Claude call acts as the payment agent: it reads the stage and proposes one action (release, hold and re-shoot, remind the crew, fund). The server checks every proposal, and a person always presses the button.
  • The live demo runs Claude through a local bridge; the repo also supports the Anthropic Messages API with your own key.

Fraud guard: image fingerprints (perceptual hash and SHA-256) reject the same photo used as proof for two items. Uploads are re-encoded, which removes hidden location data.

Stack: Python, FastAPI, vanilla JS, SQLite, Pillow, Docker. Each visitor gets their own demo workspace, and AI checks are rate-limited.

Challenges we ran into

  • Holding money without paying it out. Modeling a stage as a PayPal authorization meant handling three outcomes cleanly: capture, void and expiry. The owner also needs to see when the hold expires.
  • Getting strict, honest AI verdicts. The model must say "unclear" instead of guessing. In testing it refused a pressure-gauge photo because the dial was unreadable. That is exactly the behavior you want before money moves.
  • Keeping the agent useful but harmless. It drafts the next payment step, while the server and a human signature decide.

Accomplishments that we're proud of

  • It catches the skipped item. Claude spots old floor tiles left in one corner (95% confidence), the money does not move, and after the re-shoot the item passes.
  • Every step is accounted for: each one carries real PayPal Sandbox IDs on a tamper-evident timeline.

What we learned

  • Authorization plus capture is a natural escrow for stage-based work.
  • Structured output turns a vision model into a reliable inspector.
  • The best agent prepares a decision; it does not take it.

What's next

  • Stage-based payouts for lenders and insurers after storms, for example Florida hurricane repairs.
  • Multi-party escrow for long projects.
  • Pilots on real PlanMy.Space renovation jobs.

Try it

Built With

Share this project:

Updates

Submission history