-
-
Paid $3,700, photos prove $1,050 of work, claim $2,650
-
The promised items, read from the estimate and the text messages
-
Each unfinished item with the photo the model judged; the claim is computed in code
-
The contractor's $1,200 offer next to the $2,650 claim (PayPal sandbox)
-
PayPal decided for the homeowner: $2,650 back (sandbox claim PP-R-DVT-10197552)
Inspiration
Paid the contractor $3,700. The job is half done. Claim: $2,650. You file it yourself on PayPal.
That is our demo case. A homeowner paid Ridgeway Electric (a made-up contractor) $3,700 up front through PayPal for six promised items. The outlets and the ceiling fan are in, and four of the six recessed lights are installed. The panel upgrade, the flood lights and the EV charger circuit never happened. After two weeks with nobody on site, the contractor texted: "All done on our end!"
California's contractor board opened 20,092 complaints against contractors in 2025 and says most consumer complaints it gets allege "incomplete or defective work" (CSLB 2025 Accomplishments and Activities Report). In Illinois, construction and home improvement was the largest category of written complaints to the Attorney General in 2024, 2,256 of 18,901, and consumers cited "the complete failure by companies to conduct any work after receiving a down payment".
Escrow-style tools hold the money until the work is done. Make It Right is for a job you have already paid for.
What it does
- Reads the estimate (PDF), the text messages and the job-site photos, and lists every promised item with where it was promised ("estimate p.1", "text Sep 9").
- Judges each item from the photos: done, partly done ("4 of 6") or not done. A contractor's text saying "all done" never counts as proof.
- Computes the claim in code: $3,700 paid, $1,050 of work proven by photos, $2,650 for the unfinished work, item by item, the way PayPal's claim form asks for it.
- Shows the payment, the day count against PayPal's 180-day window and the claim reason, "Service not received".
- You file the claim on PayPal's site. The app finds it, reads your transaction id from it, and follows the contractor's offer, your answer, the escalation and the refund. It never files a claim or accepts an offer for you.
We attacked the photo check before trusting it with 18 hostile files on the demo job: an estimate with a hidden line telling the model to mark every item done, a contractor text saying the job is finished, one photo reused for two items, a photo from another job and more. 16 of 18 were repelled (one by the upload check before the model, 15 by the model's own answers), and 5 of 5 honest jobs passed unchanged. All 18 cases, the two we missed among them, with the model's answers: https://github.com/retailbox-automation/make-it-right/tree/main/eval/hostile; replay them on the demo: https://makeitright.retailbox-automation.com/hostile
On 9 Nov this demo case went all the way on the hosted demo, in PayPal's sandbox: a $2,650 claim, a $1,200 offer, escalation and PayPal's decision, with $2,650 back to the homeowner within 20 minutes of filing (steps below). The demo case is made up and its photos are generated, one photo per promised item. On the hosted demo the photo check replays the model's recorded answer for this case; the PayPal steps are live in the sandbox.
How we built it
PayPal, sandbox only:
- Orders v2: the payment is created with the promised items as order items, so PayPal's claim form lists them one by one, and is captured after the homeowner approves it. Every order carries a
custom_idstarting withMIR-. - Disputes API: a tracker lists claims every minute, matches them to our payments and reads the homeowner's transaction id from the claim, then maps PayPal's answers to offer, escalation and outcome. A contractor stand-in makes an offer (
make-offer) or accepts the claim (accept-claim); the case can be escalated (escalate); in the sandbox the app can ask PayPal to decide (adjudicate). - No webhooks: the PayPal app is shared, so the app polls.
AI: Gemma 4 31B, an open model, runs on our own machine with Ollama (temperature 0, strict JSON schema). It reads the estimate, the texts and the photos, lists the promised items and judges each one. It never computes an amount; every amount comes from code, in integer cents. The hosted demo has no model on the server. It replays the recorded answer of Gemma 4 31B for the demo case (6 Nov 2026); on those photos the model gave the same claim, $2,650.00, in 5 runs out of 5. The PayPal calls on the hosted demo are live, in the sandbox.
We measured the photo check on a synthetic set of jobs with real job-site photos from Wikimedia Commons. On 26 development jobs (100 items) it got 87 % of verdicts right and 18 of 26 claims exact to the cent; on 12 held-out jobs (48 items), 91.7 % and 10 of 12. A baseline without a model ("everything not done") gets 32 %.
Node 22, TypeScript, Hono, PGlite and pdf.js; 718 automated tests. Hosted on Zeabur; the service reads its sandbox keys from a vault when it starts.
Proved in the PayPal sandbox
On 9 Nov 2026 the demo case went all the way on the hosted demo:
| Step | UTC |
|---|---|
| $3,700.00 approved by a sandbox buyer and captured, six items | 17:15 |
| "Service not received" claim filed on PayPal's site: four items, $2,650.00 | 18:33 |
| Contractor stand-in offers $1,200 through the app | 18:42 |
Escalated through the app; the case goes to PayPal (stage CHARGEBACK at 18:50) |
18:47 |
PayPal decides for the homeowner (adjudicate, sandbox) |
18:51 |
| $2,650.00 credited to the homeowner | 18:52 |
| Case resolved | 18:53 |
The finished case is open to anyone: https://makeitright.retailbox-automation.com/cases/de35db1b41bc/money-back
Earlier, on 3 Nov, through the app's own API:
| Scene | Claim | What happened | Money back |
|---|---|---|---|
| Contractor's offer, homeowner accepts on PayPal | $460 | offer $300 | $300 |
| Contractor accepts the claim | $400 | accept-claim |
$400 |
| Offer declined, homeowner escalates, PayPal decides | $1,350 | offer $500 declined, adjudicate |
$1,350, 12 min 9 s after filing |
In each 3 Nov scene the tracker found the homeowner's claim by itself; the app was never told the case id.
Challenges we ran into
- PayPal has no buyer API for claims. Our call to open a claim on the homeowner's behalf answered 400 ("No permissions to set target_client_id"), so the homeowner files on PayPal's site and the app finds the claim by polling. In the sandbox demo the escalation and PayPal's decision are pressed from the app, through seller-side sandbox calls.
- In the sandbox, claims of $3,150 went straight to chargeback and skipped the contractor's step; all claims from $400 to $2,690 took the normal path. The demo claim stays at $2,650.
- The contractor's balance has to cover the refund. When it cannot, PayPal answers
INSUFFICIENT_FUNDS, and the app reports that the contractor's balance is short. - Files attached to a claim by our test script did not reach it:
documentswas empty on all ten claims. - Counting finished units is the hardest part for the model (5 of 7 partly done items right on the development set).
Accomplishments that we're proud of
- The model only reads. Every amount is computed in code, so the claim adds up to the cent.
- The whole claim path, from payment to money back, ran on real sandbox claims, the last time on the hosted demo with the demo case itself.
What we learned
- A half-finished job sits between PayPal's claim reasons. "Service not received" gives 180 days to file; "Significantly not as described" excludes custom-made work and can have a 30-day window. The app files the unfinished items as "Service not received", and PayPal decides.
- After an accepted offer PayPal rewrites the claim amount, so the app keeps the original claim.
- Once the case goes to PayPal, the contractor's open offer drops out: PayPal refunded the full $2,650, and the $1,200 offer was never paid. PayPal's offer history records no cancel event, so the app infers it from the stage.
What's next for Make It Right
- A hosted model for live analysis of a homeowner's own files.
- Photos attached to each item of the claim when filing.
Built With
- gemma
- gemma-4
- hono
- node.js
- ollama
- paypal
- paypal-disputes-api
- paypal-orders-api
- pdf.js
- pglite
- typescript
- zeabur