Inspiration

I live in India, where people use PayPal mainly to get paid by buyers abroad. Craft makers here, such as the chikankari embroiderers of Lucknow, can find those buyers online. But a custom order needs a long conversation about size, colour, length, price and deposit, and the maker and the buyer often share no language. Translation tools handle single messages.

They don't turn a whole negotiation into terms both sides confirm, or into a payment that follows those terms. I wanted the conversation itself to become the deal, with PayPal holding the money from the deposit to the delivery.

The problem and who it's for

Makers who take custom orders, such as tailors, embroiderers, jewellers, woodworkers and leather workers, often find buyers abroad who speak another language. A custom order needs a lot of back-and-forth: size, colour, length, deadline, price and a deposit. Across a language gap, that work goes to a middleman, or the misunderstanding comes back later as a return or a dispute.

  • Buyers want their own language. In a 2020 CSA Research survey of online shoppers in 29 countries, 76% said they prefer to buy when product information is in their own language, and 40% said they never buy from websites in other languages. (CSA Research)

  • The maker side is large. Official Indian figures count 64.66 lakh handicraft artisans, about 6.5 million people. (PIB background note)

  • PayPal already fits the money side. Since April 2021, PayPal in India has focused on cross-border payments, so Indian sellers use it to get paid from abroad. (TechCrunch)

The demo pair is Ravi Chikan, a chikankari tailor in Lucknow who speaks Hindi, and Emily Carter, a buyer in Ohio who speaks English. The same room works between any two of seven languages: Hindi, English, Spanish, French, Brazilian Portuguese, German and Japanese. The interface, the invoice text, dates and units follow each side's language.

What it does

  • Two private links, one deal room. Each side talks in their own language: push-to-talk, a voice note, a sample clip or typing. The other side reads each message in their own language, in bold, with the original underneath, and hears it read aloud in a voice that matches the speaker.

  • Always clear whose move it is. An amber "Your turn" line names each side's one next step (confirm, approve the invoice, pay the $36.00 deposit, send a progress photo…). Colour marks who is who (maker green, buyer indigo) and what needs doing (amber), and the payment card shows paid, due and total as tiles, with the amount on the pay button.

  • An agreement that writes itself, with sources. Item, size, length (inches converted to cm), colour, price, deposit, delivery date and shipping fill in as they talk. Every term links back to the message it came from.

  • The AI asks when something is unclear. When Ravi quotes a price without saying whether shipping is included, the AI asks him, in Hindi, and his answer settles the term.

  • Both sides confirm the same terms. Each confirmation is tied to a SHA-256 fingerprint of the agreement. If a term changes afterwards, both must confirm again.

  • PayPal runs the money. Only after both confirm does the agent propose a PayPal invoice, and only the maker can approve it. One invoice takes a 30% deposit first: the pay link and a QR code appear in the room, and Ravi sees "deposit received" in Hindi. Ravi then sends a progress photo, Gemini checks it against the agreement, and once Emily approves it she pays the balance on the same invoice. Ravi says the tracking number out loud, approves the card it creates, and the tracking is added to the PayPal payments.

  • A production board gated by payments. The maker's Bryntum Gantt board shows each order's steps. "Deposit paid" and "balance paid" are gates: work after a gate stays locked until PayPal confirms that payment.

  • A money timeline. Every PayPal call, status change, payment and webhook is listed with its PayPal debug ID.

  • Judge mode. `/demo` opens a fresh deal with the maker (Hindi) and the buyer (English) side by side, with the model name, latency and IDs shown. Each room keeps its desktop layout, scaled to fit its half of the screen. Sample Hindi clips that I recorded and sample buyer messages (in the buyer's language) go through the same pipeline as live input, so no microphone is needed.

How PayPal is used

Everything runs against the PayPal sandbox (`api-m.sandbox.paypal.com`); the REST client refuses any other origin.

  • OAuth 2.0 client credentials (`POST /v1/oauth2/token`), with the token cached until 60 seconds before it expires.

  • Invoicing v2 with partial payment. `POST /v2/invoicing/generate-next-invoice-number`, then `POST /v2/invoicing/invoices` with `configuration.partial_payment` (`allow_partial_payment: true`, `minimum_amount_due` set to the deposit). One invoice covers both the deposit and the balance. Its note, terms, item and reference carry the bilingual agreement and its SHA-256 hash.

  • Send without email. `POST /v2/invoicing/invoices/{id}/send` with `send_to_recipient: false`; the payer link and a QR code appear in the room instead.

  • The invoice is the source of truth. `GET /v2/invoicing/invoices/{id}` every 10 seconds while money is due, and after every webhook. Status goes UNPAID → PARTIALLY_PAID (deposit) → PAID (balance). Each read also checks that the agreement text on the invoice still matches what both sides confirmed, and raises an alert if it does not.

  • Webhooks with signature verification. At boot the server finds or registers its webhook (`/v1/notifications/webhooks`) for `INVOICING.INVOICE.PAID`, `UPDATED` and `CANCELLED`. Each delivery is checked with `POST /v1/notifications/verify-webhook-signature`, counted once per event id, and only triggers a fresh read of the invoice. Polling covers late or missing events.

  • Add Tracking API. `POST /v1/shipping/trackers-batch` adds the tracking number to every PayPal payment of the invoice, then `GET /v1/shipping/trackers/{transaction_id}-{tracking_number}` reads each one back as SHIPPED.

  • PayPal Agent Toolkit. `send_invoice_reminder` from `@paypal/agent-toolkit`, run locally in sandbox mode, behind the maker's "Remind buyer" card.

  • Idempotency. Every write carries a `PayPal-Request-Id`. The invoice keys come from the agreement hash, so a retry cannot create a second invoice.

How AI is used

All model calls go to Gemini on Vertex AI (express mode) through the Google Gen AI SDK. AI output is advice: code owns money, dates, units, hashes and every PayPal guard.

  • Interpreting each turn. For a voice clip or a typed message, one `gemini-3.8-flash` call (thinking level LOW) returns the transcript, the translation, the agreement changes with the message they came from, clarifying questions, and action intents (add tracking, send a reminder). If Flash has not answered after 8 seconds, `gemini-4-argon` is asked too while Flash keeps running, and the first valid answer wins. Every answer must match a JSON schema and pass Ajv validation; prices are then handled in integer cents, and dates and unit conversions are computed in code.

  • Photos. The same models describe the buyer's reference photo to the maker in both languages, and check the maker's progress photo against the agreed item, colour, size and measurements. The check is advisory: only the buyer's approval unlocks the balance.

  • Voices. `gemini-3.8-flash-tts` reads each translation aloud in all seven languages, in the female voice Aoede for Emily's messages and the male voice Umbriel for Ravi's. If it fails or has not started within 10 seconds, the browser's own voice reads the text.

  • The AI never moves money. Intents such as "shipped with DHL, number 4567891234" only create approval cards. The maker approves, and code checks the carrier, the number and the invoice state before PayPal is called: AI proposes, code checks, the maker approves, PayPal executes.

  • Spend cap. Every model call, including fallbacks and voices, takes one call from a daily budget (500 calls).

How I built it

  • Stack: TypeScript throughout. Node 24 with Fastify 5 and Socket.IO; React 19, Vite and Tailwind; SQLite (`node:sqlite`); `@google/genai`; `@paypal/agent-toolkit`; Bryntum Gantt Community; hosted on an AWS Lightsail server behind Caddy, with a backup mirror on Render.

  • Shape: each deal is one document, changed only under a per-deal lock. After every change, each side gets a snapshot filtered for its role, in its language.

  • Trust and correctness: both sides confirm the same SHA-256 fingerprint; no invoice before both confirm and the maker approves; money in integer cents; PayPal's text limits checked in UTF-8 bytes (Hindi and Japanese take about three bytes per character, and a test checks a long agreement for all 42 ordered language pairs); idempotent PayPal writes; verified, deduplicated webhooks; per-IP rate limits and per-deal caps for the public demo; secrets only in environment variables.

  • Tests: 194 unit tests (Vitest) and Playwright end-to-end tests. The full-flow test takes a deal from the first message to a fully paid invoice against the real Gemini and PayPal sandbox APIs, locally and against the hosted app (it records the payments through the API instead of a checkout). A second full deal runs between a Japanese maker and a French buyer. I also paid real sandbox checkouts by hand during the build, both deposit-then-balance and paid-in-full, each followed by tracking.

  • Coding agent: I built it with Kiro as the coding agent (details in the AI tools field).

Challenges I ran into

  • PayPal reads invoice dates in US Pacific time. An invoice dated with the UTC date, which can be a day ahead of California, came back as SCHEDULED. Sauda now dates invoices and shipments in Pacific time.

  • Hindi and PayPal's text limits. The limits count bytes, and Devanagari takes about three bytes per character, so every invoice text is measured in UTF-8 bytes before anything is sent.

  • Gemini slow periods. At times Flash took 30 seconds or more while the fallback model returned quota errors. Sauda now keeps the slow Flash call running while it asks the fallback, and takes the first valid answer.

  • The sandbox checkout can stall on "Thank you for using PayPal". The room never depends on that redirect: it reads the invoice itself.

Built during the hackathon

All code was written for this hackathon between Fri Nov 6 and Thu Nov 12, 2026 (IST), and the public commit history shows each step. A quick test of the PayPal partial-payment invoice and of Gemini on Hindi audio ran on Nov 6, before the repository was created.

  • Fri Nov 6: first commit (20:48 IST); the foundation and the text deal room.

  • Sat Nov 7: push-to-talk voice, speech playback and photo understanding; the PayPal invoicing lifecycle, webhooks, tracking and the money timeline; the UI redesign; judge mode (`/demo`), rate limits, the AI budget, the mobile buyer layout and the Render Blueprint. (The "Initial commit" that day is GitHub's MIT license file.)

  • Sun Nov 8: my Hindi sample clips and the deployment on Render; the production board with payment gates, then the Bryntum Gantt version.

  • Mon Nov 9: UI polish (voice first, live agreement, payment moment, named progress steps) and the sample photos.

  • Tue Nov 10: the Gemini fallback fix for slow periods.

  • Wed Nov 11: natural Gemini voices matched to each speaker; the judge-facing README and screenshots.

  • Thu Nov 12: the hosted demo moved to an always-on AWS Lightsail server with persistent storage; Render stays as a backup mirror. The language and colour refresh: French, Brazilian Portuguese, German and Japanese, the reader's own language first in each message, the "Your turn" line, colour-coded roles and states, and shorter text.

Limitations

  • Sandbox only. One US sandbox business account invoices in USD. Real Indian makers would rely on PayPal India's cross-border receiving, which I did not test.

  • One small server. The demo runs on a single always-on AWS Lightsail instance in the eastern US, without a CDN, so far from the US the first load and the production board take a few seconds longer. `/demo` creates a fresh deal on every visit.

  • Gemini capacity. When Vertex AI is busy, Flash can take longer than 30 seconds and the fallback can return quota errors. A turn with no valid answer after 38 seconds shows a Retry button.

  • One maker account, no production login. Rooms are protected by a secret token in each link, which is fine for a demo but not for real use.

  • Turn by turn, not streaming. Each turn is one clip and one model call, about 5 to 8 seconds.

  • Machine-drafted translations. The interface text beyond Hindi, English and Spanish was drafted with Gemini and checked by back-translation, a model judge and a spot check, not by native speakers.

  • Hindi-only sample clips. The recorded sample clips are in Hindi. In the other languages, speak or type.

  • Sandbox quirks. Reminders through the Agent Toolkit take about a minute in the sandbox and return no debug ID. Tracking needs a real PayPal checkout payment.

  • No disputes or refunds yet.

What's next

  • Onboarding many makers through PayPal's partner onboarding, each paid into their own account.

  • Using the confirmed, hashed agreement on the invoice as evidence if a dispute is opened.

  • Streaming speech-to-speech interpretation for live calls.

  • More languages, and native-speaker review of the interface text

Built With

Share this project:

Updates

Submission history