π€ Inspiration
Moving money still means clicking through forms: open a tab, log in, fill an invoice, find a card, confirm. Voice assistants can talk but can't be trusted to pay β one stray script or keystroke and your money is gone.
We wanted the opposite: run a real PayPal business entirely by voice, where nothing moves until a human physically approves it. Our base app, keyboard.wtf, already gave us a voice-first desktop assistant ("Jarvis"). We turned it into Keyflow β a hands-free PayPal command center.
β‘ What it does
You speak; Jarvis picks the right tool and shows the result on a live canvas:
| You say | Keyflow does | Powered by |
|---|---|---|
| "Invoice Demo User 1 for 250 dollars for the website" | Creates + sends a PayPal invoice, renders a pay-by-QR code | PayPal Invoicing |
| "Pay 10 dollars for coffee" | Prepares an order you approve with a physical click, captures it | PayPal Orders + Payments |
| "Refund that payment" | Issues a PayPal refund on the capture | PayPal Payments |
| "Find the best wireless keyboard" | Live product gallery with prices, ratings, price history | Channel3 |
| "Show my payments" | Interactive donut + bar + grid you drill into by voice | AG Charts + AG Grid |
| "What was I looking at earlier?" | Recalls it across sessions | Elastic hybrid memory |
π οΈ How we built it
- C# / .NET 8 WinForms desktop app; Gemini Live is the voice brain that selects from 18 commerce tools.
- A CommerceHost routes each tool to
PayPalClient(Orders, Payments, Invoicing, Subscriptions, Catalog, Disputes),Channel3Client, and an Elastic hybrid-memory store. - Results render in a WebView2 canvas from validated structured specs with vendored AG Charts / AG Grid β never model-generated HTML.
- Secrets live only in an ACL-locked, git-ignored
.env; they are never logged, serialized, or sent to the model.
Our Elastic recall fuses lexical and semantic ranks with Reciprocal Rank Fusion:
$$ \text{RRF}(d) = \sum_{r \in R} \frac{1}{k + \text{rank}_r(d)} $$
β¦and every query is scoped by an opaque per-user owner hash, so memory is private by construction.
π The safety model (the part we're proudest of)
PREPARED β AWAITING_APPROVAL β APPROVED β PAYPAL_APPROVAL β CAPTURED
The approval dock accepts only physical input β Windows' GetCurrentInputMessageSource must report a hardware mouse/touch/pen. Injected SendInput clicks, posted messages and typed "yes" are all refused. Voice approval is opt-in, needs your recognized speech plus a read-back of the exact amount and payee, and still never bypasses PayPal's own approval page.
π What we learned
- PayPal's Orders β capture lifecycle is the only honest signal that money actually moved (an approval isn't a payment).
- Real integration means reading the error body: Channel3's
extra_forbidden, PayPal'sINR_FOREIGN_CURRENCY_BLOCKED, idempotency viaPayPal-Request-Id. - Hybrid (lexical + semantic) retrieval beats either alone for fuzzy recall like "that keyboard I looked at."
π§ Challenges we faced
India-account currency block. Both seeded sandbox accounts are India-registered, and PayPal blocks domestic India payments in every currency. We diagnosed it precisely and solved it with guest-card checkout (
PAYPAL_CHECKOUT_LANDING_PAGE=GUEST_CHECKOUT), verifying order β capture β refund end-to-end.Channel3
422 extra_forbidden. Locale and agenticmodemust nest insideconfig, not the request top level. Fixed the schema, added a one-shot retry and an honest public-catalog fallback.Proving the click is human. Building an approval the demo's own automation cannot fake took a native input-origin check plus a minimum-exposure and double-submit guard.
Invoice idempotency & cleanup. A refused send now deletes its unsent draft and explains why, so the account never fills with junk.
Built With
- ag-charts
- ag-grid
- channel3
- csharp
- dotnet
- elasticsearch
- gemini
- invoicing
- javascript
- orders-v2
- paypal
- playwright
- powershell
- rest-api
- subscriptions
- webview2
- winforms