🎀 Inspiration

Moving money still means clicking through forms: open a tab, log in, fill an invoice, find a card, confirm. Voice assistants can talk but can't be trusted to pay β€” one stray script or keystroke and your money is gone.

We wanted the opposite: run a real PayPal business entirely by voice, where nothing moves until a human physically approves it. Our base app, keyboard.wtf, already gave us a voice-first desktop assistant ("Jarvis"). We turned it into Keyflow β€” a hands-free PayPal command center.

⚑ What it does

You speak; Jarvis picks the right tool and shows the result on a live canvas:

You say Keyflow does Powered by
"Invoice Demo User 1 for 250 dollars for the website" Creates + sends a PayPal invoice, renders a pay-by-QR code PayPal Invoicing
"Pay 10 dollars for coffee" Prepares an order you approve with a physical click, captures it PayPal Orders + Payments
"Refund that payment" Issues a PayPal refund on the capture PayPal Payments
"Find the best wireless keyboard" Live product gallery with prices, ratings, price history Channel3
"Show my payments" Interactive donut + bar + grid you drill into by voice AG Charts + AG Grid
"What was I looking at earlier?" Recalls it across sessions Elastic hybrid memory

πŸ› οΈ How we built it

  • C# / .NET 8 WinForms desktop app; Gemini Live is the voice brain that selects from 18 commerce tools.
  • A CommerceHost routes each tool to PayPalClient (Orders, Payments, Invoicing, Subscriptions, Catalog, Disputes), Channel3Client, and an Elastic hybrid-memory store.
  • Results render in a WebView2 canvas from validated structured specs with vendored AG Charts / AG Grid β€” never model-generated HTML.
  • Secrets live only in an ACL-locked, git-ignored .env; they are never logged, serialized, or sent to the model.

Our Elastic recall fuses lexical and semantic ranks with Reciprocal Rank Fusion:

$$ \text{RRF}(d) = \sum_{r \in R} \frac{1}{k + \text{rank}_r(d)} $$

…and every query is scoped by an opaque per-user owner hash, so memory is private by construction.

πŸ”’ The safety model (the part we're proudest of)

PREPARED β†’ AWAITING_APPROVAL β†’ APPROVED β†’ PAYPAL_APPROVAL β†’ CAPTURED

The approval dock accepts only physical input β€” Windows' GetCurrentInputMessageSource must report a hardware mouse/touch/pen. Injected SendInput clicks, posted messages and typed "yes" are all refused. Voice approval is opt-in, needs your recognized speech plus a read-back of the exact amount and payee, and still never bypasses PayPal's own approval page.

πŸ“š What we learned

  • PayPal's Orders β†’ capture lifecycle is the only honest signal that money actually moved (an approval isn't a payment).
  • Real integration means reading the error body: Channel3's extra_forbidden, PayPal's INR_FOREIGN_CURRENCY_BLOCKED, idempotency via PayPal-Request-Id.
  • Hybrid (lexical + semantic) retrieval beats either alone for fuzzy recall like "that keyboard I looked at."

πŸ§— Challenges we faced

  1. India-account currency block. Both seeded sandbox accounts are India-registered, and PayPal blocks domestic India payments in every currency. We diagnosed it precisely and solved it with guest-card checkout (PAYPAL_CHECKOUT_LANDING_PAGE=GUEST_CHECKOUT), verifying order β†’ capture β†’ refund end-to-end.

  2. Channel3 422 extra_forbidden. Locale and agentic mode must nest inside config, not the request top level. Fixed the schema, added a one-shot retry and an honest public-catalog fallback.

  3. Proving the click is human. Building an approval the demo's own automation cannot fake took a native input-origin check plus a minimum-exposure and double-submit guard.

  4. Invoice idempotency & cleanup. A refused send now deletes its unsent draft and explains why, so the account never fills with junk.

Built With

Share this project:

Updates

Submission history