PayFlow

An autonomous expense-settlement agent for small teams.

Comment 2

Inspiration

PayFlow is an AI agent that turns a single receipt photo into a completed batch payout, leaving the human exactly one button: final approval.

We built it because every small team that shares a company card runs the same unpaid job at the end of each month. The corporate card limit is too low, so someone pays out of pocket and files a claim by hand. Receipts land in Slack, invoices in email, and the payment ledger lives in PayPal. If ten people are owed money, someone sends ten transfers. When last month's invoice is submitted twice, a human has to catch it by eye. And when a contractor files ten items but only eight are paid, there is no way to find out which two dropped and why.

We are a three-person team, and we had lived this. The work is not hard. It is reconciliation, and reconciliation is exactly what software should do. The judgment in the middle had been missing: reading a crumpled receipt photo, deciding whether a charge is business or personal, noticing an invoice that looks like one from last month, writing the message that asks a colleague for a correction.

We started from an uncomfortable constraint: this agent moves real money. That fact shaped every architectural decision below.

What it does

  1. A claimant drops a receipt photo into Slack. The backend verifies the Slack signature, stores the raw image, and acknowledges within 3 seconds; parsing happens asynchronously.
  2. Gemini parses the image into structured JSON and maps it to an accounting category (AWS charge → service fee, team dinner → welfare expense).
  3. The claimant agent reviews the parse, classifies business vs. personal, and drafts the re-request message when something does not add up.
  4. An executor selects a settlement period on the web dashboard. The backend pulls the PayPal ledger and matches it against claims deterministically (amount, date window, merchant name).
  5. The executor agent explains the match failures and describes anomalies: duplicate claims, receipt/claim amount mismatches.
  6. A follow-up loop DMs the claimant about unclaimed charges, reminds once if there is no answer, and expires the item if it never arrives.
  7. The safety agent writes a risk report right before approval.
  8. The human clicks approve. Only then does a batch PayPal Payout go out, followed by reconciliation, cancel-and-resend for UNCLAIMED items, and an XLSX export for the accountant.

How we built it

Stack. Next.js/TypeScript frontend, FastAPI/Python backend, and a Google ADK agent service (three Cloud Run services in one GCP project). Gemini 3.7 Flash through Vertex AI, Firestore (Seoul) as the state store, GCS for raw receipt images, Cloud Tasks for every asynchronous hop, Secret Manager for credentials, all provisioned with Terraform and deployed by GitHub Actions over Workload Identity Federation. External integrations: Slack (Events API, interactivity, signature verification) and the PayPal Payouts API in sandbox.

Three design rules we never broke.

  1. The agent never touches money. IAM enforces this, not team discipline. The agent service account has no access to PayPal credentials, and you can read that fact off the Terraform. The agent's job is to write a settlement proposal, not to send money.
  2. The payout endpoint does not execute without an approval token. The token is bound to run_id plus a hash of the amount, expires in 10 minutes, and is burned on use. It never enters an LLM context.
  3. The LLM does not compute amounts. The LLM narrates its reasoning; code produces the numbers. Integer minor units and an explicit currency, never float.

Where we drew the line between code and the LLM.

Owner Responsibilities
Deterministic code Matching, summation, per-person allocation, cap checks, token verification, state transitions
Agents (LLM) Reviewing parses, classifying, judging, writing prose

We kept ledger↔receipt matching out of the LLM by design: it is the one place where a mistake collapses the entire demo.

The safety agent advises. It does not gate approvals. Caps, tokens, CAS transitions, and duplicate suppression are all enforced in code. If an LLM were the gate, a prompt-injected receipt could talk its way through approval.

One-way call graph. web → api → agent, with no synchronous path back. The agents never call each other; the backend invokes each one through Cloud Tasks at its pipeline stage, and results come back only as a Firestore draft. Even Slack messages are sent by the backend. One choke point, one place to inspect. That is what keeps an injected receipt from exfiltrating itself through a Slack DM.

Waiting without keeping a session alive. The follow-up loop is a Firestore state machine (PENDING → REMINDED → RESPONDED | EXPIRED) driven by Cloud Tasks schedule_time. A scheduled task wakes up, reads current state, and branches, so duplicate delivery is harmless and no agent session sleeps for a day. Demo and production differ by exactly one environment variable (REMINDER_DELAY_SECONDS=20 vs 86400); we built no fake clock.

How the three of us worked in parallel. Three repos, three tracks (claimant experience, executor experience, money and safety), with a written schema contract as the single source of truth. Pydantic models in the backend are canonical; the web generates TypeScript types from OpenAPI, and the agent pins the backend as a git dependency at a tag. Cross-repo changes always flow api → agent → web, never backward. Direct pushes to main were allowed for speed, with one exception: anything touching the payout path, token issuance, or CAS state transitions requires a branch and a second pair of eyes. We gated only the code that spends money.

Challenges we ran into

Challenge Root cause Fix
Dedup bypassed by concurrent Slack retries Read-then-write query left a race window Firestore doc ID is the dedup key, so the database enforces uniqueness
Reminder loop that never fired Two timeout env vars shared the same 86400 default; four absorbing states returned 200 without scheduling anything Derived the reminder default from the TTL relationship; attached an expiry task to all paths
1000× rounding error in currency exponents Parser assumed every currency uses two decimals Handled per-currency exponents; also fixed NaN/Infinity and a bracket-indexed nullable field
Rollback refunded the wrong amount Refund used the run's total instead of the failing recipient's share Refund now uses the failing recipient's share
Prompt injection via receipt text Raw receipt text could enter the agent's request body, get persisted in Cloud Tasks, and ride into the audit log Pass a GCS URI instead of raw text; scope agent read access to one prefix
Contract mismatch: claimant agent output shape Tool emitted {classification, requery_message}; contract required needs_requery: bool Fixed the field to match the contract
Contract mismatch: missing agent env vars Five missing env vars silently routed session memory to the wrong Firestore database and ADK to the wrong API Restored the required env vars
Container fine locally, broken in Cloud Run Dockerfile dependency drift plus missing fixtures sent the parser into an infinite retry loop Added _UnavailableParser, which raises a retry signal instead of returning fabricated data

A dedup that only worked when nobody raced it. Our first Slack ingest dedup was a Firestore query: read, then write if absent. Concurrent Slack retries walked straight through it. We replaced it with a receipt_dedup_keys/{slack_file_id} document whose ID is the dedup key, making the collision the database's problem instead of ours.

A reminder loop that never fired. Per-task review passed it. Whole-branch review found two things per-task review structurally could not see. First, REMINDER_DELAY_SECONDS and CLAIM_REQUEST_TTL_SECONDS both defaulted to 86400, so with no env override the reminder was scheduled exactly at expiry and fired zero times, deterministically. The e2e test never caught it, because the fixture used a TTL twice the code's default and never exercised the relationship the code creates. Second, four absorbing states (no_message, no_target, a permanent Slack failure, and a reschedule failure) returned 200 without scheduling anything, stranding the request in PENDING forever. The claim would sit in DRAFT, excluded from every future settlement, with no error raised.

We derived the default from the TTL relationship and attached an expiry task to all three paths. Reviewing a diff task by task cannot see a bug that lives in the relationship between two tasks.

A 1000× rounding error hiding in currency exponents. Receipt parsing treated every currency as two-decimal. For zero-exponent currencies (KRW, JPY) that is a 1000× error on real money. Review found it, along with NaN/Infinity escaping the parser and a nullable field indexed with brackets that pinned receipts in RECEIVED forever.

A rollback that refunded the wrong number. When we lifted the single-recipient restriction on payouts, we found that failure rollback used the run's total amount instead of the failing recipient's share. Under the single-recipient assumption the two values were identical, so the bug had been invisible from the day it was written.

Prompt injection is a data-flow problem, not a prompt problem. Receipt text is wrapped in an <untrusted_receipt_text> block, and side-effecting tools pass through a before_tool_callback. The real fix was refusing to put raw receipt text into the agent's request body at all. That text would have persisted in the Cloud Tasks queue and could have ridden the draft's reason field into the audit log. We send a GCS URI instead, and gave the agent service account read access to exactly one prefix. Masking would not have saved us: the masking rules cover four PII types, and business registration and phone numbers are not among them.

Two silent contract mismatches that would have looked like "the AI isn't very good."

Contract mismatch 1: the claimant agent's output shape. The scaffolded tool emitted {classification, requery_message} while the contract required a needs_requery: bool. The backend rejected every draft without raising an error. Our contract test only compared enums, so it passed.

Contract mismatch 2: missing agent environment variables. Five environment variables missing on the agent service would have sent session memory to the wrong Firestore database and routed ADK to the Developer API instead of Vertex, both failing without a visible error. Wiring an LLM in on top of either mismatch would have produced a system that looked stupid rather than broken.

A container that booted fine locally and not at all in Cloud Run. Dockerfile dependency drift, plus fixtures missing from the image, sent the parser into an infinite retry loop. We added an _UnavailableParser that raises a retry signal rather than returning fabricated data. When the parser cannot run, retrying beats writing a wrong number into a settlement.

Accomplishments that we're proud of

PayFlow turns a receipt photo into money in someone's account, end to end. Slack upload → Gemini parse → claimant agent review → deterministic matching → executor agent analysis → follow-up DM with a button → human approval → PayPal batch payout → reconciliation → XLSX for the accountant. The whole pipeline runs, not a slide of it.

Retrying a payout does not send money twice. We verified this against the PayPal sandbox on day one rather than assuming it: sender_batch_id = settlement_run_id, and a resend of the same batch is rejected. Amounts are integer minor units with an explicit currency throughout; no float appears anywhere near money. The FX rate is frozen at approval time, because the approval token is bound to a hash of the amount.

Waiting is a state machine, not a sleeping agent. PENDING → REMINDED → RESPONDED | EXPIRED in Firestore, driven by Cloud Tasks. Duplicate delivery is harmless by construction, and the same code runs the demo at 20 seconds and production at 24 hours through one environment variable. No fake clock anywhere in the system.

Roughly 900 tests, gating every deploy. ~850 in the backend and ~50 in the agent service, including money-safety units for idempotency, approval tokens and caps, a pipeline test that runs the agent path with no LLM call at all, and a schema-contract test across repos. CI blocks the Cloud Run deploy if any of them fail. We also used mutation testing on the parsing acceptance gate and found a real verification gap it had been missing.

What we learned

Narrow the agent's job and accuracy goes up. Our first instinct was to let the agent do everything. The system we shipped narrows the agent's job: it reviews, classifies, judges, and explains, while code matches, sums, allocates, and gates. Every step we moved out of the LLM made the system more reliable without making it less autonomous.

A guardrail only visible in code has to be trusted, not verified. We put the money boundary in IAM, not in an if statement. That means we can show a reviewer the Terraform and the 403 response, instead of asking them to take our word for it.

Durable state beats a long-lived session. Anything that waits (a reminder, an expiry, a retry) is a Firestore state machine plus a scheduled task, not a suspended agent. That makes every wait idempotent by construction, and it turns "compress a day into 20 seconds" into a one-variable change.

Review at two altitudes. The costliest bugs we found lived between tasks, not inside them, and only the whole-branch pass caught them. We now run both.

Contracts between repos need executable tests, not just a document. Every polyrepo bug we hit was a place where the document was right, but no test checked that the code still matched it.

What's next for PayFlow

Widen the ingest surface. We scoped Google Drive and email collection, business registration number extraction, VAT-deductibility judgment, contract unit-price comparison, and timesheet analysis out as Won't-Have for the hackathon. Each is a straightforward extension of the parsing path now that the pipeline exists.

Move from PayPal sandbox to production, carefully. Idempotency, caps, the approval gate and reconciliation are all in place, but a real-money rollout needs per-organization caps, a dry-run mode, and an operator-facing view of every audit log entry. The cancel-and-resend path also only covers UNCLAIMED items. Money a recipient has already accepted cannot be recovered through the API, and the product should say so instead of implying an undo that does not exist.

Built With

Share this project:

Updates

Submission history