ShortPay

CFO teams pay extra freight fees before they check the bill. Shortpay lets you approve the right amount, and dispute the rest before cash leaves and keep the cash.

Comment

Shortpay: Project Write-up

Pre-Pay Freight Surcharge Audit for the Autonomous Office of the CFO Track 2: Syndicate by Maximor Hackathon Beautiful Preview at : https://gist.github.com/DevloperHS/dbec4a1e1b12c0958b67013b8fcee8a4

1. Why we built this project

Freight audit and payment already have clear owners in the CFO's office. Logistics books the load, AP pays the carrier, and finance owns tolerance and coding. The leak happens before payment. Most AP teams sample some carrier invoices, or they outsource recovery to contingency freight-audit (FAP) firms. These firms take a cut after the money has left the business. FAP vendors claim that 3–7% of freight spend leaks through accessorial errors. Examples include a liftgate charge at a dock door, detention billed past free time, the wrong DOE fuel week, and duplicate invoices. Shortpay does not rely on that estimate. It proves one invoice.

We built Shortpay as an AP desk that audits carrier accessorials before payment:

  1. A logistics controller uploads the rate baseline (contract terms CSV).
  2. Carrier invoices (PDF/text), IoT dock timestamps, and facility facts land beside them.
  3. Each LTL/TL invoice is matched to its shipment. An authorized payable is computed in integer cents by a pure mathematical engine. No LLM runs on the money path.
  4. The case either auto-closes under policy, or one card with one intuitive math grid is put in front of a human controller who approves a short-pay packet: pay $925.00, dispute $195.00, attach the dock receipt.

The framing was deliberate. Maximor's product story is a loop: learn a short-pay policy, auto-close $38, escalate $495, tighten the policy. Shortpay applies that loop to a real AP document instead of building a general "invoice copilot." The ERP remains the system of record. Shortpay proposes the payable and the dispute notice, but it does not become the GL.

The hero case shows the full thesis. FedEx Freight SHP-88220 was billed $1,120.00. The authorized amount is $925.00, and the dispute is $195.00. The liftgate is not authorized because the destination has a dock. Detention is recomputed from 93 minutes of dock dwell against 30 minutes of contract free time. That gives 1 completed hour × $75. One unit test checks those exact cents.

2. Pain points it solves

Pain point today How Shortpay solves it
Sampling instead of auditing. Auditing takes minutes per line and hours per invoice, so teams sample. The unsampled invoices can still contain errors. Every invoice is matched and computed automatically. Exceptions land on a Kanban board, and clean invoices close themselves.
Swivel-chair reconciliation. The clerk opens the bill, searches the TMS for the BOL/PRO, and opens the rate con, POD, and dock log for every line. All four evidence sources (contract baseline, facility master, IoT dock log, invoice) are ingested and joined per shipment. The variance is computed for them.
Error-prone re-math. Detention must be recomputed from timestamps, and liftgate depends on whether the destination facility has a dock. People can get this wrong, especially at high volume. match_evidence() is a pure function: billable = max(0, dwell − free), hours = billable // 60, and liftgate is authorized at $0.00 when a dock exists. The same inputs always produce the same cents.
Accessorial leakage. Liftgate, detention, residential, and reweigh lines are exactly what a sample audit can miss. v1 proves base freight against the contract rate, liftgate against facility facts, and detention against dock dwell. These are the lines that a dock clock and facility master can prove.
Untrustworthy AI output. An accountant cannot approve a result by simply accepting an agent's answer. The controller approves a math grid. The LLM only extracts billed facts. The prompt and the architecture prevent it from computing a payable.
Post-payment recovery is slow and adversarial. Contingency FAP firms chase refunds after payment. Short-pay happens pre-payment: pay the authorized amount, dispute the variance with evidence attached.
Opaque automation. Nobody can answer "where did $75 come from?" after the fact. Every extraction, match, and decision emits Neatlogs spans with a stable trace id (trace-freight-shp-88220) and a SHA-256 evidence-pack hash stored on the case.

3. Challenges faced while building

3.1 Keeping the LLM off the money path

The biggest architectural challenge was keeping the model from doing the audit. An LLM that emits payable cents cannot be locked to $925 and cannot support an audit. The solution is simple. The LLM acts as an extraction adapter only. It maps invoice text to a validated InvoiceFact (Pydantic AI PromptedOutput, temperature 0, max_tokens 768, retry 2). Its instructions say, "Do not calculate an authorized payable, dispute amount, or policy decision." All money math lives in match_evidence(), a pure function with zero I/O and zero model calls. If extraction fails, such as when "gate fee" cannot be mapped, the system creates an UnexplainedLine. That blocks auto-close instead of silently paying $0.

3.2 Integer-cents discipline

Floating-point money is a silent killer (0.1 + 0.2 ≠ 0.3). The rule is absolute: no floats on the domain path. Money and Minutes are frozen dataclasses whose __post_init__ rejects bools and floats outright, and Pydantic field validators re-assert integrality at the evidence edge. The hero lock test (92500 / 19500) was written before any UI existed, per the PRD's "must" list.

3.3 Two-tier LLM resilience and sponsor guardrails

Sponsor APIs can fail, rate-limit, or lack keys. The extraction adapter uses a strict fallback chain. TensorMux (glm-4-7-flash) is primary, and Groq (qwen/qwen3.8-27b) is secondary. The adapter raises an explicit ExtractionError with the failure reason from each provider only when both providers fail. The hackathon also required a sponsor guardrail. Each provider allows 60 outbound HTTP attempts per rolling 60 seconds. A thread-safe ProviderRequestLimiter connects to the httpx transport layer, so SDK retries and concurrent requests count too. Attempt 61 is blocked before transmission.

3.4 Human-in-the-loop that fails closed

A controller could click "approve short-pay" on a stale screen and pay the wrong amount. To prevent this, ApproveShortPay must echo the payable shown on screen. AuditOffice.decide raises StaleDecisionError when that amount differs from match_result.expected_total_cents. The HITL amount is deliberately "ceremonial." It makes the stale-screen race fail closed without adding a version field. Dispositions are a sum type (NeedsReview | AutoClosed | ShortPaid | PaidAsBilled | SkippedOutOfScope), so the system cannot represent illegal AP states such as short-paid and paid-as-billed.

3.5 Idempotency and convergence

Re-ingesting the same invoice must not create a duplicate card. Fact identity is keyed on (invoice_id, shipment_id) per fact type; same-values re-ingest is a no-op; changed values on a non-terminal case replace the fact and rematch; terminal cases never reopen in v1 (new invoice id → new case); match is a pure function of current facts, so re-runs converge.

3.6 Architecture selection under deadline pressure

The team used an "arena" process. We sketched two structurally different candidates, A (AuditCase aggregate plus disposition machine) and B (dual money ledger as the root). We judged them against a rubric and then combined the strongest parts. We selected A as the root, with B's ledgers inside the case. B's outer join on ChargeType is the right model for money, but the specialist's work item is a case with a column and a button. If we had shipped B as the root, it would still have needed a case object, which would have created two work items. The rejected options are also documented: hunter→extractor→matcher as pipeline modules (temporal decomposition), LLM-emitted cents, mutable invoices, HITL amount entry, and IEEE-ceil detention.

3.7 Frontend pivot mid-build

ADR 0001 planned a Next.js frontend. What shipped is a Vite React SPA served by a Flask BFF (ADR 0004, which supersedes the Next.js ADRs). Flask owns cookies, static files, and /api/ui/* proxies. React never calls FastAPI directly. This removes CORS from the demo path and gives the BFF a natural place to prevent the human from posting a typed payable. The Flask layer validates that expected_payable_cents is an integer and forwards it unchanged. The matcher still rejects a stale number. The cost is documented clearly: a clone must run npm run build before Flask can serve a working UI.

3.8 Domain edge cases

Overnight dock logs (departure before arrival) are explicitly rejected in v1 rather than guessed; ocean/parcel modes surface as human-readable SkippedOutOfScope cards instead of running wrong math; PDF ingestion is bounded (≤ 15 MB, minimum extractable text) and failed PDFs raise PdfParseError instead of limping into the LLM.

4. Technicalities & architecture

4.1 Stack at a glance

Layer Technology Port
Domain engine Python 3.11+, Pydantic v2, pure-function matcher N/A
Domain HTTP API FastAPI + Uvicorn, CORS enabled, auto-ingests fixtures on startup 8000
BFF / UI server Flask (python -m frontend), owns static files + /api/ui/* proxy 5000
Specialist UI Vite + React SPA (built to frontend/static/react/), lucide-react icons N/A
Extraction (primary) Pydantic AI Agent + PromptedOutput(InvoiceFact) over TensorMux (glm-4-7-flash) N/A
Extraction (fallback) Groq OpenAI-compatible endpoint (qwen/qwen3.8-27b) N/A
Observability Neatlogs spans: extract_billed, match_evidence, decide; SHA-256 evidence hash N/A
Tests Pytest (37 passing; live-sponsor suite opt-in) N/A

4.2 System context

flowchart LR
    subgraph Sources["Evidence sources"]
        CSV["Rate baseline CSV<br/>(controller upload)"]
        PDF["Carrier invoice<br/>(PDF / text)"]
        DOCK["IoT dock log<br/>(arrival/departure)"]
        FAC["Facility master<br/>(has_dock)"]
    end

    subgraph Frontend["Presentation (port 5000)"]
        UI["Vite React SPA<br/>Kanban board + Math Grid modal"]
        BFF["Flask BFF<br/>/api/ui/* proxy, cookies,<br/>static files, PDF upload"]
    end

    subgraph Backend["Domain (port 8000)"]
        API["FastAPI<br/>/api/ingest /api/cases<br/>/api/cases/{id} /api/decide"]
        OFFICE["AuditOffice<br/>ingest · match · decide"]
    end

    LLM["TensorMux (primary)<br/>Groq (fallback)<br/>Pydantic AI extraction"]
    NL["Neatlogs<br/>observability"]
    ERP["ERP proposal +<br/>dispute packet (mock)"]

    CSV --> API
    PDF --> BFF --> API
    DOCK --> API
    FAC --> API
    UI --> BFF
    API --> OFFICE
    OFFICE --> LLM
    OFFICE --> NL
    OFFICE --> ERP
    BFF --> UI

4.3 Domain core: knowledge ownership, not pipeline stages

The shortpay/ package is deliberately not a hunter→extractor→matcher pipeline. It is a map of who owns which knowledge:

flowchart TD
    UI["harvest / UI / demo"] --> OFFICE["AuditOffice<br/>ingest · match · decide"]
    OFFICE --> STORE["CaseStore (in-memory)<br/>facts + case snapshots"]
    OFFICE --> CASE["AuditCase<br/>two ledgers + disposition"]
    OFFICE --> MATCH["match_evidence<br/>(pure function, no I/O)"]
    STORE --> CASE
    ADAPTERS["adapters/<br/>baseline_csv · invoice_extract<br/>dock_log · facility_master<br/>erp_propose · vault_hunter"] --> STORE
    POLICY["PolicyBook<br/>SHORT-PAY-01<br/>$50 auto-close threshold<br/>per-lane approved rules"] --> OFFICE

Module map:

shortpay/
  money.py          Money, Minutes            (frozen, rejects bool/float)
  evidence.py       InvoiceFact, ContractFact, FacilityFact, DockDwellFact
                    + constructors that validate at the edge
  ledgers.py        BilledLine, ExpectedLine, LedgerJoin
  matching.py       match_evidence (pure), completed_detention_hours
  policy.py         LaneKey, PolicyBook (SHORT-PAY-01)
  case.py           AuditCase, Disposition sum type, decisions
  store.py          InMemoryStore (fact + case storage; the seam for a future
                    Postgres adapter, all persistence flows through this one object)
  office.py         AuditOffice, the only public verbs: ingest / match / decide
  neatlogs.py       failure-isolated tracer facade + evidence hashing
  adapters/         vendor I/O; never exported from __init__

The money model. Inside a case, truth is two frozen ledgers plus a join. The carrier invoice is immutable (BilledLedger), expected money is computed separately (ExpectedLedger), and variance is an outer join on ChargeType. Payable = sum(expected). Dispute lines are billed lines where billed > expected. Liftgate is an explicit expected $0 when a dock is present, so the math grid is a total function. Every line gets an explanation.

Disposition is a sum type, so illegal AP states (short-paid and paid-as-billed) are unrepresentable:

stateDiagram-v2
    [*] --> NeedsReview: overbill above $50<br/>or unapproved lane rules
    [*] --> AutoClosed: dispute ≤ $50 AND all<br/>fired rules pre-approved on lane
    [*] --> SkippedOutOfScope: mode not LTL/TL
    NeedsReview --> ShortPaid: ApproveShortPay<br/>(amount must equal expected, else fail closed)
    NeedsReview --> PaidAsBilled: OverridePayAsBilled<br/>(reason required)
    ShortPaid --> [*]: ERP payable proposed<br/>+ dispute packet attached
    PaidAsBilled --> [*]: override reason stored
    AutoClosed --> [*]: auto ERP posting
    SkippedOutOfScope --> [*]: human-readable skip reason

The autonomy loop. Every human ApproveShortPay records the fired rule ids, such as LIFTGATE_DOCK_PRESENT and DETENTION_HOURS, on the lane's PolicyBook. The next small overbill on the same lane, based on carrier, mode, and dock configuration, auto-closes under SHORT-PAY-01 when the dispute is ≤ $50, there are no unexplained lines, and all fired rules are pre-approved. The product learns from human approvals. That is Maximor's loop made real.

4.4 Full request lifecycle (hero case)

sequenceDiagram
    autonumber
    participant C as Controller (React UI)
    participant F as Flask BFF (5000)
    participant A as FastAPI (8000)
    participant O as AuditOffice
    participant L as TensorMux→Groq
    participant N as Neatlogs
    participant E as ERP / Dispute (mock)

    A->>A: startup, auto-ingest fixtures<br/>(baseline CSV, dock, facility, invoice JSON)
    C->>F: paste invoice text / upload PDF
    F->>A: POST /api/ingest/invoice
    A->>L: extract billed facts (Pydantic AI,<br/>60 req/60s guardrail per provider)
    L-->>A: validated InvoiceFact (integer cents)
    A->>O: ingest(invoice_fact)
    A->>O: match(CaseKey)
    Note over O: match_evidence():<br/>dwell 93 − 30 = 63 min → 1 hr × $75<br/>liftgate $0 (dock present)<br/>expected 92500 · dispute 19500
    O->>N: span match_evidence + SHA-256 evidence hash
    A-->>F: case payload (disposition NeedsReview)
    F-->>C: render Math Grid modal
    C->>F: POST /api/ui/cases/{id}/decide<br/>(ApproveShortPay, 92500, echoed from screen)
    F->>A: POST /api/decide
    A->>O: decide(key, ApproveShortPay)
    Note over O: stale amount → StaleDecisionError;<br/>else record rules on lane PolicyBook,<br/>disposition → ShortPaid(92500)
    O->>N: span decide
    O->>E: PayablePosting(92500) + dispute packet
    A-->>F: ShortPaid + ERP proposal + dispute packet
    F-->>C: card moves to Short-paid column

4.5 Boundaries, idempotency, observability

  • Parse at the boundary. Every wire format (CSV rows, PDF JSON, IoT payloads, TMS tables, NetSuite vendor-bill objects) is converted into domain facts by an adapter and never leaks past it. The architecture test is simple. If TensorMux and the hunter adapter disappear, AuditOffice.ingest still works with hand-built facts. The agents are not the architecture.
  • Idempotency. One invoice per (invoice_id, shipment_id), one dwell / facility / contract per shipment. Same-values re-ingest is a no-op; changed values on a non-terminal case replace the fact and rematch; terminal cases never reopen in v1; match re-runs converge; re-sending the same decide on a terminal case is a no-op.
  • Observability without coupling. Domain types never import Neatlogs. A thin, failure-isolated tracer facade wraps AuditOffice methods and adapter calls. Every match stores a SHA-256 hash of the canonical-JSON evidence pack on the case, so a later auditor can see exactly what the $75 was computed from. Trace ids are stable per shipment (trace-freight-shp-88220).
  • Evals are fixtures, not vibes. HERO, DOCK_FALSE, DWELL_AT_FREE, OCEAN_SKIP, IDEMPOTENT, BAD_EXTRACT, AUTO_CLOSE, and STALE are executable fixtures with exact cent expectations. Pytest locked them before any UI existed.

5. Future improvements

5.1 Domain breadth (highest value next)

  • More accessorials. Fuel surcharge (validated against the contract's DOE week), residential delivery, reweigh/reclassification. The ChargeType join and rule-firing pattern are designed to extend without touching the case aggregate.
  • Duplicate-invoice detection. Same BOL + carrier + amounts under different invoice ids is a classic leak; a fuzzy-match warning lane would catch it pre-payment.
  • Overnight dock logs & timezones. v1 rejects departure-before-arrival. Facilities in other timezones and cross-midnight dwell need normalizing before dwell_minutes math.
  • Reopen terminal cases. Late-arriving dock IoT on a decided case currently creates nothing; a controlled reopen (new snapshot, preserves audit trail) would make the office more forgiving without breaking trust.

5.2 Platform hardening

  • Durable persistence. All persistence already flows through a single InMemoryStore object; extracting it into a CaseStore protocol and adding a Postgres adapter is a contained change. Add optimistic concurrency (version on AuditCase) instead of relying on fail-closed echo amounts alone.
  • Auth, orgs, tenancy. v1 has none by design (per PRD: "must not build auth, orgs, or a marketplace"). Multi-tenant lanes and role-based approval limits are the first production blockers.
  • Real ERP integration. Replace the PayablePosting / DisputePacket mock with a NetSuite (or EDI 210) adapter behind the same proposal contract, plus carrier-side dispute email delivery with the dock receipt attached.
  • Production serving. Flask's dev server and single-process Uvicorn need a WSGI/ASGI production setup; extraction calls should move off run_sync onto async paths.

5.3 Extraction & AI quality

  • Extraction confidence & review queue. Score InvoiceFact extractions and route low-confidence ones straight to NeedsReview with a highlighted "extracted, verify" badge.
  • Few-shot per carrier. Carrier PDF layouts vary wildly; per-carrier extraction examples (FedEx Freight, XPO, Estes) would lift first-pass accuracy and reduce Groq fallbacks.
  • Expanded eval set. Grow the 8 fixtures into a regression suite of real (anonymized) invoices with per-line gold labels, run in CI against both providers.

5.4 Product depth

  • Audit trail UI. The evidence-pack hash is stored today; surfacing "what exactly produced this $75" (facts, fired rules, evidence hash, Neatlogs link) inside the case modal turns observability into a controller-facing feature.
  • Policy authoring. Let the controller define thresholds per lane beyond the fixed $50 / SHORT-PAY-01 (e.g. per-rule dollar caps, auto-approve clean underbills), with the same human-approves-first gating.
  • Board ergonomics. Bulk approve, SLA timers on stale exceptions, CSV export of decided cases for finance close, and WebSocket-driven board refresh.
  • Anomaly surfacing. Lane-level trends, such as detention creep on one dock or a liftgate spike on one carrier, from accumulated case history. This can turn the AP desk from line auditor into spend controller.

6. Verification

  • uv run pytest backend/tests returned 37 passed, 4 skipped (opt-in live-sponsor suite). The suite includes the hero cents lock (92500 / 19500), API contract tests, extraction fallback tests, guardrail limiter tests, and Neatlogs emission tests.
  • Full stack runs locally: FastAPI on :8000 (Swagger at /docs), Flask + React on :5000; fixtures auto-ingest on startup and the hero case appears in Major exceptions with billed $1,120.00 / dispute $195.00.

If mermaid is not rendering, kindly check with any of the render, it will help.

Built With

  • fastapi
  • neatlogs
  • pydantic
  • pydanticai
  • pypdf
  • python
  • react
  • tensormux
Share this project:

Updates

Submission history