PeppaPrice - Your context-aware speech intelligence assistant

The Problem

A bank balance is a number, not an answer. It does not know rent is due in two days, that a subscription renews tomorrow, or that a paycheck has not landed yet. So people check a number that is correct but practically useless, then buy anyway and find out what it actually cost them a week later.

That gap between what your balance says and what you can actually spend is where real financial damage happens, quietly, one small yes at a time. Existing budgeting apps ask you to open them, dig through categories, and do the math yourself, long after the moment of the purchase has already passed.

PeppaPrice closes that gap at the only point it actually matters: the second before you buy.

Your bank balance tells you what's there. PeppaPrice tells you and does what's actually yours to spend.

PeppaPrice is a screen aware, voice native, smart macOS/Windows agent that fuses realtime multimodal capture with a deterministic, cent accurate financial engine. Point at anything and ask out loud, and it triggers a low latency voice pipeline while pulling live, Every affordability answer is fully grounded, computed server side in integer cents, with zero hallucinated numbers ever reaching your ears. Harder questions spin up concurrent AI specialist agents in parallel, running affordability, spending, and risk analysis as independent, auditable threads before final synthesis. Layered on top is a verified checkout pipeline and a local credit simulation engine, turning one voice interaction into a fully traceable, end to end financial decision loop.

Inspiration

A balance answers the wrong question. It says what is there now, not what is safe to spend after the bills already on their way. We wanted that second answer, spoken out loud, at the exact moment someone is looking at a price.

The Core Formula

$$ \text{safeToSpend} = \max\left(0,\ \text{balance} - \sum \text{upcomingBills} - \text{reserve}\right) $$

Every dollar figure spoken out loud traces back to this equation, computed in integer cents. No model ever invents a number.

System Architecture

flowchart TD
    U[User: voice or text] --> App[PeppaPrice macOS app]
    Screen[Screen capture] --> App
    App --> Voice[Realtime voice session]
    App --> Nessie[Capital One Nessie sandbox]
    Nessie --> Engine[Deterministic financial engine]
    Engine --> Answer[Spoken answer with real numbers]
    App --> Research[Claude specialist planner]
    Research --> Specialists[Affordability, spending, tradeoffs, horizon]
    App --> Shop[Product search and basket]
    Shop --> Checkout[Simulated sandbox checkout]

Tech Stack

Layer	Technology	Role
Native app	Swift, SwiftUI, AppKit	Menu bar companion, panels, overlay
Screen and input	ScreenCaptureKit, Core Graphics	Display capture, global hotkey
Audio	AVFoundation	Microphone capture, playback
Voice engine	Realtime voice API	Listening, thinking, speaking in one session
Reasoning	Claude	Screenshot analysis, specialist planning and synthesis
Financial data	MathWorks	Capital One Nessie API
Shopping	Serper API	Product search and price comparison
Infrastructure	Cloudflare Workers	Secret proxying, short lived credential minting
Financial engine	TypeScript	Deterministic cents based forecast baseline
Testing	Vitest, Playwright	Unit, integration, and end to end checks

Financial Engine Specifications

All money represented as integer cents, never floating point dollars
Bill statuses recognized: pending, cancelled, completed, recurring
Outflows always processed before inflows on the same day, so a same day shortfall cannot hide behind a later deposit
Overdue obligations not yet reflected in balance are marked incomplete rather than assumed settled
Scheduled income only counts when explicitly provider scheduled, never inferred from past deposits

Verified Against Real Data

Test scenario	Result
$200 purchase against a real seeded account	Projected minimum of negative $80
$10 purchase, same account	Projected minimum of $110
Live bill status enum	Confirmed as exactly 4 values via a real API validation response
Dollar versus cent ambiguity	Resolved by round tripping known values through live create and read calls
Full test suite	142 tests passing across domain, service, and integration layers

Credit Simulator Specifications

$$ \text{payment} = P \times \frac{r}{1 - (1+r)^{-n}} $$

Where $P$ is principal, $r$ is the monthly rate, and $n$ is the number of payments.

$$ \text{modeledAPR} = \text{minimumAPR} + \frac{850 - \text{score}}{550} \times (\text{maximumAPR} - \text{minimumAPR}) $$

Accepted inputs: credit score 300 to 850, principal $1,000 to $100,000, monthly income $1 to $1,000,000, existing monthly debt $0 to $1,000,000.

System Requirements

macOS 14.2 or later
Xcode with a valid signing team for local builds
Screen recording, accessibility, and microphone permissions granted
Node.js and npm for the supporting proxy and gateway services

How We Built It

Built the deterministic financial engine first, normalizing every dollar amount into signed integer cents and walking a rolling 14 day event timeline with outflows processed before inflows, before a single line of AI or UI code existed
Verified Capital One's Nessie sandbox live and field by field through authenticated GET and POST calls, letting a real HTTP 400 validation payload reveal the true bill status enumeration and an undocumented, server computed upcoming_payment_date field, rather than trusting a decade old SDK or inconsistent documentation
Added a native macOS menu bar companion in Swift and SwiftUI backed by AppKit, using ScreenCaptureKit for multidisplay aware capture and a low level CGEvent tap paired with Carbon's global hotkey registration for modifier based push to talk that works even when the app is unfocused
Wired a realtime, full duplex voice session over a persistent WebSocket streaming 24 kHz mono PCM16 audio in both directions, with a short lived, timing safe credential gateway minting scoped session tokens and generation based cancellation preventing a stale response from ever interrupting a newer one
Layered Claude on top for screenshot grounded reasoning, running a planning pass that selects up to three concurrently scheduled specialist roles using structured concurrency, each returning a discrete, completion driven finding that gets reconciled against the deterministic engine's ground truth before synthesis
Added a shopping search integration feeding a persistent, schema validated basket, plus a crash aware, actor isolated checkout state machine that durably persists intent before any network mutation and reconciles ambiguous or timed out responses instead of blindly retrying
Closed the loop with a fully local, zero network credit simulator running standard amortization mathematics and a linear score to APR mapping, backed by a full automated verification suite of 142 passing tests spanning the deterministic core, provider adapters, and service boundary
Incorporated a Matlab Graph analyzing the data and finance for robust visualization

Challenges

Nessie's sandbox was inconsistently available, so every field had to be verified against live responses
A fully passing test suite still hid a real bug, because a mock skipped the one thing that actually broke
Fast parallel teamwork meant merging carefully so nobody's work silently disappeared

What We Learned

Verify a third party API against the real service, not its documentation. Stale SDKs and inconsistently reachable docs will confidently lie to you, so we treated Nessie as a black box contract to be reverse engineered through live requests, letting real HTTP status codes and validation payloads define the actual schema instead of a decade old sample. Keep the model on explanation duty and the arithmetic on a system you can audit. A large language model is a phenomenal router and communicator, but it becomes a liability the moment you let it originate a financial number, so every dollar figure in this product traces back to a deterministic, integer cent engine that a language model can only read from, never write to. Run the real thing at least once, because a fully mocked test suite gave us false confidence for most of the build, and only a live, unmocked integration test surfaced a genuine, demo breaking recognition bug that every green checkmark had been quietly hiding. We walked away treating test coverage and production readiness as two separate claims that happen to correlate, not the same fact stated twice.