Receipts

Inspiration Twitch chat already has a perfect memory for one thing: the dumb promise a streamer makes mid-game and immediately forgets. "If we hit 500 subs, I'll do the hot sauce challenge" gets typed into chat fifty times over the next two weeks, then quietly dies when the moment actually arrives. Nobody wrote it down anywhere durable, not the streamer, not OBS, not the stream software. We wanted that receipt to live outside chat scroll, something on screen that gets louder the longer it's ignored and comes back with the exact original quote, dated, the moment it's due.

What it does Receipts is a single channel Twitch overlay and agent with memory across streams. Say a commitment out loud or type !promise, and it gets filed with a timestamp and a condition, maybe a sub target, a date, or "next stream." The overlay shows the open receipts, aging from plain to orange to red the longer they go unpaid. Chat can redeem Call in a Debt to open a 60 second vote over the three oldest promises, and the winner returns full screen with its original quote, the original date, and a stamp. The streamer can mark any debt Paid, Broken, or Dismissed, and the original record stays intact either way.

How we built it It's Node.js and TypeScript, with Express serving a transparent OBS browser source, a streamer control panel, and a microphone page, all synced over one WebSocket.

Twurple handles broadcaster OAuth and listens over EventSub WebSocket, so the whole thing runs loopback only with no public tunnel or webhook server needed. For hearing a promise, the browser's MediaRecorder captures 15 second chunks with a two second overlap and ships them to OpenAI for transcription.

Deciding what actually counts as a promise is the part we spent the most time on. A rolling 60 second chat window feeds a strict JSON schema extraction call, and a commitment is only accepted if it's an exact verbatim substring of what was said and clears a confidence floor:

$$ \text{accept}(u) \iff \text{confidence}(u) \geq 0.7 ;\wedge; u \in \text{transcript} $$

Softer stuff, like "I should probably clean my desk someday," gets logged as a rejected candidate instead of a receipt. Viewer chat is treated purely as context, never as a promise source.

To avoid double counting, every accepted promise gets embedded and compared against open debts using cosine similarity, computed in plain JavaScript since one channel's worth of promises doesn't need a vector database:

$$ \text{sim}(u, v) = \frac{u \cdot v}{\lVert u \rVert , \lVert v \rVert}, \qquad \text{sim}(u, v) > 0.88 \Rightarrow \text{restate, don't duplicate} $$

Everything persists in SQLite through better sqlite3, which holds streams, utterances, promises, votes, and embeddings stored as Float32 blobs. A condition watcher polls Helix every 20 seconds for metric and date thresholds, and it survives restarts without losing a queued callback or an open vote.

Challenges we ran into The hardest problem was making "verbatim" actually mean verbatim. The easy version of this lets the model paraphrase a promise into a tidy summary. The useful version has to show the exact words someone said, or the receipt has no teeth, so extraction had to anchor every acceptance to a real substring of the transcript instead of just trusting the model's gloss of what happened.

Close behind that was telling the difference between restatement and spam. Chat repeating a promise for two weeks shouldn't spawn fifty duplicate debts, but it should make the existing one visibly louder. Getting that similarity cutoff tuned so paraphrases collapse into one debt, without accidentally merging two genuinely different promises, took real trial and error against the fixture set.

We also had to make sure state actually survives a restart. Vote sessions, callback queues, and rollover debts all needed to persist through a server restart without losing an in flight vote or firing a callback twice for something already resolved. That pushed most of the logic out of memory and into SQLite as the real source of truth.

And honestly, staying honest about what's actually live was its own challenge. It's easy to fake a demo that looks like real transcription and a real chat audience. We drew a hard line between the scripted, fixture driven rehearsal and the live path that needs a real Twitch OAuth connection and an OpenAI key, and we labeled demo mode directly in the UI, because an app whose whole point is keeping honest receipts shouldn't fudge its own.

Accomplishments that we're proud of A promise can survive five weeks, a dozen paraphrases, and a server restart, and still come back as the exact original sentence instead of a summary of it. The whole pipeline, intake, extraction, dedupe, conditions, callbacks, and voting, runs identically in demo and live mode, so the demo isn't a mockup, it's the real code with fixtures standing in for Twitch and OpenAI. We also built a 15 case labeled extraction contract, eight promises and seven non promises, that catches regressions every time the extraction prompt changes, instead of relying on eyeballing output. And there's no vector database, no job queue, no multi tenant scaffolding anywhere in it. Just one SQLite file and brute force cosine similarity, because one channel's worth of promises genuinely doesn't need more than that.

What we learned Confidence thresholds and similarity cutoffs are easy to pick and hard to trust. The fixture set ended up mattering more than any amount of manual spot checking, because it caught regressions every time extraction got tightened. We also learned that the hard part was never whether a model could detect a promise. It was making sure the app still remembers that promise in five weeks, across a restart, without counting it twice. Durability turned out to be the actual feature, not detection.

What's next for Receipts We want to validate against real streamer speech to measure extraction precision outside the fixture set, and support multiple distinct promises inside a single utterance instead of just pulling out the strongest one. Vague, context dependent conditions like "when I beat this boss" need to get more reliable without the app trying to infer actual game state. And we'd like to add VOD timestamps and clips so a receipt can link straight back to the moment it was made, plus let streamers tune how aggressively old debts resurface on their own.