Inspiration
Every time a language model writes a word, it first computes a probability distribution over its entire vocabulary — tens of thousands of possible next tokens, each with a confidence — then picks one and discards the rest. That distribution is the model's doubt: its hesitation, and the sentences it almost wrote. No chat interface ever shows it to you. We wanted an instrument that makes that invisible quantity visible and audible, live, as the model thinks.
What it does
UNCERTAIN streams a model's response token by token and, for every token, renders its next-token entropy as a continuous, accumulating "long-exposure" portrait — never a chart. Near-certain tokens land as sharp, saturated single strokes; uncertain ones scatter into wide, desaturated marks, with faint branches peeling off for the alternate tokens nearly chosen instead — ghosts of the sentence that didn't happen. A Web Audio engine sonifies the same stream: the top-k probabilities become a chord, and rising entropy detunes it into beating dissonance. Playback follows real token timing — network jitter and all — so you hear the actual pace of the model thinking, not a smoothed animation loop.
It ships in two clearly-labeled modes: Demo replays a real captured logprob trace through the identical rendering pipeline (zero setup, no key), and Live streams from your own key against any OpenAI-compatible endpoint.
How we built it
One self-contained index.html — vanilla JS, Canvas, Web Audio, no framework, no backend. Per token, over the API's returned top-k candidates:
- Entropy:
H = -Σ pᵢ · log₂(pᵢ)(bits) - Decisiveness:
gap = p(top) − p(runner-up)
The demo traces are real, captured from @cf/qwen/qwen3-30b-a3b-fp8 via Cloudflare Workers AI's OpenAI-compatible endpoint (top-5 logprobs plus real inter-token timing), and injected into the page so anyone can see the full piece instantly. Measured entropy ordered exactly as the prompts were chosen to: "capital of France" ≈ 0.31 bits (most certain) < "I opened the door and…" ≈ 0.42 < "feared or loved" ≈ 0.45 < "the most important thing I never told my mother" ≈ 0.54 bits.
What's real vs. pre-recorded (stated plainly)
Live mode is the model generating in front of you, right now, on your key. Demo mode is not live — it replays a trace from one real API call, at the real recorded timing. Both run through the exact same math and renderer, and the two modes are visually flagged at all times so it's never ambiguous which you're seeing. Nothing is hand-faked; if a live stream breaks, the piece shows the true state (offline/reconnecting) rather than substituting invented numbers.
The honest approximation
The API returns only the top-k (≤5) candidates, not the full ~100k-token distribution, so our entropy is computed over just those k — which means it systematically under-estimates the true entropy: the hidden tail mass still carries positive entropy our number can't see. The bias is negligible for near-certain tokens and largest for genuinely uncertain ones. We didn't just note this in a comment — the in-page self-test suite constructs the exact case and asserts the direction of the bias, and it runs on every page load with its pass/fail count shown in the header.
The test suite that caught a real bug
The entropy math is the one thing that must be right, so we wrote plain-JS tests asserting analytical answers: uniform over k tokens gives exactly log₂(k), a certain outcome gives 0, [0.5, 0.25, 0.25] gives 1.5, and edge cases (single candidate, empty list, zero-probability entries) don't produce NaN or crash — 13 tests in all.
The bug we didn't expect came from rendering, and a pipeline test caught it: running a full captured trace through the real Renderer and inspecting the canvas pixels, every token's mark landed on the same coordinate. _advanceCursor handled line-wrapping but never moved the cursor forward in the normal case — so 476 tokens were painting on top of each other into a single blob. One line (this.cursor.x += width) fixed it, but the math tests alone would never have revealed it.
Challenges we ran into
The hardest part was getting real logprobs. We learned the hard way that Cloudflare's native /ai/run endpoint streams {"response":"…"} with no probabilities at all, and that logprob support is per-model — Llama-3.1-8B returns none, while qwen3-30b (vLLM-backed) exposes full top_logprobs through the OpenAI-compatible /ai/v1 endpoint. So we made both the app and the capture script detect an endpoint that streams text-but-no-logprobs and fail loudly with the exact fix, instead of silently drawing a blank canvas.
What we learned
A reasoning model turned out to be an unexpectedly good subject — the demo text is the model deliberating out loud, so you literally watch its entropy spike at the moments it's genuinely unsure. And "prove the math" and "prove the render" are two different jobs: our analytically-verified entropy was perfect while the picture was silently broken.
What's next
More providers, a wider top_logprobs window where models allow it, and exportable "certainty portraits" as shareable prints.
Log in or sign up for Devpost to join the conversation.