Inspiration Large Language Models are everywhere — ChatGPT, Claude, Gemini — yet most people interact with them as inscrutable black boxes. You type a prompt, magic comes out, and you move on. I kept asking myself: what actually happens inside the model between my input and its output? How does a word like "bank" know whether I mean a river bank or a financial institution? How does the model decide what comes next?

I wanted to build something that tears open the black box — not with abstract diagrams or research papers, but with a live, interactive dissection of a real transformer model running right in your browser. No server, no API calls, no "trust me, this is how it works." You type a sentence, and you see the model think.

The "bank" disambiguation example became my north star: when you feed the model "The bank of the river was muddy, but the bank closed at 5 PM," the attention heatmap vividly shows the second "bank" attending to "river" — the model resolving word meaning through context, in real time. That "aha" moment is what I wanted to give everyone.

What I Learned Building this project was a deep dive into the internals of transformer models and the @xenova/transformers library. Here are the key lessons:

The ONNX runtime doesn't always expose what you expect. The Xenova/distilgpt2 model exports only logits and past_key_values — not attention weights. I had to reconstruct attention patterns by computing k⋅k T⁄√(d_k ) from the key vectors in past_key_values, then applying a causal mask and softmax. This was the hardest bug — output.attentions simply doesn't exist in this ONNX export. Tokenizer APIs vary wildly. GPT-2's BPE tokenizer in @xenova/transformers doesn't have a .tokenize() method. You have to use .encode() for IDs and then reverse-lookup the vocabulary to get token strings. The "Ġ" character (Unicode 288) that represents a preceding space is GPT-2's internal convention — you have to manually clean it for display. Tensor access patterns matter. Tensor.toArray() doesn't exist — you access raw data via Tensor.data (a Float32Array). For logits, you slice the last token's distribution: logits.data.slice((seqLen-1) * vocabSize, seqLen * vocabSize). Shannon entropy reveals attention head specialization. Computing H=-Σ_(P_i ) log_2⁡PI for each head's attention distribution shows that some heads are highly focused (low entropy — attending to specific tokens) while others are broad (high entropy — diffuse attention). This matches the "specialized heads" finding from attention research. Spec-driven development saves you from yourself. Planning the architecture, scope, and technical decisions before writing a single line of code meant that when bugs like the missing output.attentions hit, I had a clear mental model of where the problem was and how to work around it.

How I Built It AI Under the Hood is a single-page Next.js 16 application that runs a DistilGPT-2 (82M parameter) transformer model entirely in the browser using WebAssembly. Here's the architecture:

Tokenization Module — The input text is tokenized using GPT-2's Byte-Pair Encoding. Each token is displayed as an interactive pill with color-coded indicators for word-start vs. sub-word tokens. A Token Context Window shows how the model "sees" your text, and a Frequency Analysis panel provides statistical breakdowns. Self-Attention Heatmap — The core visualization. After running model inference, I extract key vectors from past_key_values and compute attention scores.

The heatmap uses a perceptually-uniform viridis color scale (dark purple → indigo → teal → yellow-green) for accessibility. Users can navigate 6 layers × 12 heads = 72 attention patterns with keyboard shortcuts (arrow keys, L/H to cycle layers/heads).

Token Embedding Space — A 2D SVD projection (power iteration) of multi-head attention distributions, rendered as an interactive SVG scatter plot where hovering a token reveals its attention-weighted connections. Attention Head Similarity — A 12×12 cosine similarity matrix between heads, labeled by pattern type (diagonal/self-attention, first-token, broad, local). Token Similarity Network — A force-directed graph where nodes are tokens and edges represent attention-based similarity, with a threshold slider and draggable nodes. Next-Token Prediction — The model's logits are converted to probabilities via softmax, and the top-8 candidates are displayed with animated probability bars. Text Generation — Autoregressive token generation with full control: greedy vs. sampling modes, temperature (0.1–2.0), top-k (1–100), top-p nucleus sampling (0.1–1.0), and typewriter-style rendering. Analysis Dashboard — Real-time stats: token count, focus score, self-attention ratio, inference speed, strongest attention pair, and top prediction. The entire app is a single page.tsx file (~4000+ lines) with a dark glassmorphism UI, Framer Motion animations, keyboard shortcuts, export functionality (text/JSON/PNG), sentence comparison mode, and attention evolution playback with layer interpolation.

Challenges Faced Missing attention weights — The biggest challenge. The ONNX export of DistilGPT-2 doesn't include output.attentions. I spent hours debugging why output.attentions[l][0][h] was undefined before discovering this is a known limitation. The solution was to reconstruct attention from key vectors, which required understanding the transformer's internal tensor shapes: [1, 12, seqLen, 64] for each layer's key tensor. Tokenizer API mismatches — tokenizer.tokenize() doesn't exist in @xenova/transformers. I had to use tokenizer.encode() for IDs and then reverse-lookup tokenizer.model.vocab to get token strings — with null-safe guards because vocab can be undefined in certain environments. Null safety in browser environments — The teardown button crashed with "Cannot convert undefined or null to object" because Object.entries() was called on undefined vocab objects. I added defensive patterns with try/catch fallbacks to tokenizer.decode(). Invisible generated tokens — DistilGPT-2 loves generating newline characters after period-ended sentences, which were invisible in the UI. I added display mappings: \n → ↵, → ␣, plus whitespace-pre-wrap for proper rendering. Performance in the browser — Running a transformer model in WebAssembly is impressive but slow on first load (~10-15s model download). I cached the model and tokenizer in useRef and added a progress bar with step indicators.

Built With

  • canvas-based-heatmaps
  • cosine-similarity-for-head-comparison
  • css-frameworks:-next.js-16-(app-router)
  • css-grid-attention-matrices-math:-power-iteration-svd-for-2d-projections
  • framer-motion-(animations)
  • languages:-typescript
  • lucide-react-(icons)-ai/ml:-distilgpt-2-(82m-parameter-transformer-via-onnx-runtime-/-webassembly)-visualization:-custom-svg-scatter-plots
  • react-19
  • shadcn/ui-(component-library)
  • shannon-entropy-for-attention-analysis
  • tailwind-css-4-libraries:-@xenova/transformers-(in-browser-wasm-inference)
Share this project:

Updates

Submission history