Fathom

Inspiration

When an AI answer starts going in the wrong direction, you usually have two options: stop it and rewrite your prompt, or wait for it to finish and ask it to try again. Either way, you have to infer what went wrong from the text it produced.

We wanted a more direct way to intervene. What if you could see an interpretable signal from the model’s internal state at key points in an answer, choose a different direction, and continue from there?

That question became Fathom: a model that doesn’t mind being interrupted.

Fathom combines natural-language descriptions of model activations with interactive steering. It lets users explore and redirect an answer one step at a time.

What it does

Streams an answer. You ask Qwen2.5-7B-Instruct a question, and its response appears live.
Identifies checkpoints. Fathom detects section openings, such as numbered steps and Markdown headings.
Explains the model’s focus. An activation verbalizer translates a hidden-state vector from a replay of the answer into a natural-language description. Fathom condenses this into a short label, such as “defining the budget,” displayed alongside the relevant section.
Suggests alternative directions. Qwen generates two or three other routes the section could take.
Lets you steer. You can select an alternative, enter your own direction, such as “focus on growing in containers on a balcony.”
Creates a new branch. Fathom preserves the text before the checkpoint and streams a new continuation. You can steer again at a later checkpoint, or revisit an earlier choice through a clickable history.
Measures the change. An activation reconstructor scores how closely the new state aligns with descriptions of the original and requested focus.

These readings are interpretations of activation patterns, not a literal transcript of the model’s thoughts. The alternatives are generated suggestions, and the scores measure activation alignment rather than answer correctness.

How we built it

Model infrastructure

We deployed our model stack on a RunPod A100:

Qwen2.5-7B-Instruct, served through SGLang for the original streaming response.
A natural-language autoencoder (NLA) pair: an activation verbalizer (AV), which maps a 3,584-dimensional activation to text, and an activation reconstructor (AR), which maps text back into activation space.
A FastAPI sidecar running Qwen through Hugging Face Transformers, with forward hooks for capturing and modifying hidden states.

Our SGLang serving setup did not expose the hooks we needed, so we used the sidecar to replay answer prefixes, inspect activations, and generate steered branches.

Reading activations

When Fathom detects a section opening, the sidecar replays the prompt and answer up to that point. It captures the residual stream after decoder block 20, hidden_states[21], at the final token of the section’s opening line.

The AV translates that vector into a description, which Qwen condenses into a readable focus label. The display is paced so readings can appear alongside their sections.

Because the capture happens after the opening line has been processed, the reading describes the state at that checkpoint; it does not establish that we can predict a section before its heading exists.

Steering the answer

Steering combines a contrast between model states with a target heading.

1. Generate a target opening.

Qwen writes a title for the requested direction. We insert that title into the original heading’s formatting. For example:

Original: 1. **Choose the Right Location**:
Target: 1. **Grow in Containers**:

2. Compute the contrast.

We replay the preserved prefix with each opening and subtract the resulting block-20 states:

$$\delta = h_{20}\left(p \Vert o_{\text{target}}\right) - h_{20}\left(p \Vert o_{\text{original}}\right)$$

Here, $p$ is the preserved prefix, $o$ is a section opening, and $\Vert$ denotes concatenation.

3. Apply the steer and anchor the heading.

We add the contrast vector to the residual stream during the opening:

$$h_{20} \leftarrow h_{20} + \delta$$

The branch begins with the target heading as text. The intervention ends at the end of that heading, and Qwen continues generating the section body.

Steer away has no single target heading. It steers toward the average state of the alternative openings and lets the model generate its own opening.

Measuring the change

The AR measures alignment; it does not generate the steering vector.

For a focus description $n$ and an activation $h$, we compute centered cosine similarity:

$$\operatorname{score}(n,h) = \cos\left(\operatorname{AR}(n)-\mu,\; h-\mu\right)$$

The calibration mean $\mu$ removes a shared offset across residual states, making the comparison more informative.

Frontend and branch management

We built the interface with Next.js 16, React 19, Zustand, and shadcn/ui, using our seaside Surface & Deep theme.

Each run and branch has a unique run_id. Steering later in a branch inherits earlier interventions, while retrying the same checkpoint creates a sibling branch. Events are associated with their runs so delayed responses cannot appear in the wrong branch.

Our CI runs Ruff, unit tests, and a type-checked frontend build on pull requests.

Challenges we ran into

A useful measurement vector was not necessarily a useful steering vector. Our first approach used AR reconstructions directly for steering. Those vectors were topic-selective enough to help with scoring, but interventions often produced weak or degraded output.
Stronger steering did not mean better steering. Qwen’s own contrast vectors preserved fluency more effectively, but unanchored steering reached the requested target only about one-third of the time in our experiments. Increasing the strength or intervening across several layers could break generation.
Reliable redirection needed structure. Combining the activation intervention with a target heading gave our strongest results. In a small evaluation, a blind judge rated all five anchored target cases as fully following the requested direction, while their unsteered controls received “none.” This evaluates the combined approach; it does not isolate how much improvement came from the activation intervention versus the heading.
Alternatives had to be meaningful to people. Nearby vectors did not reliably translate into distinct, useful choices. Generating alternatives with Qwen from the activation reading produced clearer options.
Small formatting differences mattered. A mismatched heading format or a title-generation prompt that echoed its schema could derail continuation. Generating only the title and preserving the original markup helped.
Memory and replay added constraints. The GPU-resident models used roughly 73 GB of an 80 GB A100, so we ran the AR on CPU in FP32. Branch replays also had to preserve earlier interventions to keep readings consistent with the branch being shown.

Accomplishments that we're proud of

Building an end-to-end loop of reading → explaining → steering → measuring with live models.
Producing fluent, targeted branches and evaluating them against unsteered controls.
Supporting repeated steering, sibling retries, and navigable branch history.
Turning activation data into an interface that people can use without working directly with tensors.
Clearly distinguishing activation readings, generated suggestions, and measured alignment scores.

What we learned

Interpretability becomes more useful when people can act on it. A readable focus label provides a starting point for deciding whether to continue, redirect, or explore another branch.

Reading and intervening are different problems. A vector that identifies a concept well does not necessarily cause the model to generate that concept when added to its hidden state.

Text and activation interventions can work together. Our strongest observed results combined a target heading with a contrast-based steer. Separating their individual contributions is an important next experiment.

Small deterministic components matter. Checkpoint detection, markup preservation, bounded intervention windows, and branch tracking made the generative components easier to control.

Evaluation needs careful interpretation. Five successful cases are encouraging, but they are a small sample. Our original and branched responses also use different decoding setups, which matters when comparing their behavior.

What's next for Fathom

Capture activations during generation. Integrate hooks into the serving path to reduce replay latency and memory use.
Support more models and layers. Extend the approach as compatible NLA pairs become available.
Learn from user choices. Use steering histories to study where generated answers diverge from user preferences and inform future evaluation or fine-tuning.