Inspiration
More and more teams fine-tune and ship their own language models, but checking those models for hidden safety problems before release is mostly done by specialist research labs. Red-teaming and black-box interpretability methods miss problems that stay hidden, like backdoors, weakened refusals and sycophancy. I wanted an open-source tool that lets any team look inside its own model and catch these problems in-house, before the public ever sees them.
What it does
You describe what you’re worried about in plain language, for example “did our fine-tune weaken refusals?” or “could this model have a hidden backdoor?” An agent then plans experiments that look inside the model. A 3D view shows exactly which layers the problem sits in, and a checkpoint timeline shows when during training it appeared. That tells the team where to look before they ship.
How I built it
- Engine: Python, FastAPI and the MCP Python SDK, with nnsight for reaching inside models. It runs as an MCP server and a command-line tool, so it works from Claude Code or any MCP client.
- Agents: the built-in agent and the reviewer both use the Claude Agent SDK, with hooks that enforce approval.
- Web-app: React, Zustand, three.js through React Three Fiber, and Plotly.
- AI Safety risks: sycophancy, harmful requests refusals, backdoor detection, and jailbreaks.
Challenges I ran into
- Building on a laptop with no GPU, so every check had to run on CPU.
- Keeping harmful test data inside the engine, so the agent only ever sees aggregates and redacted examples.
- Keeping the 3D view in sync with live events from long-running jobs, and letting the user dynamically control the model view in the canvas.
Accomplishments that we're proud of
- The tool catches planted problems in test models.
- Ability to identify and localize areas prone to risks during pre-training.
- The same engine works from the web app and from Claude Code.
- Everything runs locally on the user's hardware.
What I learned
- Interpretability results are easy to overclaim. Plain statements of what wasn't checked matter as much as the methods themselves.
- An agent is good at planning experiments, but approval has to be enforced outside the agent.
- Showing exactly where in the model a signal sits makes the results understandable to people who aren't researchers.
What's next for Model Safety Investigation Tool
- Checks for behavior drift between versions, hidden bias, data poisoning, evaluation awareness and alignment faking. Their interfaces are already designed.
- Distributed model support across several GPUs, and for quantized models.
- A standalone package install.
