Inspiration

Watch an agent solve a real multi-step task and you see the same thing every time: twenty or thirty tool calls of trial and error to find a path that only needed four or five. Then the next run starts from zero and rediscovers it, paying the same tokens, the same minutes, and the same risk of taking a wrong turn. That is friction nobody chose. Several projects already let an agent save such a path as a new tool. What none of them answer is the question an engineer asks before letting generated code run in production: why should I trust it, and what happens when the tools underneath it change? tooldistill is our answer. The agent proposes; the library verifies. Nothing the model claims about a tool is believed; everything is measured.

What it does

tooldistill is a Python library, with a CLI and a Claude Code plugin, that you drop into an existing agent loop with a handful of lines. It records every tool call of a run into a trace. When a run succeeds and was expensive, it nudges the model once to propose a distilled tool: a small Python function that completes the whole task by calling the base tools in a fixed, known-good sequence. The next time a similar task comes up, the agent makes one call instead of thirty, and the work inside that call costs no model tokens at all. In our invoice demo a discovery run of twenty to thirty base calls became a single distilled call with four inside it.

Before any agent can use a proposal, it passes three checks:

Static: the source is inspected with the AST before it is ever imported. Only a small allowlist of imports and statements is accepted, every base tool call must name a tool that exists and pass its required arguments, and the dependency list comes out of this pass.
Replay: the tool is run against the responses recorded in the source run, matching each call on its exact arguments. No side effects, and a proof that the code reproduces the run.
Efficiency: with $m$ measured calls, baseline $b$ and a configured reduction $r$, the tool passes only if $m \le b\,(1-r)$.

Verified tools move through a lifecycle, draft → verified → active → suspect → deprecated, driven by evidence rather than by what the model says. When a base tool's schema changes, dependents are re-verified. When a tool that was down during discovery comes back, the affected distilled tools are flagged and can be rebuilt into a shorter version. When a distilled tool fails at run time, the agent receives a structured error, finishes the task with base tools, and that very run becomes the source of the next version. Base tools can be plain functions or MCP servers, and inside Claude Code the plugin's hooks record the session while its MCP server offers the verified tools back.

How we built it

Spec first. A single design document fixed the terminology, the file format for distilled tools, the store schema, the verifier, the lifecycle table and a build order with acceptance criteria, and every step shipped as one commit with its tests green. The library is Python 3.12+, managed with uv, with a SQLite store in WAL mode so the agent process and the CLI can write at the same time. The runtime uses a context variable so the same call() helper runs live through the developer's executor or replays from fixtures. The built-in distiller asks Claude for a proposal with structured outputs, and treats the answer exactly like one from the agent. The MCP adapter runs the async MCP client on a background thread so the rest of the library stays synchronous by contract. The Claude Code plugin is a thin adapter: a SessionStart hook creates the trace, PostToolUse hooks record calls, an MCP server exposes the tools, and a slash command replaces the nudge. We tested without the network (338 tests with fake tools and a scripted model), then ran the four-step demo against the real model and a real Claude Code session, and published 0.1.0 to PyPI.

Challenges we ran into

Replay is unforgiving. A single extra keyword argument the source run never sent makes the fixture lookup fail, so the rules the model reads had to say exactly that, and the hook-recorded outputs inside Claude Code had to be byte-identical to what the live executor records. They are.
Incentives leak into generated code. Early runs showed the model dropping a step the task required, a calendar conflict check, because fewer calls read as a better verdict, and proposing tools whose inputs were the answer. Both were fixed in the prompts, not the checks: a tool must do everything the task asked, and its inputs may only be what the request states.
A verdict that leaves the model guessing is a bug. "verified: 4 calls vs baseline 6" once led Claude to tell the user to promote a version that was already active. Verdicts now always state what became of the version.
Two event loops, one contract. Bridging synchronous library code with the async MCP client and Claude Code's own server loop took care: anyio insists a connection is closed by the task that opened it.
Small portability traps: uuid.uuid7 exists only on Python 3.14, so trace ids got a portable, time-ordered implementation; Windows shells treat <name> as redirection.

Accomplishments that we're proud of

The complete loop works end to end in both integrations: discovery, proposal, verification, reuse, recovery of a base tool with a rebuild, and a schema change producing a structured error, a fallback and a replacement version. The library never trusts the model and never calls a base tool on its own. It is on PyPI, and the plugin installs from the repository as a marketplace.

What we learned

Verification is a product feature, not a safety afterthought: users will only hand an agent generated tools if something other than the model vouches for them. Replay proves reproduction, not generalization, so a runtime safety net matters as much as the checks. And the text the model reads shapes the code it writes more than any rule in the verifier does.

What's next for Tool Distill

The problem has a buyer. Every team running agents in production pays for rediscovery twice: in tokens and latency on every run, and in the engineer who eventually hand-writes the tool the agent kept reinventing. tooldistill turns that into a measured saving per task, which is the number a team lead already tracks.

Distribution is in place. The library installs from PyPI, the Claude Code plugin installs from the repository as a marketplace, and both are where developers already look for tools. A written walkthrough of the sandbox agent is the first piece of developer-facing content; adapters for the other agent frameworks widen the funnel without changing the core.

The product is the shared store. A single developer gets the library for free. A team gets a hosted store where every agent proposes into one registry, one verification pipeline vouches for every tool, and a change in a base API retires or rebuilds the affected tools for everyone at once. That is the paid tier, and it is the same code behind a server.

On the roadmap next: a heuristic check that rejects inputs which only appear in the source run's tool outputs, a subprocess sandbox for distilled tools, and an efficiency measure that credits saved model turns and tokens rather than calls alone.