Tool Replay

Tool Replay — Local evidence console
Rerunning is not reproducing
The Tool Replay workflow
Async Record, Offline Replay, Fork, and Diff

Inspiration

I spend a lot of time building applications with AI models and LLM-powered workflows.

While working on these systems, I repeatedly ran into the same problem: when an AI application failed, reproducing the exact incident was extremely difficult. Running the application again could produce a different model response, a different tool call, a different retrieval result, or a completely different sequence of events.

The original failure would disappear, and I would no longer be debugging the same execution.

I started looking for a tool that could capture an AI incident, reproduce it without calling external services again, let me modify one recorded result, compare the new execution with the original, and preserve the verified correction as a permanent regression test.

I found observability platforms, evaluation frameworks, caches, and tracing systems, but I could not find a tool that covered that complete workflow at the runtime level.

That gap became the main inspiration for Tool Replay.

I kept coming back to one question:

What if AI executions were treated the way Git treats code?

Instead of hoping that a failure happens again, a developer could capture the original execution, replay it deterministically, create an alternate branch, inspect the first observable difference, and promote the corrected behavior into an offline test.

I was also motivated by the opportunity to create something open source that could be genuinely useful to the AI developer community.

Rather than building another closed dashboard, I wanted to create a lower-level debugging primitive that developers could inspect, extend, run locally, and integrate into their own applications.

Tool Replay does not attempt to reveal a model’s private chain of thought. It focuses on what can be captured and verified: model interactions, reasoning summaries explicitly exposed by providers, tool calls, inputs, outputs, failures, retries, application decisions, and structural divergences.

What it does

Tool Replay is a local-first behavioral debugger and regression-testing system for applications containing non-deterministic effects.

It allows developers to:

record external effects;
replay them without executing the real services;
reproduce both successful results and failures;
fork a recorded execution;
replace one exact recorded output;
run the application against the modified behavior;
compare the original and modified traces;
identify the first observable divergence;
promote a corrected execution into a normal offline pytest regression case.

A simplified example looks like this:

A CRM returns:

active = true

The application decides:

approved

Tool Replay records that execution.

I can then create a fork and replace the recorded CRM result:

active = false

The same application logic now decides:

rejected

The structural diff reports:

First observable divergence:

crm.customer.get
$.active: true → false

The modified branch is stored as an independent trace and can be replayed without the original source trace.

Once I verify the corrected behavior, Tool Replay can promote that branch into a self-contained regression case containing:

a content-addressed trace;
a deterministic case manifest;
a normal pytest test;
an explicit expectation over the corrected application result.

If the application bug is later reintroduced, the generated test fails as a behavioral regression while strict replay remains complete.

Tool Replay also supports non-streaming OpenAI Responses calls through explicit synchronous and asynchronous adapters.

This enables the full workflow with a real LLM execution:

real OpenAI call
  → record
  → strict offline replay
  → fork the recorded response
  → observe changed application behavior
  → diff both executions
  → promote the corrected branch to pytest

During replay, fork, branch replay, case verification, and generated pytest execution, OpenAI transport calls remain at zero.

How I built it

I started with a very small experiment: record the result of one Python function and replay it without running the function again.

Once that worked, the project grew milestone by milestone. I added failure replay, strict matching, counterfactual forks, structural trace comparison, and the ability to turn a corrected execution into a normal pytest case.

The most important design decision was to keep the core provider-neutral. Tool Replay sees external operations as effects, whether they come from an API, a database, a tool, or an LLM provider.

For persistence, I use SQLite traces, canonical JSON, and SHA-256 fingerprints. ContextVar keeps the active replay session local to the current execution context, while pytest cases make corrected incidents portable and runnable offline.

I then built explicit adapters for OpenAI and AsyncOpenAI. A live response is converted into a portable envelope, stored in the trace, and later reconstructed as a real public OpenAI Response object.

I built Tool Replay using Python, uv, SQLite, pytest, Ruff, and Pyright.

I used Codex and GPT models throughout the development process. GPT-5.6 Sol supported architecture, research, milestone planning, and technical review. Terra supported focused implementation and testing, while GPT-5.4 helped me explore the codebase, understand existing behavior, and identify the relevant execution paths before making changes.

I made the product, architecture, scope, and acceptance decisions, and independently validated each milestone.

Challenges I ran into

Making replay trustworthy

The first challenge was making sure replay could never silently use the wrong recorded operation.

A replay that looks successful but returns the wrong result would be worse than a visible error. For that reason, Tool Replay fails closed when an operation is missing, ambiguous, repeated unexpectedly, or left unused at the end of the execution.

I also had to prove that replay was actually offline. Removing an API key was not enough, so I used injected transports that counted every physical request and failed immediately if contacted.

That allowed me to verify that replay, fork, branch replay, case verification, and generated pytest tests all ran with zero provider transport.

Adding native async support

The hardest part of the project came when I added native async effects.

The original runtime was built around normal Python functions. Calling an async def function does not immediately return its result; it returns a coroutine. The first failing test exposed this directly when the existing runtime tried to serialize the coroutine itself.

Fixing that required a real async execution path rather than a wrapper around the synchronous one.

I also had to define what async support actually meant. Tool Replay supports sequential effects:

await effect_a()
await effect_b()

But it does not pretend that concurrent effects are deterministic.

The runtime must distinguish between an instrumented effect starting another effect in the same task and a second task attempting to start one while the first is still active. Both cases fail explicitly before matching a trace, consuming an operation, or even creating the live coroutine.

Cancellation introduced another difficult edge case. asyncio.CancelledError is control flow, not a normal application failure, so it could not be stored using the existing error-replay mechanism. I added a provider-neutral cancelled lifecycle while preserving compatibility with older traces.

This was the point where Tool Replay changed from a synchronous prototype into a more credible runtime.

Supporting AsyncOpenAI without building a second adapter

Adding AsyncOpenAI was not about copying the synchronous adapter and adding await.

The synchronous and asynchronous clients needed to share the same request validation, portable response format, error handling, version checks, security rules, and reconstruction logic. Only the invocation shell should be different.

This also created a surprisingly difficult typing problem. A single instrument(...) function supports both OpenAI and AsyncOpenAI, but callers still need to receive the correct concrete wrapper type.

I had to tighten the public typing without hiding the problem behind broad Any, ignored errors, or repeated casts throughout the tests.

Real provider responses broke my assumptions

The real OpenAI acceptance run exposed something the mocked tests had missed.

The adapter originally used a closed allowlist of top-level response fields. A valid response from gpt-5-nano included additional JSON-portable fields, so the live call succeeded but normalization rejected the result.

Instead of looking for a model that happened to return a smaller response, I corrected the contract. Tool Replay now preserves top-level JSON fields losslessly while remaining strict about the semantic structures it understands, such as output items, message content, usage, and unsupported function calls.

I also found a version-specific SDK incompatibility:

serialized:
in_memory

accepted during public reconstruction:
in-memory

It was a very small difference, but enough to break Response.model_validate(...).

I isolated that compatibility shim and reused it in both the synchronous and asynchronous adapters.

The final failure was not in the product

Near the end of the real AsyncOpenAI workflow, every important phase had passed:

live record:               1 provider request
strict replay:             0
strict fork:               0
independent branch replay: 0
case verification:         0

The generated pytest test also passed, but Python created a __pycache__ file beside it.

The acceptance harness recursively hashed the whole case directory and incorrectly reported that the promoted case had been mutated.

The canonical assets had not changed at all.

That turned out to be a bug in the acceptance harness, not Tool Replay. I corrected the audit to hash only the manifest, content-addressed trace, and generated test, then reran the remaining workflow completely offline.

Windows also contributed its own collection of sandbox, temporary-directory, and file-permission surprises along the way.

Accomplishments that I’m proud of

The accomplishment I’m most proud of is that Tool Replay now supports the complete workflow I originally imagined:

Capture
  → Replay
  → Fork
  → Diff
  → Promote to Test

It started as a small experiment that could record and replay one Python function. By the end of the project, it had become a provider-neutral runtime capable of reproducing successful results and failures, creating independent counterfactual branches, comparing executions, and turning a verified correction into a permanent regression test.

I’m also proud that I was able to preserve the same model for both synchronous and sequential asynchronous applications. Tool Replay supports normal Python effects, native async def effects, OpenAI, and AsyncOpenAI without creating separate replay systems for each one.

The real OpenAI workflow was an important milestone because it proved that the project works beyond mocked responses:

live record:               1 provider request
strict replay:             0 provider requests
strict fork:               0 provider requests
independent branch replay: 0 provider requests
case verification:         0 provider requests
generated pytest:          0 provider requests

The generated test passed without an OpenAI API key and without access to the original incident trace.

I’m especially proud that Tool Replay does not silently fall back to live execution. If an operation is missing, ambiguous, unexpectedly repeated, or left unused, replay fails explicitly instead of returning a result that only appears correct.

Another important result was preserving the original artifacts throughout the workflow. Source traces, branches, and promoted case files remained byte-identical after replay, diff, verification, and repeated pytest execution.

The final validation finished with:

210 tests passed
1 justified Windows permission skip
Pyright: 0 errors
Ruff: green

Finally, I’m proud that I was able to release Tool Replay as an open-source project. My goal was not only to build something for the hackathon, but to create a useful foundation that other developers can inspect, run locally, and extend for their own AI applications.

What I learned

A large part of this project was new territory for me.

Before building Tool Replay, I had not worked deeply with execution traces, deterministic replay, SQLite-backed runtime state, or the edge cases involved in reproducing asynchronous behavior.

I learned how to design and inspect SQLite schemas for traces, persist operations safely, keep source artifacts immutable, and reconstruct an execution without running the original effect again.

I also learned that replay is much more than storing a response and returning it later. It requires deterministic identity, occurrence tracking, strict consumption rules, lifecycle states, failure handling, and clear boundaries around what can and cannot be reproduced.

Adding async support taught me a lot about Python coroutines, ContextVar, cancellation, task ownership, and the difference between sequential asynchronous execution and true concurrency.

The OpenAI adapter also showed me how important SDK details are. A feature can look simple from the outside, but real compatibility depends on response models, serialization behavior, public reconstruction APIs, typing, retries, and transport-level validation.

I also improved the way I build software with AI.

Instead of using one model for every task, I worked with an agent-driven workflow where each role had a specific responsibility. I used an orchestrator to break milestones into smaller contracts, implementation agents for focused coding tasks, explorers to inspect the repository before changes, and reviewers to independently challenge the implementation and verify the evidence.

This taught me that AI-assisted development works much better when responsibilities are explicit and the agents do not all try to solve the same problem at once.

The most useful pattern became:

define the contract
  → inspect the existing system
  → delegate a focused task
  → implement with tests
  → review independently
  → validate the complete result

I also learned not to trust a green result without understanding what it proves.

A passing test did not automatically prove that replay was offline, that a trace remained immutable, or that a provider was never contacted. I had to add transport counters, byte-level hash checks, negative tests, and independent validation.

The biggest lesson was that building reliable developer tools is less about making the happy path work and more about defining exactly how the system should fail.

Tool Replay became much stronger each time an assumption was challenged by a real execution.

What’s next for Tool Replay

The next major step is to make replayed AI executions easier to understand, not only easier to reproduce.

Today, Tool Replay can capture, replay, fork, diff, and promote synchronous and sequential asynchronous effects into offline regression tests.

My next goal is to build a richer execution inspector that explains how an AI application arrived at its observable result.

For each captured execution, a developer should be able to inspect a timeline containing:

model requests and responses;
reasoning summaries or reasoning artifacts explicitly returned by the provider;
tool calls and their arguments;
tool results;
retrieval operations;
retries and failures;
handoffs between agents or components;
application decisions;
patched operations;
downstream behavioral changes;
the first observable divergence between two executions.

The goal is not to expose hidden chain of thought or private model reasoning.

Instead, Tool Replay should reconstruct the observable execution path:

model request
  → model response
  → tool selection
  → tool arguments
  → tool result
  → next model interaction
  → application decision

A local visual inspector could allow developers to move through that sequence, compare the original trace with a fork, and understand which recorded interaction changed and how the application reacted.

Other future areas include:

streaming record and replay;
partial and interrupted stream behavior;
support for additional model providers;
richer tool-call and agent-workflow inspection;
controlled retry replay;
improved integrations with existing AI frameworks;
easier trace sharing and portable regression-case workflows;
broader support for real-world agent applications.

Tool Replay will remain focused on observable and reproducible execution behavior. It will not claim access to private model reasoning that the provider does not expose.

The long-term goal is: