Inspiration
Every AI coding tool we tried was willing to tell us a change was faster. None of them could give us a reason to believe it. "This should be roughly 30% faster" is a guess with a number attached, and on a laptop where run-to-run noise is 8%, a 4% improvement is indistinguishable from the machine being busy with something else.
The failure mode is worse than being wrong. A patch that is subtly faster and subtly incorrect looks exactly like a win, and it lands.
So we built the opposite tool. The model is allowed to be creative, speculative, and wrong. Something else — a harness the model cannot reach — decides what is true.
Most AI coding tools generate code. Hotpath generates evidence.
What it does
Point Hotpath at a repository. One command, hotpath go <url>, runs nine stages: it clones the repo, works out how to test and benchmark it, confirms the baseline is green and not flaky, runs the search, opens a draft pull request with one commit per verified change, and then watches the repository's own CI.
The loop inside it splits cleanly into who is allowed to be wrong:
profile → plan (AI) → write patches (AI) → verify correctness (harness) → benchmark (harness) → accept or reject (harness) → repeat
- A profiler finds where the time actually goes.
- A planner model reads the profile, the hottest source, and every past verdict, and proposes structured hypotheses. It stops proposing what already failed.
- Fast worker models write each hypothesis as a patch, in parallel, each in its own isolated git worktree.
- The harness runs the repository's locked test suite. Locked paths are enforced in code before a single byte is written, so a patch that tries to edit the tests is rejected before it ever runs. The planner is also told about them, but the enforcement is the guarantee.
- Only then does it benchmark, under an exclusive lock so nothing else on the machine is competing.
A change is kept only if it is correct and faster than the noise. The acceptance threshold is max(min_speedup, 1 + noise_multiplier × measured_noise), and the 95% bootstrap confidence interval over 2,000 resamples of the median ratio must exclude 1.0. On a quiet machine the 3% floor applies. On a noisy one the bar rises automatically. A noisy benchmark makes Hotpath stricter, never more permissive.
Everything rejected is recorded with the exact reason: the failing test's output, or the interval that still contained 1.0.
If the repository has no benchmark, Hotpath profiles its test suite, has a model write a benchmark over the hottest functions, and validates it by running it before trusting it — then shows it to you, and commits it, so the pull request says exactly what "faster" meant.
It refuses. hotpath check reads a repository in about a second and names what would stop a run before you spend anything, with a token cost estimate. And a run that proves nothing produces nothing: on mahmoud/boltons, benchmark generation failed, so Hotpath fell back to timing the whole test suite, measured 8.1% noise, put the bar near 16%, and reported 7 candidates and 0 accepted with no pull request. On life4/textdistance it stopped at the baseline, because that repository's property tests carry a 200 ms per-example deadline: a correctness suite that fails when the machine is busy cannot define "correct" for a timing experiment.
A real pull request: Nijjea1/inflect#1 — opened by Hotpath from a bare GitHub URL in 5 minutes 3 seconds, with no human step in between. 214 tests, 3/3 baseline runs green and not flaky, a benchmark Hotpath wrote itself and labelled (noisy: ±13%), 8 candidates, 2 accepted, 1.472× against baseline. The accepted head measured 1.409× with a 95% CI of [1.31, 1.45], against a bar that its own declared noise had raised to roughly 26%.
The rest is product surface around that core: a live local dashboard showing the experiment tree with a red node and a reason for every rejection, a metric-aware chart with the noise band shaded so a rejected_speed verdict is visible rather than something you have to read, a before/after bottleneck diff, leave-one-out ablation with verified joint pruning, and hotpath export for a PR-ready bundle.
How we built it
Two halves that do not trust each other.
The harness is deterministic and never asks a model anything. workspace.py manages git worktrees and enforces locked paths before any write. runner.py runs async subprocesses with hard timeouts and reliable process-tree cleanup. execution.py is a fail-closed Docker backend — no network, read-only source, non-root, resource quotas — with an explicit opt-in local backend for repositories you trust. benchmark.py does the parsing, the statistics, the bootstrap CI, and the accept/reject decision. harness.py sequences patch → test → bench → verdict and holds the quiet-machine lock.
The agent is where the models live. context.py does AST-based retrieval so the worker sees the actual hot functions rather than a truncated file. agent.py runs one planner call per iteration and N worker calls in parallel. The OpenAI provider uses the structured-output beta.chat.completions.parse API, so a hypothesis arrives as a typed object rather than prose we have to guess at. A mock provider replays recorded patches, which lets the entire loop — including deliberately wrong patches — run offline with no API key and real measured verdicts.
The search adds beam search, retry-with-feedback (a patch that fails to apply or fails the tests goes back to the worker once with the failure text, recorded as its own experiment), and interleaved parent re-benchmarking so machine drift between iterations cannot bias a decision.
Stack. Python 3.11+, Pydantic for every schema that crosses a boundary, FastAPI and uvicorn for the local dashboard, SQLite as the single source of truth for every number shown anywhere, Docker for isolation, cProfile and torch.profiler for profiling, PyTorch for the GPU target. OpenAI gpt-4.1 planned and a Baseten-hosted moonshotai/Kimi-K2.7-Code wrote patches. Sentry is wired through the whole run as traces, logs, and metrics, and is a no-op without a DSN. The site is TanStack Start, React 19, Tailwind 4, and three.js on Vercel. CI runs the suite cross-platform; a tag-driven workflow publishes to PyPI with trusted publishing.
Nothing in the dashboard or on the site is a placeholder. Every figure is read from a row the harness wrote.
Challenges we ran into
Our pull request broke someone else's repository. The first PR Hotpath opened failed every CI job on every platform. The benchmark file it committed imported hotpath.benchlib at module level. Our virtualenv has that package; jaraco/inflect's CI does not, and inflect collects doctests, so the suite died at collection with ModuleNotFoundError: No module named 'hotpath'. We had made a stranger's repository depend on our tool. Both benchmark templates are now self-contained stdlib-only scripts behind if __name__ == "__main__", with a test that runs them with PYTHONPATH stripped and asserts they produce no output on import.
A real repository is not a fixture. inflect/__init__.py is 107,342 characters. Our worker source budget was 14,000 — smaller than one ordinary module — so all 12 candidates failed before a model was ever called. The refusal itself was correct (a worker shown a prefix of a file it must write exact search/replace text against will invent the rest), but the budget was never fitted to the repository. Neither this nor the next one was reachable from our own demo targets, which are small, purpose-built, and LF.
A line ending cost us a day. Git for Windows sets core.autocrlf=true in its system config. Our git wrapper disables the system config on purpose, so untrusted repository config cannot point hooks and filters at arbitrary commands — and that disabled autocrlf too. Every file Hotpath rewrote then looked modified, and the repository was refused as dirty. Separately, a \r that survived into a copied .env put a carriage return inside the Baseten API key, and an entire H100 run failed every worker call for that reason.
Deciding what counts as a result. The hardest calls were the ones where the honest answer was "nothing". Keeping boltons in the evidence ledger as 0 accepted, and textdistance as a refusal, was more uncomfortable than shipping a number — and it is the part of this project we would defend hardest.
Accomplishments that we're proud of
- A pull request against a repository nobody here wrote, from a bare URL, with no human step in between, in five minutes.
- Zero CI failures introduced. On the replacement PR: 6 checks passed, 6 were already failing on the base commit, and 22 had no base result to compare against. Hotpath reports that last group as unknown rather than counting it as green, because a fork's default branch has no CI history.
- The refusals. Three repositories where the tool declined to produce a number, each with a written reason. A tool that always finds a speedup is not measuring anything.
- A noise floor that raises the bar. The inflect benchmark's own 13% noise pushed the acceptance threshold to roughly 26%. It would have been trivially easy to let a noisy benchmark make acceptance easier.
- Shipped. v0.1.0 on PyPI,
pipx install hotpath-agent, cross-platform CI, a documented CLI whose docs are generated from the argument parser with a test that fails on drift.
What we learned
The measurement is the product. We assumed the interesting part would be prompting a model into good optimizations. It wasn't. Models produce plenty of plausible ideas — fused attention, CUDA graphs, autocast, TF32. In one H100 run, 51 attempts produced 22 that broke correctness and 18 that were not faster than noise. Three shipped. The value was entirely in the machinery that sorted them.
A statistically insignificant win is the default outcome. TF32 measured 1.004× faster. A slice-assignment change in boltons measured 1.046× with a CI of [0.99, 1.07]. Both look like wins in a terminal. Neither is one. Without a confidence interval we would have shipped both and believed it.
Correctness has to be independent of load. This is the constraint that decides whether a repository can be optimized at all, and almost nobody writes tests with it in mind. A suite with timing-dependent assertions cannot define correctness for an experiment whose whole purpose is to change timing.
Refusing is a feature, and it has to be designed. "No result" needs to be as legible as a result: which stage stopped, why, and what flag would get past it. That is what hotpath check exists for — every finding in it cost us a ten-minute failed run to discover by hand.
What's next for Hotpath
- Automatic ablation-driven pruning at the end of every run. Verified joint pruning already works on request (
hotpath ablate --prune); it should just happen. - Device-time call-stack attribution. Torch CPU event trees give a real flame graph. CUDA device time and cProfile runs currently show a flat hotspot comparison, because those collectors do not provide a reliable nested hierarchy for that metric.
- A profile per accepted experiment. The orchestrator already profiles surviving beam nodes and throws the result away, so any node could be diffed against its own parent for free.
- Fire the CI repair loop on a real failure. Its control flow is tested — a fix the harness rejects is never pushed, the attempt budget bounds it, CI is re-read rather than assumed — but it has never repaired a live pull request, and we will not claim it until it has.
- A hardware-attested GPU result on a real model. Today's GPU evidence measures a bundled TinyGPT stand-in. A claim about a production model needs that model's own correctness definition, an attested host, and an independent re-check of the exported tree.
- More languages. The harness only needs a command that exits 0 for correct and a command that prints timing samples. Nothing in it is Python-specific except the profiler and the AST retrieval.
Log in or sign up for Devpost to join the conversation.