Dragstrip

Race the same coding agent with and without Paritok compression on any GitHub repo — live token meters, a savings receipt, and a blind judge scoring both answers.

Comment

Inspiration

My Claude Code bill is mostly input tokens. Every file an agent reads gets re-sent, and re-billed, on every turn after it. So when Paritok showed up claiming context compression could cut most of that, my first reaction wasn't "great, I'll install it" — it was "prove it on my repo, on my questions." No tool existed to show me that. Benchmarks are someone else's repo and someone else's questions. So I built the proof as a spectacle: put the raw agent and the compressed agent on a drag strip and let the meters settle it.

What it does

Dragstrip races two identical coding agents against any public GitHub repo. You ask one hard question ("how does request routing work, end to end?"). Both lanes get the same model, the same four repo tools, the same turn budget. The only difference: the Paritok lane pushes every request through Paritok's compression engine before it hits the API.

The UI is a live timing tower — cumulative billed input tokens per lane (read straight from the provider's usage field, not estimated), cost tickers, every tool call and compression event timestamped in each lane's feed. At the finish you get a receipt: tokens saved, dollars saved at the actual model's rate and projected at Claude Sonnet rates, plus a blind judge's verdict — a third model scores both answers without knowing which lane wrote which. Every race is saved and replayable.

On my recorded races the Paritok lane came in 18–71% under the raw lane on billed input tokens, with the judge scoring the answers even (it preferred the compressed lane's answer once).

How I built it

FastAPI backend, one-file frontend, no framework, deployed on Vercel serverless. The Paritok integration is SDK-mode, not the proxy: each turn's messages go through ParitokEngine.process_request() before the API call, and when the model calls Paritok's expand_context virtual tool to pull back an original file, resolve_virtual_call() answers it from local shadow storage — the lane feed marks those "resolved locally, zero re-read." Per-turn savings stream to the UI from the engine's own CompressionStats, and dollar figures use Paritok's pricing table. Compression runs on Paritok's hosted GPU, so every request also lands on my Paritok dashboard — the savings are verifiable, not self-reported. The agent side is any OpenAI-compatible provider; the deployed app runs Gemini 3.5 Flash Lite.

Challenges I ran into

Keeping the comparison honest was most of the work: billed usage instead of token estimates, identical prompts and budgets in both lanes, a judge that's blind to lane identity, and a guard that refuses to accept an answer from an agent that never opened a file. Serverless was its own fight — the whole race now runs inside a single streaming request so it works without shared state. And I hit a real Paritok bug along the way: the hosted GPU sometimes returns an empty string as a "successful" compression, which silently destroys the tool result. I filed it upstream with a repro and a proposed fix (paritok-4b-v1 issue #20) and guarded against it in the app.

Accomplishments that I'm proud of

The race format turns a benchmark table into something you can watch. The numbers are real billed usage. And the tool is honest enough to show compression losing ground — small explorations save 18%, deep ones save 70%, and the feeds show the wall-clock cost compression adds. That honesty is what makes the wins believable.

What I learned

Compression is query-aware summarization, not truncation — watching paritok-4b rewrite a 5,300-token file read into 583 tokens that still name the right classes changed how I think about agent context. The clever part is expand_context: compression is safe because it's reversible. Also: tool ecosystems are young — the difference between "it demos" and "it holds up" is finding the empty-compression bugs before your users do.

What's next for Dragstrip

CI mode. Run your agent's eval suite through both lanes on every Paritok release and get the savings/quality delta as a PR comment. The race becomes a regression test — for Paritok itself and for anyone deciding whether compression is safe for their stack.

Scope note: race results are single runs, not benchmarks — savings swing with repo size and how deep the agent digs, and the two lanes can take different exploration paths since compression changes what the model sees. The blind judge is an LLM and should be read as a smoke test; both full answers are shown so you can judge yourself. The live demo runs on free-tier API quotas, so lanes occasionally pause to ride out rate limits — the feeds display those stalls rather than hiding them.

Built With

Share this project:

Updates