Inspiration
Flat-rate inference is the cheapest way to run a coding agent — $25/month for unlimited tokens across 21,000+ open models. The catch is the context ceiling: on that plan every model is capped at 32,768 tokens, even ones whose native window is 202,752. Go over and the request isn't slow, it's refused:
Maximum context length for model 'zai-org/GLM-5.1' allowed on your plan is 32768 tokens. Your prompt has 38435 tokens.
Paritok compresses agent context in flight. The obvious question is whether that buys back enough room to run a real agent on cheap infrastructure — and nobody had measured it on a genuine multi-turn coding workload. So we built the agent and the instrument to measure it.
What it does
Headroom is a coding agent that fixes real bugs in real repositories — native tool calls, a governor tracking context against the ceiling, retry/backoff, and pagination for oversized tool output.
Around it sits a comparison harness. headroom ab <task> runs the identical task under three strategies for oversized tool output — truncate it, paginate through it, or paginate through it via Paritok — against a pristine copy of the repo each time.
Comparing only "compression on/off" would be misleading, because the real alternative most agents use is truncation. The paginate and paritok arms are byte-identical apart from base_url, which is what makes the number attributable to compression and nothing else.
There's also headroom doctor, a preflight that verifies compression will actually fire before you trust a number — Paritok's own issue #29 asks for exactly this.
Four seeded tasks ship with it, so anyone can reproduce every number in five minutes.
The headline result: against an identical uncompressed arm, the Paritok arm used 12–52% fewer input tokens end to end (45–81% measured at the proxy) while seeing the same evidence and reaching the same fix — same patches, tests green, on every task.
How we built it
Python. httpx for the agent loop, FastAPI for the dashboard, SQLite for runs and transcripts. Upstream is zai-org/GLM-5.1 on Featherless via the Paritok proxy's --openai-url passthrough; compression runs on Paritok's hosted GPU.
Two things were chosen by measurement rather than assumption:
- The model.
GLM-4.7-Flashlooked right on paper (lowest concurrency cost). Probed, it measured 50.3s p50 with a 1-in-3 failure rate.GLM-5.1measured 2.7s with perfect tool-calling.tools/probe_models.pyis in the repo. - The message format. We first built the agent with a text protocol — JSON actions inside message content — deliberately, to route around Paritok's open tool-calling issues. It compressed 0%. Paritok segments compressible blocks out of
role:"tool"messages; the identical content as nativetool_callscompressed 98%. We rewrote the agent around that finding.
Challenges we ran into
Our own harness lied to us. The verifier ran pytest | tail -5 and read tail's exit code, so every run reported green regardless of outcome. Caught it on the first real run. Every result before that fix was worthless. There are now 38 tests over the harness, and that exact regression is one of them.
history.context_window defaults to 200000. Compression triggers at 80% of it — 160,000 tokens. Against a 32,768-token upstream that threshold is unreachable, so compression never fires and everything forwards at full size. Setting it to 32768 is what made the project work at all.
The demo we planned didn't survive contact with the data. We set out to show an agent dying without compression and surviving with it. It didn't happen: a competent agent greps instead of reading exhaustively and pipes verbose output through head. Our 217KB, 20-module, three-bug task peaked at 11,994 tokens against a 32,768 ceiling. We report what we measured, not what we hoped for.
And then we got a headline finding wrong, publicly. A single huge tool result — ~82,000 tokens from one pytest call — came back from /api/compress uncompressed. We swept block sizes, twice, and both sweeps showed the same clean cliff: fine below ~4,000 tokens, verbatim above ~6,000. We reported it as a size limit.
It wasn't. Another participant pushed back on our issue: the hosted GPU intermittently stops compressing and echoes input back. We re-tested with the control we should have run first — alternating 3,000- and 8,000-token blocks so both sizes sit in the same time windows. Under a size effect every pair must disagree. Result: 16/16 compressed, 0/8 pairs disagreed. We had swept sizes in ascending order against a time-varying fault, which manufactures a perfect size correlation. Two runs agreeing felt like replication; it was the same confound twice.
We retracted it on the issue and in the README, with the original text preserved rather than edited away.
The bug underneath is real and worse: while not compressing, /api/compress returns HTTP 200, gpu_available: true, and a plausible body, and the client reports below_refusal_threshold — which reads as a decision about content. We were probing with a purpose-built script, distrusted the result enough to run it twice, and still got it wrong. Ordinary users see tokens_saved: 0 and assume their content wasn't compressible.
Accomplishments that we're proud of
Publishing the result that contradicted our pitch. It would have been easy to build a task rigged to overflow and film that. Instead the README opens with "we set out to show X, the data says Y" — and the honest finding turned out more useful than the one we wanted, because it produced a reproducible bug report and a working fix.
Also: every number is reproducible by a judge in five minutes, and two of the measurement tools (probe_models.py, probe_block_size.py) are useful independently of this project.
What we learned
Compression is a substantial, reliable cost win on accumulated multi-turn agent context — roughly half the input tokens end to end, 65% at the proxy, with no measurable quality loss.
Two things surprised us. First, agents do a great deal of their own context management — we built a task with 462 failing cases across 41 pages and a brief instructing the agent three separate times to read every page, and it stopped after about ten anyway, reasoned correctly, and fixed the bug. Measure how much the agent is already saving before crediting a tool.
Second, truncation — the strategy everyone reaches for first — is not reliably the cheap one. On one task it burned 86,739 input tokens against pagination's 48,901, 77% more, and discarded evidence. On another it came out 16% cheaper, but again only by throwing a tool result away. A truncated chunk sits in context at full size on every later turn; pagination spends its tokens once.
What's next for Headroom
Push the reason field upstream — one field distinguishing no_relevant_content from backend_unavailable from below_threshold removes the entire class of confusion that made us misdiagnose our own data. Extend the harness beyond bug-fixing to RAG and long-document workloads. And run the same comparison across several upstream models, since how much an agent self-limits turns out to be a property of the model, not just the harness.
Log in or sign up for Devpost to join the conversation.