Token Audit

Inspiration

Feeding a 200,000-token CI build log to an LLM works — and it's expensive. Compression tools promise to cut that cost, but every one of them ships a single marketing number and no way to check it on your workload. Practitioners are left to adopt compression blind or refuse it blind. We wanted a tool that answers the only question that matters: is the cheaper answer still correct?

What it does

TokenAudit (ships as the paritok-bench CLI) replays a real workload twice — once straight to the provider, once through the Paritok proxy — and reports cost and quality side by side. Then it turns that into a CI gate, so a prompt change that quietly makes your agent more expensive fails the pull request instead of shipping.

Our demo workload is a CI-failure triage agent that reads an entire failed build log and tells you what broke, which test failed, and where the fix goes.

What we built

• A paired A/B harness where the two arms differ only in whether the request traverses the Paritok proxy — same fixture, same model, same pinned settings, so the comparison is causally valid.
• Token counts read from the provider's own usage field, never a local estimate. A response without usage is excluded, never defaulted to zero.
• A structural grader, a regression gate, and an HTML report that pins git SHA, model, and Paritok version so every number is reproducible.
• A keyless demo: pytest runs Against a real failing GitLab-runner CI log with a 40-tool coding-agent catalogue attached, the provider's own usage reported 40.9% fewer input tokens through Paritok's tool-discovery pipeline — 10,299 → 6,083 tokens, 4,216 saved on a single turn. It reproduced three independent ways and both repeats were byte-identical.

We deliberately do not publish this as a headline: at 1 fixture × 2 repeats the harness stamps it PRELIMINARY and refuses to publish — the exact discipline the tool enforces on everyone else.

What we learned

The hardest bug was in our own storefront: a fact-check found the marketing site telling visitors to run commands and flags that didn't exist. For a tool whose entire purpose is failing a PR that ships an unverified claim, that was the precise failure mode it exists to catch. New rule: a command in documentation is a claim, and gets executed before it ships.

Challenges

• GitHub Actions logs require auth (403 unauthenticated), so harvesting real fixtures needed a token.
• The 4B content-compressor has a ~4k-token input ceiling and silently echoes above it — so end-to-end content compression needs proxy-side chunking, which we documented honestly rather than papered over.

Built with

Paritok, Python 3.11+, Apache-2.0.

Built With

• 2.0
• actions
• ai/ml
• apache
• ci/cd
• cli
• cost
• devops
• git
• github
• html/css/javascript
• license:
• llm
• open
• openai
• python
• sqlite