Context Cost Auditor

• 

  README overview showing the hosted Paritok result and evidence-retention verdict.

• 

  Observed hosted sweep report showing high token savings but QUALITY_RISK.

Inspiration

Paritok can make long LLM contexts much shorter. But for coding agents and RAG workflows, token savings are not the only question. Developers also need to know whether important evidence survived compression.

Context Cost Auditor was built to measure both sides: token reduction and evidence retention.

What it does

Context Cost Auditor runs a long-context task through a Paritok compression lane and generates a Markdown audit report.

Each test case defines:

• the task
• source context
• required facts
• required paths or citations

The auditor reports estimated token savings, retained facts, retained paths, latency, and a qualitative verdict.

Hosted Paritok result

In a hosted Paritok GPU run on a noisy coding-agent debugging context, Paritok compressed about 2,900 estimated tokens down to 88-97 estimated tokens.

That is about 96.7%-97.0% token savings.

But the audit flagged QUALITY_RISK: only 1 of 5 required facts was retained, and only 1/3 to 2/3 required paths were retained.

The compressed output became a compact code patch rather than an evidence-preserving compressed context.

Why it matters

This is useful because token savings alone can hide workflow risk. For coding agents, code review, RAG, and multi-agent handoff, the downstream model may need evidence, constraints, citations, and audit trails, not only a plausible short answer.

Context Cost Auditor helps developers decide when compression is safe for the workflow and when it needs evidence-preserving behavior.

How we built it

The project is a small Python audit harness. It uses Paritok as the compression layer, local token estimates, JSON test cases, and Markdown report generation. The repository includes a reproducible example case and an observed hosted Paritok sweep report.

What we learned

The main lesson is that compression quality should be measured against the downstream task. In our observed run, Paritok achieved very strong token savings, but the compressed context dropped important debugging evidence.

This suggests a useful product direction for Paritok: an evidence-preserving compression mode or structured retained/dropped evidence metadata.

What's next

Next steps include adding more workflow cases, provider-backed answer comparisons, and structured audit output that can be used in CI or agent evaluation pipelines.

Built With

• benchmarking
• github
• json
• llm
• markdown
• paritok
• python
• rag
• token