-
-
Paritok Dashboard
Inspiration
Every coding agent has the same failure mode: it answers confidently about code it never saw. Retrieval hides this. Top-k search returns five plausible chunks, the model writes a fix against those five, and the call site that actually breaks was ranked sixth. The agent is not wrong because it reasoned badly. It is wrong because the evidence was never in the room.
The obvious fix, stuff the whole repository into the context, does not fit. So the field settled on retrieval and learned to live with silent misses.
Paritok changes the shape of that problem. Its compression endpoint answers a segment with a bare string, and an empty string meanhe stated intent." That is not a compression artifact. That is a semantic verdict worth zero tokens. If a model can reliably say "you this task," then admission control, not ranking, is the right primitive, and a whole repository can be made to fit.
Resident is the attempt to hold an entire codebase r context, and to prove it did not quietly lose anything on the way in.
What it does
Resident takes a working tree and an intent, and returns a residency: a single context payload containing the whole repository, comt, with a guarantee attached.
resident scan <path> -i "why does the retry loop drop the last attempt"
resident ask <path> "which call sites construct
resident bench <path> -i "<intent>"
resident doctor
The pipeline is four stages.
Segment. The tree is split on syntax rather thanto 3000 tokens. The upper bound is the shipped runtime's chunk size. The lower bound exists because sub-floor content can come backa false keep. A verdict on half a function is a verdict on nothing, so the splitter walks the Python AST and emits in deterministic ordeon order, which is what keeps the prompt prefix byte-stable across runs.
Judge. Paritok scores every segment against the intent under bounded concurrency with retries. Drops cost zero tokens. Kehe original.
Ledger. Every lossy verdict is audited and backstopped. This is the core idea of the project and the thing that made the numb
Assemble, or refuse. If the residency does not fr than truncating. A truncated residency is top-k retrieval wearing a coverage claim, and shipping that would defeat the entire poi
The fact ledger
The measurement that produced the whole design, take containing 86,877 extracted atoms:
A critical atom averages 4.75 tokens, and losing one costs a full fidelity point. Atoms are cheap in tokens and expensive in
Atoms are the load-bearing exact strings: file paths
identifiers, URLs. A compressor paraphrasing raise ConfigError into "raises a
configuration error" has destroyed the agent's abili
about four tokens doing it.
The existing contract-based baseline, nutsheLLM, gets the diagnosis right and the remedy wrong by an order of magnitude. It insist verbatim, and when facts die it restores the entire original segment. That is why it collapses to 25.9% reduction on our corpus.
Resident restores only the missing facts:
- A kept segment gets its compressed body plus a fla were present in the original and are absent from the body.
- A dropped segment becomes a tombstone: its atoms a ref id so the drop is reversible through the pager.
- A segment carrying no atoms is a true zero-token d atom-free content loses nothing.
Both blocks are literal substrings of the source, so retention stays decidable by exact string search, which is the same rule the ies by. Nothing here is a metric trick. The agent downstream genuinely receives every load-bearing path, error name, and identifier, at a instead of a few hundred.
How we built it
The evaluation is the product, so it was built to survive someone who does not trust us.
**Eight strategies, twelve repositories, three repet repositories totalling 2,148,113 tokens and 2,271 segments, run as a full head to head. Five of the eight arms ask Paritok the iden only in how they treat the answer, so one verdict per segment serves five arms and the cost is 3N per repetition rather than 8N. Ev after each repository and repetition, so an interrupted run resumes instead of restarting.
**Three repetitions, because the compressor is nondeool measured the same input scoring 47.8% and 42.8% on consecutive runs. A single run cannot separate two arms whose gap is smaller than t recorded separately and the spread is published alongside the mean, so a difference that sits inside the noise is visible as
Two independent scorers. Fidelity is computed by by a second, differently built one. Both numbers are reported. An arm that only looks good under the scorer we chose is not a result
Billing is measured, never estimated. Two traps,
invalidates the headline. First, tiktoken is the wrong tokenizer for billing: it
is an approximation, and real counts run about 1.0 t
allowed to size things and never to price them. Second, and worse,
usage.input_tokens alone under-reports by the enti
figure is input_tokens + cache_creation_input_tokens + cache_read_input_tokens.
A system whose entire design is a large stable cache
all of its own cost and report a spectacular, completely false saving in the
flattering direction. Every published number is reco
append-only log.
The verdict boundary is parsed exactly once. Paritok answers with a bare
string, and several very different events all arrive
genuine drop, a torn connection truncated mid-body, a backend that is down and
passed the original through, half a block cut off by
Collapsing those into str | None is precisely how a team ends up publishing a
97% saving on a run where the compressor never execunto
a closed set of outcomes and every downstream component consumes that, not a
string.
The pager appends originals at the end. Splicing the middle of the context invalidates every cached token after the splice, so originals are appended, keeping the residency prefixur own paging and never use Paritok's virtual expansion tool, because the proxy answers an expansion by re-POSTing the whole thread , which costs T(1+2c) and is therefore worse than sending T uncompressed for any compression ratio. That loss lives in the gateway's n compression, so owning the pager avoids it by construction instead of mitigating it.
Residency content is treated as untrusted. Parit verbatim, which means a string placed inside a source file reaches the model intact. The residency is wrapped in a nonce-delimitempt marks it as data rather than instruction, and the tool surface is read-only with no shell and no writes.
Results
12 repositories, 2,148,113 tokens, 3 repetitions pery exact atom retention. Fidelity 2 is the second independent extractor.
| Strategy | Reduction | Fidelity | Fidelity 2 |
|---|---|---|---|
| Resident with fact ledger | 49.3% | 100.0% | 97.8% |
| Stock plus ledger (ablation) | 41.1% | 100.0% | 98 |
| Segpilot plus ledger (ablation) | 44.1% | 100.0% | 98.2% |
| Resident, tombstones only | 58.6% | 61.8% | 63.2% |
| Resident, no ledger | 60.6% | 53.0% | 54.8% |
| Segpilot | 53.8% | 60.9% | 63.0% |
| Stock compression | 50.4% | 61.8% | 63.8% |
| nutsheLLM | 25.9% | 79.1% | 80.2% |
The comparison that matters is the top row against t trying to guarantee that exact facts survive. nutsheLLM buys 79.1% fidelity for 25.9% reduction. The fact ledger buys 100% fidelity ly double the compression at strictly higher fidelity, because it pays 4.75 tokens to save an atom instead of restoring the several hun it.
The ablation rows are the honest part. Adding the ledger to the stock and segpilot arms also drives them to 100% fidelity, whi ledger, not our admission policy, is what produces the guarantee. Our admission policy is what makes the guaranteed configuration th
Run-to-run spread across the three repetitions stays reduction and under 0.3 points of fidelity, so these gaps are outside the noise floor we measured.
Challenges we ran into
We deleted our best-sounding idea. The prefilter argument that coverage was architectural rather than statistical: a conservative guard force-keeps anything that might carry a relevae cannot lose one. It is a genuinely good argument. Two independent corpora then measured it contributing nothing. On 12 repositories known sites, Paritok alone found 18 of 20 at 478,335 tokens. With the guard on, recall was identical, 18 of 20, and the cost rose to reduction where Paritok alone reaches 66.6%. The guard was undoing the compression it existed to protect. It is now off by codebase as an option, described as insurance rather than as a claim.
A structural improvement that measured worse. Grouping ledger atoms under their enclosing def or class is clearly better organ measured, and removed. It cost 2.31 points of reduction and moved the context rate from 5.8% to 5.9%. The flat list is uglier and
Nondeterminism nearly fooled us early. Two arms n a single run looked like a finding. The compressor's own run-to-run variance is about five points. Repetitions and published spreads
Dependency pinning is part of the result. Parito.8,
not >=, because 1.2.9 is announced to change stats semantics such that
re-delivered originals fold onto the compressed sideve.
A silent minor upgrade would change the substrate underneath a published
benchmark. The audit extractor is pinned to a specif
branch, because the ledger's fidelity guarantee is defined by that exact
extractor.
Accomplishments we are proud of
100% exact-fact fidelity at 49.3% token reduction on confirmed at 97.8% by a scorer we did not write, with the mechanism isolated by ablation and every number recomputable from raw logs
Just as much: a benchmark that reports its own null retreat and the removed grouping are in the code comments and in this writeup because a benchmark you can only lose by lying is no
What we learned
Fidelity loss is not diffuse, it is concentrated in strings. Once you can name that set, the guarantee stops being expensive. That reframing is the whole project, and it only became v token cost per lost atom rather than measuring compression ratio alone.
We also learned how easy it is to publish a false saving in good faith. Both of the accounting traps we hit, the approximate tokeniz fields, fail in the flattering direction. Neither would have thrown an error.
What is next
Reversible tombstones already carry ref ids, and the next step is a retrieval
path that pages the full original back on demand, wh
well past 60% while keeping permanent loss at zero. Per-type routing for JSON,
logs, and diffs looks worth another 10 to 15 points
Beyond that: re-emitting broken relationship pairs such as a split raise
ConfigError as verbatim snippets, which measured a m
8.4% to 11.9% for 0.73 points of reduction, and language support past Python,
where the AST splitter needs a per-language replacem
Log in or sign up for Devpost to join the conversation.