TK Compression Chamber

• 

  82.7% fewer tokens, with all 11 required deliverables retained.

Inspiration

AI coding agents repeatedly exchange large amounts of project context: repository structure, requirements, constraints, prior decisions, and content the model has already seen. That repetition consumes tokens in both directions.

Simply summarizing the context is unsafe. A shorter prompt is not useful if it changes a path, hash, version, prohibition, acceptance rule, or decision boundary. We wanted to explore a different question:

Can we send less context to an AI coding agent while still reconstructing the complete, authorized project deterministically?

That led to Torch & Key Compression Chamber, or TK_Compression Chamber: a governed environment that treats compression as a controlled compilation process rather than ordinary summarization.

What it does

TK_Compression Chamber converts human intent and selected repository state into a compact, canonical sidecar representation.

Project Foreman classifies information into distinct authority classes:

• Exact: identifiers, paths, versions, hashes, numbers, prohibitions, security constraints, and acceptance rules. These are never semantically compressed.
• Structural: schemas, templates, renderer selections, dependency declarations, and repository references. These are represented through deterministic sidecars.
• Semantic: eligible explanatory prose that may be proposed to Paritok for compression.
• Derivable or nonessential: content that can be reconstructed locally or removed with an auditable reason.

Paritok proposes a smaller representation. A Compression Admission Gate determines whether that candidate is both safe and economically worthwhile. The model can then receive compact sidecars and unique content instead of an entire repeated repository.

After the model responds, the local Sidecar Project Compiler reconstructs the complete artifact, validates its schemas and authority rules, stages it safely, runs tests, and produces an evidence receipt.

A content-addressed local cache can also resolve previously admitted components by exact identity, version, authority, schema, and SHA-256 hash.

How we built it

We built TK_Compression Chamber primarily in Python 3.13, with PowerShell launchers for bounded Windows execution, JSON Schema for strict contracts, SHA-256 identities for custody and reconstruction checks, and a FastAPI reference archetype.

We developed the architecture through a series of controlled experimental routes:

• Route A established a conventional full-context baseline.
• Route B tested a smaller, self-contained governed context while requiring the same complete answer.
• Route C combined canonical sidecars, compression admission, model-produced deltas, deterministic local reconstruction, validation, and receipts.
• Route D explored cache-assisted reconstruction while preserving the independent evidence and authority boundaries of the earlier routes.

The Sidecar Project Compiler supports six rendering mechanisms: raw text, structured documents, template-plus-data rendering, directory scaffolds, patches to existing files, and asset references. Writes use exclusive staging, path-boundary checks, readback verification, and no-clobber finalization.

We also built the Compression Crucible, an offline governed test chain that exercises conversation normalization, authority projection, Project Foreman materialization, compression, cache resolution, reconstruction, and acceptance evidence across isolated runs.

Challenges we ran into

The hardest problem was not making text shorter. It was proving that compression had not changed authority.

A single softened prohibition or altered identifier can invalidate an otherwise fluent result. We therefore separated exact, structural, and semantic material and made the pipeline fail closed whenever preservation could not be established.

We also learned that token efficiency must be measured bidirectionally. Reducing input tokens is not enough if the model compensates with a longer response or if verification and retry costs erase the savings. Our accounting therefore includes the complete request-and-response exchange and evaluates quality independently from efficiency.

Strict structured output introduced another challenge. We had to reconcile JSON Schema compatibility, output ceilings, complete-result validation, and model response envelopes without weakening the contract.

Windows portability exposed additional issues involving CRLF materialization and byte-level preservation. We addressed these by distinguishing working-tree bytes from Git blob identity and by keeping canonical serialization rules explicit.

Finally, integrating separately governed project lineages required careful reconciliation. We retained named source baselines and evidence identities instead of replacing them with one broad “current state” assertion.

Accomplishments that we're proud of

In the controlled Route C v0.1.3 benchmark, TK_Compression Chamber:

• Reduced total bidirectional token use by 82.7% compared with the corrected Route A baseline.
• Reduced transmitted request bytes by 85.3%.
• Preserved all 11 of 11 required deliverables.
• Earned a quality score of 0.9375, with only minor loss of explanatory detail.
• Preserved every critical authority and restriction gate.
• Reconstructed a complete 12,331-byte result.
• Produced a byte-identical deterministic rerun.
• Completed with zero automatic retries.
• Passed all 189 tests in its full Windows acceptance suite.

The Compression Crucible also completed its full governed offline chain twice with matching governed material identities. Its portable validation suite passed 225 of 225 selected tests, while custody-dependent tests remained explicitly excluded rather than being simulated or silently bypassed.

These results support a deliberately bounded claim: the 82.7% reduction applies to the controlled Route C v0.1.3 benchmark and depends on governed local reconstruction. It is not a universal performance claim.

What we learned

The most important lesson was that safe compression is a systems problem.

Compression becomes much more useful when the model is not responsible for reproducing every repeated byte. Exact values can remain local, structure can be represented by deterministic references, and the model can focus on unique semantic work.

We also learned that governance can improve engineering rather than merely constrain it. Explicit authority classes, receipts, no-clobber writes, preservation tests, and bounded execution made failures easier to diagnose and results easier to trust.

Finally, quality and efficiency must be evaluated separately. A dramatic token reduction cannot rescue a result that loses a critical requirement. The quality gates have to pass first.

What's next for TK_Compression Chamber

Next, we want to:

• Separately benchmark the governed upstream Paritok route, including its hosted GPU path, without mixing those results with the existing controlled benchmarks.
• Extend the Sidecar Project Compiler from one FastAPI archetype to additional project types.
• Demonstrate richer multi-turn delta workflows with rollback and cache reuse.
• Expand the adversarial corpus for authority conflicts, negation, path attacks, schema drift, and partial reconstruction.
• Add a clearer developer-facing interface and visual evidence for compression, admission, reconstruction, and receipts.
• Continue evolving the deterministic compiler toward a safe, stateful workflow.

Our long-term goal is simple: send less to the model, preserve what controls the work, and rebuild more locally—with evidence.

Built With

• api
• codex
• fastapi
• github
• gpt-5.6
• json
• openai
• powershell
• python
• schema
• sha256
• sol
• windows