CacheForge: Context Compactor for Coding Agents

• 

• 

  .

• 

• 

• 

• 

• 

• 

• 

In short

CacheForge is an OpenAI-compatible gateway between a coding agent and Nebius Token Factory. It condenses test logs before they reach NVIDIA Nemotron 3 Nano. The agent changes one setting, its base URL.

• Cost: in all 7 pre-registered Token Factory comparisons, the same budget ran 2.2–4.6× as many agent sessions (cost per session down 55–78%).
• Quality: it doesn't reliably make the agent fix more bugs: one significant win, one significant loss, five ties. My claim is cost.
• Track: Coding and Agentic Engineering.
• Context: no CacheForge session ran out of context (none of 105), and at 32k on a local GPU it took Nemotron 3 Nano from 13 of 15 sessions out of context to 0.
• Nemotron + Token Factory: Nemotron 3 Nano on Token Factory, with the full 262,144-token context and native tool calling, is the agent's model in every Token Factory run.
• Check it yourself, no API key needed: uv run cacheforge condense and uv run cacheforge replay (see Try it). The demo video, under 3 minutes, shows it running.

Inspiration

I run coding agents on NVIDIA Nemotron 3 Nano, on Nebius Token Factory and on a 12 GB RTX 3080 Ti at home. At 32k tokens on my GPU, agents died mid-task: every test run dumped around 40,000 characters of pytest log into the conversation, re-sent on every turn. In one measured run, the agent ran out of context in 13 of 15 sessions.

On Token Factory's 262,144-token window the agent usually survives, but pays for that log again on every turn. Almost all of a coding agent's bill is input: without CacheForge, output was 0.3% of tokens and about 1% of the cost in my Token Factory runs.

What it does

Before a request goes upstream, CacheForge condenses the tool output in it.

It keeps what an agent needs to fix a failing test:

• pass and fail counts for each test file;
• each failing test's name, assertion and error lines, with file and line number;
• the test lines that set up what the assertion checks;
• if the run didn't finish (say, it timed out), a line saying so.

It drops the rest: passing-test lines, noise, and the middle of very long lines.

• It never touches your instructions or your code. System prompts, user messages and the model's own replies pass through byte for byte. So do file reads and searches.
• It's deterministic code, not another model call. Kept lines are copied verbatim, so it can't invent a test result.
• It shows its work. Every response carries headers with the condenser version and the characters before and after.

A real example: one session's first test run was 40,141 characters, and CacheForge sent 9,662 of them (24%).

Try it

git clone https://github.com/adamm285-dev/CacheForge.git
cd CacheForge
uv run cacheforge condense   # condenses a real 40,141-character log; no key
uv run cacheforge replay     # every recorded session, raw vs condensed; no key
uv run cacheforge serve      # live gateway and dashboard; needs NEBIUS_API_KEY

Point any OpenAI-compatible client at http://127.0.0.1:8765/v1. Dashboard: cacheforge.fieldsmith.pro.

Results

Every run was pre-registered (metric, a two-sided paired exact permutation test, and 15 paired sessions) before running. Result files are in the repo.

Cost: 2.2–4.6× more sessions for the same budget, in every Token Factory run

All runs used Nemotron 3 Nano on Token Factory with the full 262,144-token context, at list price ($0.06 input, $0.24 output per million tokens). v2 is the version I'm submitting. Run G tested v2 and v2.1 against the same baseline, so it has two rows.

Run	Task	Condenser	Prompt tokens/session	Cost/session	Sessions per $	Bugs fixed, without → with
v3	standard	v1	−74%	$0.124 → $0.033	3.7×	57 → 57 (p = 1.0)
B	hard	v2	−66%	$0.090 → $0.031	2.9×	29 → 32 (p = 0.77)
C	standard	v2	−79%	$0.120 → $0.026	4.6×	40 → 65 (p = 0.028)
E	hard	v2.1	−56%	$0.104 → $0.047	2.2×	21 → 20 (p = 1.0)
F	standard	v2.1	−66%	$0.122 → $0.042	2.9×	59 → 39 (p = 0.035)
G	standard	v2	−79%	$0.114 → $0.026	4.4×	52 → 60 (p = 0.52)
G	standard	v2.1	−66%	$0.114 → $0.039	2.9×	52 → 58 (p = 0.50)

Standard: 5 independent bugs, 75 per run. Hard: interacting bugs, 75 traps per run. Run G's p-values are raw; after the pre-registered correction for its three comparisons, all three are 1.0.

Also true in every Token Factory run:

• Median time to first token fell 17–33%.
• No CacheForge session ran out of context: none of 105. Three of the 90 baseline sessions did, even with the 262k window.
• In Run C, the largest baseline request was 256,167 tokens; the largest CacheForge request was 28,436.

Quality: one win, one loss, five ties

"Tie" means not statistically significant over 15 pairs. Run C (v2) was a significant win, 65 vs 40; Run G repeated that comparison and got 60 vs 52, not significant. Run F (v2.1) was a significant loss, 39 vs 59. The baseline alone scored 57, 40, 59 and 52 in four runs of the same task. v2 is the Token Factory default; v2.1 is recommended on a local GPU.

With a small context window (local llama.cpp, 32,768 tokens, condenser v1)

Model	Bugs fixed, without → with	Sessions that ran out of context
Nemotron 3 Nano	37 → 58 (p = 0.047)	13 → 0 of 15
Gemma 4 26B	30 → 71 (p = 0.0001)	15 → 0 of 15

How I built it

• Gateway: Python, FastAPI and uvicorn, streaming to Token Factory with httpx. It condenses role: "tool" messages only. If the condenser fails, the request goes through unchanged.
• Tools: a CLI with condense, replay (today's condenser reproduces 3,185 of the 3,239 recorded requests exactly), watch and serve, plus a web dashboard at cacheforge.fieldsmith.pro.
• Benchmark (TensorRoute-Bench): a 45-test Python project with 5 planted bugs, plus a harder 56-test variant whose bugs interact across modules. The agent gets native tool calls and 40 turns at temperature 0.6.
• Protocol: pre-registered runs on a frozen commit, 15 pairs in alternating arm order, raw tool outputs logged. Details in the repo.
• Cost: all the Token Factory sessions in the repo, plus the probe, cost about $14 at list price.

How I used NVIDIA Nemotron and Nebius Token Factory

• Nemotron 3 Nano (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B) is the agent's model in every Token Factory run. It also runs locally through llama.cpp (Unsloth GGUF), with the MoE experts offloaded to the CPU.
• Token Factory is the gateway's upstream: native tool calling, chat_template_kwargs to turn reasoning off in both arms, and usage reporting (including cached tokens) for the cost analysis.
• Speed: on the hard task, a median session took 40–63 s on Token Factory, against 161–331 s on my own GPU, so each 15-pair run took about an hour and $2–3.

Challenges I ran into

My first big result was fake. At 32k, CacheForge appeared to fix 41 bugs vs 7. The benchmark harness itself was condensing the test output, not the gateway. I moved the condenser into the product, pre-registered again and re-ran. The real result was 58 vs 37.

My scoring was wrong three times, including an agent that edited a test file and got credit for it: Run F was first scored 40 (p = 0.0498); replaying every session against the original tests gives 39. That one score is the only one that changed.

My condenser starved the model on the hard task. On interacting bugs, Nemotron fixed 37 traps without CacheForge and only 13 with it (p = 0.004). It had dropped the lines showing which code produced the bad value. I rebuilt it (v2) to keep exactly the lines the assertion depends on. The hard task then scored 39 vs 32 locally (p = 0.20) and 29 vs 32 on Token Factory.

The fix made local runs slower (median local session 229 s → 331 s), because v2 rewrote earlier messages. v2.1 never rewrites history and showed no measurable slowdown (p = 0.41).

My own rule went against me. Before Run F, I wrote down one exception: if v2.1 was significantly worse on the standard task, the default would go back to v2. It was, 39 vs 59, so v2 ships. A pre-registered head-to-head (Run G) then found v2 fixed 60 bugs and v2.1 fixed 58 (p = 0.86). The full trace is in the repo.

My condenser hid a hang. When a test suite hung, the condensed log dropped the timeout message. It's fixed; I don't claim the fix changes results.

Honest limitations

• Quality is mixed: one significant win, one significant loss, five ties.
• The evidence is narrow: one synthetic codebase with two task sets, my own agent harness, one model on Token Factory and two model families locally.
• CacheForge sessions sometimes edit test files. Eight did, across Runs C, F and G; no session without CacheForge did. Scoring uses the original tests, so this can't inflate a score. I don't know why it happens.
• Not a new idea: condensing tool output has prior art. What's mine is a drop-in gateway for Token Factory, and measured findings about when it helps and when it hurts.
• The dashboard's demo walkthrough is simulated. It's labeled in the UI and declared in MOCKS.md.
• AI coding agents wrote much of the code. Their reports overstated results several times, so an adversarial review re-checked every number here against the raw result files.
• About the name: the project started with prefix-cache work, which didn't lower cost on Token Factory. Condensing is what moved the numbers.

Accomplishments that I'm proud of

• On the hackathon platform, every run cut cost per session by 55–78%.
• I found my product's failures, fixed each and re-measured.
• Every number here traces to a committed result file.

What I learned

• Measure outcomes, not tokens. Fewer tokens is only a win if the bugs still get fixed.
• A rule written before the run beats any argument after it.
• Noise is bigger than it looks. Fifteen pairs can only detect large effects.
• Self-reports aren't evidence. An independent full replay caught what "all tests pass" never would.

What's next for CacheForge

• Real agents and public benchmarks, for example through OpenCode.
• Larger samples: telling versions as close as v2 and v2.1 apart takes far more than 15 pairs.
• Adaptive thinking: turn reasoning up only when tests keep failing.
• Data-heavy tool outputs (SQL results, MCP tools), untested so far.
• Per-session savings receipts.

Built With

• fastapi
• httpx
• llama.cpp
• nebius-token-factory
• nvidia-nemotron
• pytest
• python
• uvicorn