nutsheLLM

nutsheLLM trims bloated LLM context with PariTok, protects exact facts, and validates task quality—backing off automatically when compression goes too far.

Comment

Inspiration

AI agents repeatedly resend source files, logs, tool output, documentation, and conversation history. Every repeated token adds cost and latency, but blindly removing context is dangerous: the one timestamp, error, path, identifier, or measurement that disappears may be the fact the task depends on.

We built nutsheLLM to ask a harder and more useful question than “how many tokens did we remove?”:

How much context can we remove while preserving useful task performance?

That idea became the product metaphor. Excess context is the shell; task-critical meaning is the kernel. Nuto, our acorn optimizer, expresses the contract in one line: compress the shell, guard the kernel, and prove the answer.

What it does

nutsheLLM is a quality-aware context optimization workbench for AI applications and agents. It runs a fair baseline/optimized comparison in which the task model, instructions, and parameters remain identical while the supplied context is the intended variable.

For every run, nutsheLLM:

  1. classifies context as immutable, compressible, or disposable;
  2. extracts and locks exact paths, identifiers, signatures, errors, timestamps, measurements, URLs, stack frames, and quoted literals;
  3. compresses eligible segments through PariTok’s hosted GPU API;
  4. restores any segment that loses an immutable fact;
  5. calls the configured task model with the optimized context;
  6. evaluates the answer with deterministic task-specific checks; and
  7. retries through L2 → L1 → L0 → original when an attempt is not acceptable.

The workbench shows baseline and optimized tokens, PariTok-attributed savings, deterministic pruning, validation evidence, fallback attempts, latency, evaluation overhead, and every context decision. It never labels a custom task “passed” unless a reliable validator is available.

How we built it

The frontend is a responsive React and TypeScript workbench. FastAPI owns segment classification, immutable-fact extraction, PariTok integration, task-model execution, quality validation, fallback orchestration, public-demo quotas, and content-free aggregate metrics. A temporary SQLite queue makes runs recoverable. The compiled frontend and API ship as one Docker image on Railway.

PariTok is not a decorative wrapper: it is the compression engine inside the acceptance loop. Its L0, L1, and L2 levels provide the steps in the safety ladder. Exact-fact recall is checked before optimized context can reach the task model, and task quality is checked afterward.

The public deployment uses Gemini through its OpenAI-compatible endpoint as the task model. Cloudflare Turnstile, per-IP and per-session quotas, a global run limit, a spend guard, bounded requests, private run URLs, and server-only secrets protect the public demo. GitHub Actions validates Python, frontend, and Docker builds; CodeQL, secret scanning, Dependabot, and protected main provide repository safeguards.

Challenges we ran into

The main challenge was separating token reduction from trustworthy reduction. Compression ratios are easy to headline; proving that the resulting answer still does its job is much harder.

We had to:

  • compare baseline and optimized calls without changing the model or prompt;
  • preserve exact facts that semantic similarity metrics can overlook;
  • attribute PariTok savings separately from deterministic duplicate removal;
  • handle task-model rate limits and public-demo abuse without leaking keys;
  • make fallback visible instead of silently substituting a safer result; and
  • report an honest zero-savings outcome when the safety contract wins.

The hosted models also made latency and rate limits part of the product design, not just test-environment concerns. That led us to build recoverable jobs, clear failure states, quotas, and an explicit safety ladder.

Accomplishments that we're proud of

  • All three live stress scenarios passed their deterministic validators: 100% (3/3).
  • The final suite achieved 52.2% median input-token savings.
  • Across the suite, provider-reported input fell from 6,880 to 3,997 tokens: 2,883 tokens saved, or 41.9% in aggregate.
  • The tool-output scenario removed 71.58% while preserving the single anomaly in repetitive telemetry.
  • The checkout scenario intentionally records 0% savings rather than replacing a safe result with a stronger marketing number.
  • Compression cannot silently alter extracted critical facts.
  • Every transformation, validation check, and fallback attempt is inspectable.
  • The public app, Apache-2.0 source, redacted benchmark evidence, CI, CodeQL, and reproducible Docker deployment are all live.

What we learned

Compression quality is workload-dependent. A model-level target is not a product-level guarantee, and a token counter cannot establish answer quality. Reliable context optimization needs classification, invariant protection, task-aware evaluation, honest accounting, and a safe escape hatch.

We also learned that transparent failures build more trust than polished averages. The zero-savings checkout result explains nutsheLLM better than another idealized demo would: if the kernel might crack, the system steps back.

What's next for nutsheLLM

Next we would add larger benchmark suites, workload-specific policies, more OpenAI-compatible task providers, stronger evaluation for custom tasks, and exportable optimization traces for agent frameworks and observability platforms. Longer term, nutsheLLM could become a policy layer that teams place between agent context assembly and model invocation to optimize cost without weakening their quality contract.

Built With

Share this project:

Updates

Submission history