Headroom

An LLM gateway and control plane — and the measurement almost nobody publishes: at the default 0.90 threshold, a semantic cache answered 98 novel questions and 92 of them were wrong.

Comment

Everyone ships semantic caching. Here is how often it silently lies, measured.

At the industry-default similarity threshold of 0.90, a semantic cache answered 98 of 130 never-before-seen questions from a neighbouring entry — and 92 of those answers were provably wrong. On the same corpus at the same threshold it served 389 of 390 genuine paraphrases, 382 of them correctly. Both halves are the finding.

Inspiration

Every team putting an LLM in front of users eventually builds the same layer: virtual keys, per-tenant budgets, rate limits, a cache, a fallback provider, and a bill nobody can attribute. Most build it badly, in a hurry, inside their product. I wanted to build it properly and in public.

Then a second idea took over. Semantic caching is recommended in every "production LLM" writeup and measured in almost none of them — because measuring it needs a large question set with exact ground truth, and nobody has one lying around. I did: my previous project, Backline, generates a synthetic world where every eval question has an exact answer key. So Headroom got built, and then pointed at its own most-recommended feature.

What it does

Headroom sits between applications and model providers — Anthropic-dialect and OpenAI-dialect, cloud APIs and self-hosted vLLM — and every request flows through it. One pipeline, two dialects, real SSE streaming passthrough:

authenticate → scope → route → rate limit → cache → budget gate → failover → passthrough → meter

  • Virtual keys per tenant, scopeable to model patterns.
  • Budgets that enforce under concurrency. Reservation-based, on a single atomic conditional write — not a read-then-write that loses races.
  • Token-bucket rate limits on the same atomic primitive, returning a real retry window.
  • Exact + semantic response caching, with an admission decision that is logged, replayable, and refusable.
  • Provider failover with jittered backoff and circuit breakers — invisible to the caller before the first byte, and never spliced after it.
  • Per-request cost attribution priced to the picodollar at the rates it was actually billed at, by tenant, route, and model, with a live Next.js console.

How I built it

Python 3.12 + FastAPI + asyncio, uv, ruff, mypy --strict. Postgres 16 with pgvector for config, keys, the ledger, the request log, and the semantic cache. DynamoDB for the two things that must not be raceable — budget reservations and token buckets — both resolved to single conditional writes. Next.js 15 + TypeScript console on top. Docker Compose for local.

Deployed twice, by hand, both times:

  • ECS Fargate on a two-root Terraform architecture — a durable data layer that survives and a disposable compute layer that doesn't — with RDS/pgvector, DynamoDB, an ALB, CloudWatch alarms, and a nightly rollup Lambda. 17 evidence files committed.
  • EKS via eksctl, a 10-template Helm chart, IRSA, NLB, Tailscale egress to reach a GPU on my desk. Cluster up in 20 minutes, evidenced, torn down.

No secret lives in the repo, in a tfvars file, or in Terraform state.

A cold clone reaches a working demo in one command, with no API key, no GPU, no network, and $0.00 of spend:

git clone https://github.com/sergioavilax/headroom && cd headroom
make demo

It mints a tenant and a virtual key through the public admin API, then asserts 22 claims about your own clone — printing each with its expected value and exiting non-zero if any is false.

Challenges I ran into

The threshold that doesn't exist. I expected to find a safe similarity cutoff and publish it. I defined the rule for what "safe" meant before drawing the curve, replayed the admission decision offline across 0.70 → 0.99, and no such threshold exists anywhere in that range. The closest wrong answer scores 0.999539 — two questions differing by a single period token. The furthest correct one scores 0.889850. The bands overlap. One cosine number cannot separate them, and the honest result was a negative one.

Zero dropped requests took three attempts. A rolling helm upgrade under load read 8331 requests / 1 dropped, then 8326 / 2, then 8342 / 0. The single drop wasn't randomness — it was a keep-alive connection pinned by conntrack to a terminating pod, found by building a loopback-FIN rig to reproduce it in isolation. All three runs are committed. The two failures matter more than the pass.

Two Helm chart bugs at first install, both first-contact: CAP_MKNOD missing for the Tailscale egress pod (CrashLoopBackOff), and AWS_DEFAULT_REGION vs AWS_REGION for botocore (NoRegionError). Neither shows up in Compose.

A test with a calendar bomb in it. Two hard-coded dates went red on their own when UTC advanced past them. Fixed with an injected frozen clock — and it's the kind of bug that only surfaces if you keep running the suite after you stop building.

Accomplishments that I'm proud of

Every number below is recomputed from a committed artifact by tests/test_docs.py on every pull request. A README claim that no longer follows from its evidence turns the suite red.

Claim Measured
Answer quality through the extra hop, 133 questions 93.7 vs 93.3 direct — Δ +0.4 against a pre-registered bound of 3.0
Passthrough overhead, 462 live suite requests p50 0.0612 ms (p95 0.1176, p99 0.1644)
The same column on ECS Fargate behind an ALB 0.0249 ms
The same column on EKS behind an NLB 0.0175 ms
Two independent meters over one $7.54 run $7.541253 vs $7.540398 — the entire $0.000855 residual traced to one identified request
A rolling helm upgrade under load, on EKS 8342 requests, 0 dropped
A GPU killed mid-run, us-east-1 → my desk over Tailscale 92 requests, 92 ok, 0 dropped
Live API spend, entire project ≈ $8.10 against a pre-committed $20 cap

1,449 keyless tests, 0 skipped. mypy --strict. CI that never needs an API key.

What I learned

A control surface that looks excellent on the traffic you tested and poisons the traffic you didn't is not a control surface. That's the semantic cache result, but it generalizes to every threshold-based safety mechanism I've built since. If you only evaluate on the distribution you designed for, you have measured your own assumptions.

Measure the thing everyone recommends. The consensus features are the underexamined ones precisely because agreement suppresses testing.

Pinning documentation claims to artifacts changes how you write them. When a number in your README can turn CI red, you stop writing numbers you can't defend. It's the cheapest honesty mechanism I've found.

The gateway hop is basically free, and I'd been assuming otherwise. Sub-0.03ms behind a load balancer means "should this be Go" is an aesthetic question on this workload, not a performance one — but I only get to say that because I measured it instead of arguing it.

What's next for Headroom

Nothing, and that's deliberate.

Headroom was built to answer a specific question and to be a public, verifiable artifact — not to become a product. It's finished: MIT licensed, fully documented, every claim pinned to evidence, and stable. Fork it, read it, run make demo, take the semantic-cache experiment and point it at your own corpus.

I'd rather ship a closed project that's true than keep an open one that drifts.

Built With

Share this project:

Updates