LLMBoost

Inspiration

Every few weeks there's a new claim: quantise to 4 bits and inference gets 3× faster, or add a draft model and get 2–3×. Those numbers come from papers measured on data centre hardware. GPTQ used A100 and A6000 GPUs with 175B-parameter models; speculative decoding used T5-XXL on a TPU-v4. We have a single 12 GB RTX 5070. We wanted to know what these techniques actually do on hardware people own, and we noticed that the two ideas are rarely tested together.

What it does

LLMBoost runs the same prompts through four setups and compares them fairly:

Baseline: BF16 model
Quantised: GPTQ W4A16 weights
Speculative: a small draft model or n-gram drafting proposes tokens, and the big model verifies them.
Combined: quantised model + speculative decoding

Every number is labelled Measured, Estimated or Unavailable, with its source. The server refuses to compare runs that aren't equivalent. A speedup is reported with a 95% confidence interval (paired bootstrap over prompts):

$$ \text{speedup} = \frac{\operatorname{median}(\text{latency}{\text{baseline}})} {\operatorname{median}(\text{latency}{\text{optimized}})} $$

We call a result "faster" or "slower" only when the whole interval sits on one side of $1.0\times$.

We also test the theory. Leviathan et al. predict the speculative speedup from the acceptance rate $\alpha$, the number of drafted tokens $\gamma$, and the draft-to-target cost ratio $c$:

$$ S = \frac{1-\alpha^{\gamma+1}}{(1-\alpha)(\gamma c + 1)} $$

LLMBoost measures $\alpha$ from vLLM's counters and $c$ from a draft-only run. It then shows the predicted speedup (labelled Estimated) next to the measured one.

How we built it

Frontend: React + TypeScript + Vite, with Recharts. Every chart has a table alternative.
Backend: Java Spring Boot + PostgreSQL. A database-backed job queue (one GPU means one job at a time) handles persistence, and all cross-run statistics run here: speedups, confidence intervals and the theory check.
ML service: Python FastAPI. It runs vLLM as a supervised child process, times every request from the token stream, and samples GPU memory and temperature through NVML.

Our canonical decode metric is time per output token:

$$ \text{TPOT} = \frac{t_{\text{end-to-end}} - t_{\text{first token}}}{n_{\text{output tokens}} - 1} $$

Each run uses warm-ups, three repetitions in shuffled order, a fixed 256-token budget and greedy decoding. Three workloads test different behaviour: short chat, long summarisation (which suits n-gram drafting), and random text built to make speculation fail.

Challenges we ran into

A brand-new GPU generation. The RTX 5070 is compute capability 12.0, and we couldn't confirm that vLLM's GPTQ kernels support it. We check every mode on the real GPU before relying on it. If GPTQ fails, we report it as unsupported; we never relabel another format as GPTQ.
Streaming lies about latency. Under speculative decoding, one streamed chunk can carry several tokens, so per-chunk timing is misleading. We compute TPOT from the engine's token counts instead.
Caches flatter repeated prompts. Prefix caching would make repetitions look unrealistically fast, so it is off for benchmarks.
Memory numbers hide the effect. vLLM pre-reserves GPU memory for its cache, so peak memory looks the same for every setup. We compare model-weight memory instead.
The cost ratio $c$ needs two runs, which forced the theory check into the backend and added a draft-only reference run.
No GPU inside Docker on our machine, so the ML service runs natively.
24 hours. We agreed on a cut list in advance. We would drop features long before dropping confidence intervals or honest labels.

What we learned

Speculative decoding is a bet on $\alpha$. When the draft is often wrong, or $c$ is high, $S < 1$, and it can slow things down.
GPTQ is weight-only quantisation. It mainly helps memory-bound decoding, not necessarily the prefill phase.
Most honest benchmarking is controlling conditions: same prompts, same token budget, same engine, same GPU. Everything else is noise.

What's next

We'll publish our measured results on the RTX 5070 with their confidence intervals, including any setups that turned out slower. After that come more engines (llama.cpp), concurrency sweeps, and quality checks such as GSM8K accuracy and perplexity.