phoenix

• 

Inspiration

What it does

How we built it

Challenges we ran into

Inspiration

Most of the cost of running an LLM is inference, and speculative decoding is one of the main techniques serving teams use to make it faster. But how well it works depends on the models, the prompts, and the hardware, so engineers usually hand-tune it. We wanted to see whether a system could learn that tuning from its own performance data, and whether observability could tell us why it was or wasn't working.

What it does

Phoenix is an LLM inference server that uses speculative decoding and tunes itself.

• A small draft model (Qwen3-0.6B) guesses the next few tokens.
• The large target model (Qwen3-1.7B) checks all of those guesses in one forward pass. It keeps every token it agrees with, and at the first disagreement it writes the correct token and drafting starts again. With greedy decoding, the output is identical to the target running alone; only the speed changes.
• A learned scheduler (an epsilon-greedy bandit) picks how many tokens to draft on each request: 2, 4, 6, 8, or 0 (no drafting at all). Its reward is output tokens per second.
• Sentry records every request as a trace with a linked profile and a structured log, so we can see exactly where the time goes.

How we built it

Speculative decoding. We used Hugging Face Transformers' assisted generation and wrapped its candidate generator to count how many draft tokens were proposed and how many the target accepted. Transformers doesn't report acceptance on its own. We also turned off Transformers' built-in heuristic that adjusts the draft length, so our bandit is the only thing making that choice and its effect can be measured cleanly.

The bandit. Every arm is tried twice first. After that, Phoenix explores with probability

$$\varepsilon_t = \max(0.05,\; 0.3 \cdot 0.97^{t})$$

and otherwise picks the arm with the best running-average reward, updated incrementally:

$$\bar{r}_a \leftarrow \bar{r}_a + \frac{r - \bar{r}_a}{n_a}$$

Fine-tuning the draft on Veri. A draft model only helps if its guesses match the target. We shuffled the MBPP dataset (Mostly Basic Python Problems) with seed 42 and split it into 500 training and 50 held-out test prompts with no overlap. On Veri, Qwen3-1.7B acted as the teacher and answered all 500 training prompts (temperature 0, up to 256 tokens, thinking disabled). We then fine-tuned Qwen3-0.6B on those answers with LoRA (supervised fine-tuning, one epoch, one NVIDIA L4). This is distillation: the draft learns to write the way the target writes. Total Veri cost was about $2 (job 1a524a15352e).

Observability with Sentry. We used three Sentry products beyond error monitoring:

• Tracing: one transaction per request, named by mode (target, fixed_4, bandit), tagged with arm and draft_model, and carrying latency_seconds, output_tokens, draft_accepted, and draft_proposed.
• Profiling: continuous profiling tied to each trace, so every request links to a flame graph of where its time went.
• Logs: a structured phoenix request log for every request and bandit decision, with mode, arm, latency, tokens/sec, and accepted vs. proposed tokens.

Benchmark. Google Colab NVIDIA T4, fp16, batch size 1, one warm-up request excluded, greedy decoding, Transformers 4.57.6. All three modes run on the same held-out prompts in the same order.

How Sentry shaped Phoenix

Our first benchmark showed something we didn't expect: speculative decoding was slower than plain generation on a T4. Fixed four-token drafting ran at 15.0 tokens/sec, versus 21.6 for the target alone. We instrumented Phoenix with Sentry to find out why.

The traces confirmed it request by request: fixed_4 requests averaged 8.48s, against 5.82s for target. The profiles showed where the time went. In fixed_4 requests, the draft model's candidate generation (get_candidates) took 78% of request time, while the target's verification took about 22%. The draft generates its guesses one token at a time, so four guesses cost four separate passes of the 0.6B model. On a T4 at batch size 1, each of those passes costs nearly as much as one pass of the 1.7B target, so guessing cost more than verification saved.

That finding changed the design. We added arm 0 (no draft) so the scheduler could learn to skip speculation when it doesn't pay, and it worked: after its exploration phase, the bandit chose no draft for 8 of the last 10 requests. The other two were random exploration picks. It learned on its own that speculation was a net loss on this hardware.

Results

Final run: 20 held-out MBPP prompts, up to 128 new tokens, bandit arms {0, 2, 4, 6, 8}.

Mode	Mean latency (s)	p50 (s)	p95 (s)	Tokens/sec	Acceptance	Speedup
No draft (target only)	5.82	5.81	5.90	21.98	N/A	1.00x
Fixed draft (4 tokens)	8.48	8.09	10.15	15.10	63.6%	0.69x
Learned bandit	7.40	6.35	11.31	17.29	58.9%	0.79x

Compared with fixed four-token drafting, the learned bandit delivered 14.5% higher throughput and 12.7% lower mean latency. Its overall numbers are pulled down by its first 10 requests, which are forced exploration and include the slow 6- and 8-token arms. Once exploring, it mostly chose plain generation, the fastest option on this hardware.

An earlier run (50 prompts, up to 200 tokens, arms {2, 4, 6, 8}, no arm 0) showed the same pattern: target 21.55 tok/s, fixed four-token draft 14.98 tok/s, bandit 15.72 tok/s. That run is what led us to add arm 0.

Challenges we ran into

• Speculation was slower than the baseline. This was the biggest surprise, and Sentry's profiler is what explained it.
• Our first Sentry setup silently sent nothing. The DSN wasn't being read inside Colab, so traces were dropped with no error. We changed the notebook to fail loudly if the DSN is missing and to confirm when it loads.
• Our teacher deployment timed out. The distillation script sent requests before the Veri deployment finished booting. We fixed it to wait for the deployment to report ready.
• The fine-tuned model didn't upload automatically. The training config used an outdated key for the Hugging Face push, and the job produced a LoRA adapter rather than full weights. We downloaded the adapter and merged it into the base model ourselves.

Accomplishments that we're proud of

• A scheduler that learned, without being told, to turn off speculation when it hurt performance.
• A clear, measured explanation of why: 78% of request time spent in the draft model, found with Sentry profiling.
• A full distillation pipeline (teacher generation, fine-tuning, merging) for about $2.
• Honest benchmarks on held-out prompts, with acceptance measured from the tokens actually proposed and verified.

What we learned

Speculative decoding isn't free. It pays off only when the draft model is much cheaper than the target, and on a small GPU at batch size 1, each forward pass costs about the same no matter the model size, because the GPU is mostly waiting on memory and setup rather than doing math. We also learned that a system that measures itself can find the right answer even when the answer is "don't use the clever trick."

What's next

• Benchmark the fine-tuned MBPP draft under the same protocol and compare acceptance rates.
• Repeat with a larger target (Qwen3-4B or 8B), where each verification pass is expensive enough that cheap guesses pay off.
• Run the bandit longer and condition its choice on prompt length and recent acceptance.
• Batch verification across concurrent requests and test under multi-user load.
• Use Sentry alerts on latency drift to trigger automatic fallback to plain generation. ## Accomplishments that we're proud of

What we learned

What's next for phoenix

Built With

• amazon-web-services
• google-colab
• hugging-face-transformers
• jupyter
• lora
• matplotlib
• numpy
• peft
• python
• pytorch
• qwen
• sentry
• veri