Problem and inspiration
Teams choose local LLMs to keep prompts, tool calls, and model outputs inside infrastructure they control. The tradeoff is that they do not automatically get provider-wide workload optimization. Their expensive inference GPU also sits idle between traffic bursts, while a fixed speculative draft may be a poor fit for their actual coding, tool-use, or chat workload. We built SpeedLM to turn that unused local capacity into better inference without changing how clients call vLLM.
Solution
Operators keep the normal OpenAI-compatible vLLM workflow. SpeedLM forwards real streaming requests, keeps completed training traces in the operator-controlled environment, and automatically waits for a confirmed idle window. After five quiet minutes by default it safely drains accepted work, lets vLLM release serving GPU memory, and uses that GPU time to train an isolated EAGLE-3 candidate. A frozen session-disjoint suite compares the candidate with the incumbent, and SpeedLM promotes it only when the configured safeguards pass.
New traffic does not reach a sleeping engine or partially trained artifact. It waits while a preemptible stage is cancelled and verified serving state is restored. The target model still accepts or rejects every proposed token.
Innovation and impact
SpeedLM gives local-AI teams three practical benefits: their traffic stays in their environment, their clients keep the OpenAI-compatible serving API, and idle hardware becomes training capacity instead of waste. Safety does not depend on trusting the new draft: the full target model remains authoritative for every token. An archived gate measured accepted length rising from 2.3051 to 2.6507 on 287 session-disjoint contexts over eight repeats—15.0% more accepted tokens per verifier step—but rejected promotion because its wall-clock channel was non-stationary. A separate retained gate promoted a +13.4% accepted-length gain; neither result is a universal speed claim.
What we learned and next steps
The most difficult work was lifecycle safety and leakage-resistant evaluation, not launching a trainer. Next steps are bounded admission waits, broader hardware validation, a pinned CUDA container, additional trainable profiles, and long-running promotion/rollback testing.
SpeedLM is a research preview live-tested on one pinned H100 stack; idle tuning requires a supported, compatible trainable profile and warm-start draft, and a listed profile is not proof that every path has been validated.
Log in or sign up for Devpost to join the conversation.