KV-Zero: Audio LLM Inference Optimizer
44% of H100 GPU time wasted writing zeros to KV cache. One-line fix in vLLM: torch.zeros→torch.empty. 56.7x faster alloc, 35x real-time audio at 16 users, zero accuracy loss.
44% of H100 GPU time wasted writing zeros to KV cache. One-line fix in vLLM: torch.zeros→torch.empty. 56.7x faster alloc, 35x real-time audio at 16 users, zero accuracy loss.
Log in or sign up for Devpost to join the conversation.