InferQueue

InferQueue is an adaptive LLM inference scheduler for constrained local GPUs. Predicts how expensive each request will be, then uses estimate to reduce blocking while preventing requests from starving