Inspiration
Deploying open-weight LLMs often fails on simple arithmetic: where does every gigabyte of GPU HBM go, and how many concurrent users fit before the KV cache runs out? InferGrid makes GPU memory, roofline latency, and multi-pool request routing transparent and verifiable before a cluster is launched.
What it does
- Exact HBM & KV Cache Decomposition: Pick one of eight open-weight models (including NVIDIA Llama-3.1-Nemotron-70B-Instruct-HF), a layout of H100, H200, L40S, or A100 GPUs, a quantization precision, and a concurrency load. KV bytes per token (
2 x layers x KV heads x head dim x bytes) are shown line by line, and each model's parameter count is rebuilt from its architecture config. - Analytical Roofline Latency (TTFT & ITL): Computes prefill time-to-first-token and decode inter-token latency from published HBM bandwidth, dense Tensor Core FLOPs, and NVLink/PCIe all-reduce overhead.
- Pareto Multi-Pool Router & Baseline Replay: Routes each request to the cheapest GPU replica pool that satisfies TTFT, ITL, KV cache headroom, and capability constraints, comparing cost and latency against largest-only, cheapest-first, and round-robin baselines.
- Production Config Generator & Live Probe: Exports ready-to-run
docker-compose.yml, pinnedvllm servescripts, TensorRT-LLM launch flags, and router policy JSON, plus an optional OpenAI-compatible SSE streaming probe for Nebius AI Studio or local vLLM servers.
How we built it
Built in zero-dependency vanilla JavaScript (engine.js, render.js, app.js) running identically in the browser and Node. Verified by 184 automated unit and DOM smoke tests (node --test) covering KV cache math, activation buffers, roofline latency, Pareto routing, and SSE probe parsing.
Links
- Live Workbench: https://aariv.me/infergrid/
- Source Code: https://github.com/draarivpatel-ui/infergrid
Built With
- gpu
- javascript
- llm
- nebius
- nvidia
