🥝 K.I.W.I. — Knitted Infrastructure for Weight-sharded Inference

Inspiration

Large language models require GPUs with massive VRAM. A 70B-parameter model needs ~140 GB at FP16. An NVIDIA A100 costs $15,000+. Most researchers and enthusiasts cannot access this hardware.

At the same time, there are roughly 1.5 billion discrete GPUs deployed globally. Most are consumer-grade. Most are idle at any given moment. Four RTX 4090s (24 GB each) collectively hold 96 GB of VRAM. Eight of them hold 192 GB. The compute exists. It is just fragmented, unreliable, and unorchestrated.

The gap: there is no production-grade system that stitches heterogeneous consumer devices into a coherent execution substrate for large model inference. Existing approaches demonstrate feasibility but assume cooperative, stable nodes. Real consumer hardware is neither. Nodes drop mid-inference. Bandwidth fluctuates. VRAM availability changes as users open games or browsers. These systems break under real conditions.

What K.I.W.I. Does

K.I.W.I. is a fault-tolerant, pipeline-parallel LLM gateway that unifies a fleet of heterogeneous machines into a single inference endpoint. To developers, it exposes a standard OpenAI-compatible API (/v1/chat/completions). Under the hood, it orchestrates a distributed swarm of unreliable, underutilized hardware:

Weight-sharded execution: Model layers are sliced across $N$ consumer GPUs or CPU nodes. Each node holds and executes a subset of transformer layers (e.g. Stage A for layers 0–11, Stage B for layers 12–23). No single device needs the full model in memory.
Direct P2P Tensor Forwarding: Intermediate hidden state activations stream directly from Stage A to Stage B over HTTP or libp2p streams. Heavy activation tensors never touch the central gateway.
KV-Cache Active Recompute: If a worker drops mid-generation, the gateway detects the failure, soft-evicts the stale node, selects a new healthy stage pair, and sends the prompt + context generated so far as a fast prefill pass to hot-rebuild KV caches seamlessly.
Dynamic Block & Stage Assignment: Workers boot completely agnostic without static role config. On registration (/register), the scheduler counts active stage coverage and assigns layer ranges based on hardware telemetry (CPU cores, VRAM capacity).
PagedAttention Virtual Block Cache: Fixed-size block pool memory management (PagedCacheManager) eliminates KV-cache fragmentation and prevents Out-of-Memory (OOM) failures under concurrent requests.
Security-Hardened Transport: All inter-node activation payloads use safetensors binary serialization (application/octet-stream), avoiding Python pickle vulnerability vectors (CVE-2025-32434).

Architecture

The system consists of three main components:

┌─────────────────────────────────────────────────────────┐
│                    Gateway (:8080)                      │
│  ┌───────────┐   ┌─────────────────┐   ┌─────────────┐  │
│  │ Tokenizer │   │ WorkerRegistry  │   │ SSE API     │  │
│  │ (AutoTok) │   │ (Stage Balance) │   │ /v1/chat/...│  │
│  └───────────┘   └─────────────────┘   └─────────────┘  │
└────────────────────────────┬────────────────────────────┘
                             │ tiny input_ids
                             ▼
              ┌─────────────────────────────┐
              │   Stage A (Layers 0–11)     │  ← worker_1 / worker_2
              │   Embeddings + Blocks       │
              └──────────────┬──────────────┘
                             │ hidden_states (P2P safetensors + zstd)
                             ▼
              ┌─────────────────────────────┐
              │   Stage B (Layers 12–23)    │  ← worker_3 / worker_4
              │   Blocks + LM Head          │
              └──────────────┬──────────────┘
                             │ next_token_id (JSON)
                             ▼
                      Gateway → SSE → Browser

1. Control Plane (`gateway.py`)

Maintains an in-memory WorkerRegistry with stage coverage balancing and last-seen heartbeat metrics.
Exposes standard OpenAI-compatible chat completion endpoints with Server-Sent Events (SSE) token streaming.
Runs a 5-second background eviction loop: soft-evicts nodes missing heartbeats for >15s and hard-evicts after >120s.
Coordinates KV-Cache Active Recompute during mid-stream node failures.

2. Data Plane (`transport.py`)

Provides an abstract KiwiTransport layer with two implementations:
- HttpTransport: Direct inter-container HTTP POST requests (httpx.AsyncClient).
- P2PTransport: Direct peer-to-peer streams across internet NATs using hivemind.p2p (libp2p).
Implements lossless Zstd compression (KIWI_LOSSLESS=zstd) and optional INT8 / Compressed Sparse Row (CSR) activation quantization to minimize cross-node transfer overhead.

3. Node Agent (`worker.py`)

Boots agnostically and sends hardware capacity (CPU, VRAM) to /register.
Loads assigned Hugging Face transformer model slices (AutoModelForCausalLM).
Executes forward passes, maintains local per-request PagedCache KV entries, and handles automatic 60-second TTL cache eviction.
Sends graceful deregistration signals (/deregister) upon shutdown.

4. Interactive Live Visualizer (`index.html`)

Integrated dark-mode chat interface with a real-time topology visualization.
Highlights active processing node paths in neon green, renders active data stream particles, and flags evicted nodes in red.

Tech Stack

Orchestration & Serving: Python 3.10, FastAPI, Uvicorn, AsyncIO, Pydantic
Machine Learning: PyTorch (CPU/GPU), Hugging Face Transformers (AutoModelForCausalLM, AutoTokenizer)
Transport & Security: safetensors binary serialization, httpx, hivemind.p2p (libp2p streaming)
Compressions & Optimizations: Zstd compression, INT8 activation quantization, CSR sparsification, PagedAttention block pool
Frontend & Visualization: Vanilla JS, HTML5 Canvas, TailwindCSS

Core Technical Challenges & Solutions

1. Heterogeneous Bandwidth & Transfer Stalls

Challenge: Transferring uncompressed activation tensors (e.g. 10+ MB per layer boundary) over consumer internet connections causes severe pipeline stalls.
Solution: Combining binary safetensors packing with Zstd compression reduces activation sizes by ~17.5%. Optional INT8 quantization and CSR sparsification cut transmission overhead further while keeping pipeline stalls minimal.

2. Zero-Migration Fault Tolerance (KV-Cache Active Recompute)

Challenge: Worker dropouts mid-generation corrupt state. Migrating gigabytes of KV-cache memory between consumer nodes across WAN introduces unacceptable latency.
Solution: Instead of state migration protocols, Kiwi uses active recompute. The gateway re-routes the prompt and generated tokens to a healthy replacement pair in a single prefill pass, hot-rebuilding KV caches instantly.

3. Arbitrary Code Execution Prevention

Challenge: Standard Python inter-node transport relying on torch.save / pickle is vulnerable to arbitrary remote code execution (CVE-2025-32434).
Solution: Kiwi uses strict safetensors binary payload encoding over HTTP/libp2p binary streams, ensuring secure zero-pickle tensor deserialization.

4. Memory Fragmentation Under Concurrency

Challenge: Standard dictionary-based KV caches suffer from severe VRAM fragmentation and out-of-memory errors on limited consumer GPUs.
Solution: Kiwi implements PagedCacheManager, a PagedAttention-inspired virtual block pool manager that allocates fixed physical memory blocks to active requests and recycles them via a 60-second TTL background sweeper.

Benchmarking

Test Configuration:

CPU Baseline (Swap Simulation): Model executed on CPU in a VRAM-constrained environment where the model does not fit in GPU memory, forcing heavy system memory/swap paging to simulate large-model execution bottlenecks on un-orchestrated single hardware.

Kiwi P2P Network (unsloth/Llama-3.2-3B-Instruct): Heterogeneous weight-sharded execution across distributed P2P worker nodes, fitting model stage slices within active VRAM/RAM pools to eliminate swap thrashing.

Metric	Baseline (CPU / VRAM Swap)	Kiwi P2P (`Llama-3.2-3B-Instruct`)	Delta / Improvement
Avg Output Length (chars)	641.67	639.00	-2.67 chars
Avg Latency (sec)	51.60s	9.30s	~5.55x Faster
Avg Time to First Token (TTFT)	4.96s	0.55s	~9.02x Faster
Avg Time per Output Token (TPOT)	0.367s	0.069s	~5.32x Faster
Avg Tokens / Second	2.51 tok/s	14.48 tok/s	~5.77x Speedup

📝 Per-Prompt Benchmark Breakdown

1. "Write an essay on AI."

Metric	Baseline (CPU / VRAM Swap)	Kiwi P2P (`Llama-3.2-3B-Instruct`)
Length (chars)	687	679
Latency (sec)	59.66s	6.82s
TTFT (sec)	11.16s	0.62s
TPOT (sec)	0.382s	0.049s
Tokens / Sec	2.15 tok/s	18.78 tok/s

2. "write a story about colors"

Metric	Baseline (CPU / VRAM Swap)	Kiwi P2P (`Llama-3.2-3B-Instruct`)
Length (chars)	628	631
Latency (sec)	46.81s	9.21s
TTFT (sec)	2.22s	0.47s
TPOT (sec)	0.351s	0.069s
Tokens / Sec	2.73 tok/s	13.90 tok/s

3. "what is weather"

Metric	Baseline (CPU / VRAM Swap)	Kiwi P2P (`Llama-3.2-3B-Instruct`)
Length (chars)	610	607
Latency (sec)	48.32s	11.87s
TTFT (sec)	1.51s	0.57s
TPOT (sec)	0.369s	0.089s
Tokens / Sec	2.65 tok/s	10.78 tok/s

Impact

Democratized LLM Inference: Enables high-parameter model inference without expensive enterprise GPU infrastructure.
Unlocks Idle Compute: Harnesses global, underutilized consumer GPU VRAM into a unified inference substrate.
Production-Grade Resilience: Transforms unpredictable consumer hardware into a seamless, self-healing API service.

Future Work

Decentralized DHT Node Discovery: Transitioning registry coordination to a fully decentralized Kademlia DHT.
WebGPU Node Runtimes: Enabling browser-based GPU workers via WebGPU and ONNX Runtime.
Reputation & Verification: Cryptographic spot-checking and redundant computation voting to protect against untrusted worker nodes.