Fleet Yield Middleware (Zero.xyz, Pomerium, Nexla)

GPU Fleet
Monitoring Loop
Proof

Inspiration

Every public account of frontier-scale training reads like a war diary about hardware. Meta's Llama 3 Herd of Models (2024) reported 419 unexpected interruptions during a 54-day run on 16,384 H100s — roughly 78% traced to hardware, with GPUs the single largest cause — pinning effective goodput at only ~90%. The Shanghai AI Lab Acme characterization (Hu et al., NSDI'24) found that hardware faults dominate long-running-job failures, precisely because a single degraded accelerator stalls an entire tensor-parallel gang; MegaScale (Jiang et al., NSDI'24) devotes much of its engineering to detecting and recovering from exactly this.

What struck us is a structural gap in the stack. Two mature ecosystems exist and never talk to each other: health telemetry (NVIDIA DCGM, Node Problem Detector) that can only detect and binary-quarantine, and cluster schedulers (Kubernetes, KAI, Volcano, Slurm) that are excellent at packing and gang-scheduling but reliability-blind by construction. Nobody decides what to do with a GPU that is marginal but not dead. With the hackathon centered on Loop Engineering, we saw the missing piece: a closed-loop middleware that treats reliability as a pricing signal for placement.

What it does

FleetYield is reliability-aware allocation middleware that sits in front of an unmodified scheduler and turns latent failure risk into placement economics. It runs a seven-stage agentic loop —

$$\textbf{sense} \rightarrow \textbf{rank risk} \rightarrow \textbf{profile} \rightarrow \textbf{price} \rightarrow \textbf{route} \rightarrow \textbf{settle} \rightarrow \textbf{gate} \;\circlearrowleft\; \textbf{learn}$$

— that continuously (1) reclaims "yellow" hardware most systems quarantine outright, by matching it to cheap, checkpointable work; (2) reserves premium GPUs so decode-inference never squats on an H100 a training gang needs; (3) pre-emptively drains degrading GPUs before failure (the one human-authorized step); and (4) rehabilitates GPUs autonomously through canary promotion — no manual tickets.

Crucially, FleetYield never places pods itself. It exposes decisions to the real scheduler as node taints/labels (or a Scheduler-Framework Filter/Score plugin), so it composes with whatever you already run.

Tool kit:

• Nexla : Sense layer: ingests and normalizes the GPU health/telemetry stream (DCGM/Xid-style signals) that feeds the risk ranker. • Pomerium: Gate layer: authorizes the one destructive action (pre-emptive GPU drain) — the single human-in-the-loop step, everything else is autonomous. • Zero.xyz: Gate/agent layer: unblocks the agent so the loop can act on its decisions without manual friction.

How we built it

The mechanism is a deterministic, seed-reproducible simulator in Python (FastAPI + Pydantic), modeling a fleet with latent health, synthetic fault injection (NVLink degradation, HBM capacity loss, and "quiet" compute corruption that surfaces no numeric precursor), and the full loop: risk ranking, safe-operating-envelope derivation, dynamic shadow pricing, proportional-fairness routing, outcome settlement, and a rehabilitation ladder. Workloads are typed with SOTA serving/training nomenclature and routed with cited rationale — decode is memory-bandwidth-bound (Splitwise ISCA'24, DistServe OSDI'24, vLLM SOSP'23), distributed pretraining is NVLink-bound (Megatron-LM), fine-tuning is HBM-capacity-bound (ZeRO).

To make the claim credible rather than self-referential, we ran the real thing: a live Kubernetes cluster (kind on colima) running the actual NVIDIA/Run:ai KAI Scheduler v0.12.16 installed via Helm, with 24 GPUs modeled as fake nvidia.com/gpu extended resources across six NVLink-domain nodes. We then executed a controlled A/B: bare KAI versus FleetYield + KAI, where FleetYield contributes only taints/labels in front of the untouched scheduler. The front-end is a vanilla JS/SVG mission-control dashboard framed as a three-act narrative: live fleet floor → agentic loop matcher → the measured KAI proof, with each sponsor tool mapped onto its loop stage.

Challenges we ran into

The metric nearly buried the result. Our first north-star (gross useful GPU-hours) made FleetYield look worse, because reliability-blind schedulers accumulate hours on large jobs that subsequently crash. Only after redefining the objective as goodput net of wasted progress did the true advantage become measurable — a reminder that in mechanism design, the metric is the product.
Real infrastructure fought back. On the cluster we hit kube-proxy CrashLooping with too many open files (resolved by raising the colima VM's inotify limits), KAI's admission webhook rejecting pods for a missing nvidia RuntimeClass (aliased to runc), and a Helm pre-install hook timing out on the very networking failure kube-proxy was causing.
Intellectual honesty over flash. We had reimplemented Tiresias/Gandiva placement heuristics, but Gandiva has no public code and those weren't the upstream systems — so we removed the "we beat Tiresias" framing entirely and let the real KAI A/B carry the argument.

Accomplishments that we're proud of

A measured lift on a real, current scheduler, not a strawman: on the live cluster, adding FleetYield in front of the same KAI scheduler produced +45.8% net goodput, failures 3 → 0, and premium-GPU misuse 32 → 0 GPU-s — with zero changes to the scheduler. Conservatively (reclaiming half the ~10% Llama-3 hardware loss), that maps to ~\$1–2M/year per 1,000 H100s.
A genuinely closed, self-healing loop — detection → drain → resettlement → canary rehabilitation → promotion — that improves fleet productivity unattended, with a human required at exactly one gate.
An honest, defensible story: we can state precisely what is measured (real KAI placements) versus simulated (fault injection), which is rare — and disarming — for a hackathon demo.

What we learned

You layer on the scheduler; you don't replace it. The durable positioning is FleetYield + your scheduler, delivered as a Filter/Score plugin or taint controller — this is both more honest and more adoptable.
DCGM is our sensor, not our competitor. The defensible wedge is reliability economics (continuous risk → value-aware placement + graduated trust), not telemetry we don't own.
Reliability is a scheduling primitive Treating a GPU's failure risk as a first-class price — the same way schedulers price capacity — reclaims the entire "yellow" middle of the fleet that today is wasted or blindly overloaded.