cover
drill
approved
self test
ask
real servers
user bug
root cause
needs you
checkup
investigate
routing

Inspiration

Meta's Llama 3 report is the clearest picture of the problem I've seen. Training the 405B model on 16,384 H100s hit 419 unexpected interruptions in 54 days, about one every three hours, and most of them were hardware: faulty GPUs and HBM3 memory. Each one means someone reading logs to work out what broke while the whole run waits.

Monitoring already tells you that something broke. I wanted to see how much of the "why, and what do we do now" an agent could take on, without letting it touch anything on its own.

The project started as ClusterMedic. I renamed it after Florence Nightingale, who used a chart (her rose diagram) to show what was really killing soldiers in field hospitals. The logo is that chart, drawn as the 8 GPUs of one server.

What it does

Nightingale watches a GPU fleet and handles incidents the way an on-call engineer would, except that it has to ask before it changes anything.

Every 10 seconds, Nemotron 3 Nano gets a short summary of what changed across all the GPUs and decides whether anything looks wrong. If nothing changed, there's no model call at all. When Nano flags something, Nemotron 3 Ultra investigates with read-only tools: the GPU's history, the server, past incidents on that GPU, the job running there, the rest of the rack. In the demo it traced rising ECC errors to a degrading memory module in 12.9 seconds, using 5 tool calls.

The fix plan can only use actions from a fixed list (drain a server, restart a job from its checkpoint on a spare, reset a GPU, open a repair ticket), and it's tried on a copy of the data center first. Then it waits. I approve each step, it runs it, and afterwards it re-reads the telemetry to check the fix held.

Not every crash is hardware. When a training job reads memory it doesn't own (Xid 31), the right move is to leave the GPUs alone and tell the job's owners which line is wrong. That's what it does.

The same code runs against three kinds of machines: a simulated data center (256 to 4,096 GPUs, so I can break things on purpose), real GPU servers through NVIDIA DCGM and Kubernetes, and my own laptop's RTX 3050.

How we built it

Python and FastAPI on the back end, and a plain HTML and JavaScript dashboard. Nebius Token Factory speaks the OpenAI API, so talking to Nemotron is just the openai client with a different base URL. Ultra investigates through a simple JSON loop: each reply is either "call this tool, because..." or the final diagnosis.

For real servers, Nightingale reads dcgm-exporter or Prometheus plus the Kubernetes API. It can only cordon nodes, evict a job's pods or run nvidia-smi. You can't copy a real cluster to rehearse on, so each step goes through Kubernetes' server-side dry run first. The repo also has a setup script for a Nebius AI Cloud GPU VM with k3s and the NVIDIA GPU Operator.

Challenges we ran into

I don't have a data center. The AI Cloud trial wasn't available during the hackathon, so I wrote a small bridge that serves my laptop GPU's readings in DCGM's exact format. Nightingale reads my laptop the same way it would read a server in a data center.
My idle laptop kept "overheating". Laptop GPUs raise a thermal slowdown flag even at 44°C doing nothing, so Nano kept flagging it and calling the model in bursts. I had to make it ignore that flag when the GPU isn't busy.
Order matters. My first GPU reset failed because the job's pods were still shutting down after eviction, so now it waits until they've actually left.
Ultra is a reasoning model, and sometimes the JSON came back wrapped in prose or only in the reasoning field. The client now pulls the JSON out and asks once more if it can't find any.
Smaller things: pip had installed the CPU-only build of PyTorch, so the GPU self-tests saw no GPU until I reinstalled the CUDA build. And Windows notifications broke on titles longer than 64 characters.

Accomplishments that we're proud of

Watching it go from a drill to a verified fix without being told which GPU I broke.
It knows when not to touch the hardware.
It's cheap to run: a full live session cost about 25 cents, because Ultra only runs when something is actually wrong. Everything I'd spent on Token Factory by the time I recorded the first demo came to $0.79.

What we learned

Most of it was about how GPUs fail. Xid 79 means a GPU fell off the bus, Xid 48 is an uncorrectable memory error, and Xid 31 is usually the program's fault, not the GPU's. I also learned that a small model is enough to decide whether something is worth a big model's time.