DataCenterOS

Training Jobs
GPU Clusters
Real-time Multi Agent System
System Workflow
Advanced Fault Injections GPU Errors
Advanced Fault Injections Cooling
Advanced Fault Injections Node, Storage, and Telemetry

Inspiration

Training a large AI model takes weeks on thousands of GPUs, and hardware failures are inevitable. Meta's Llama 3 report documented 419 unexpected interruptions in 54 days, with 58.7% caused by GPUs.

When a training job fails, everything since its last checkpoint can be lost. A cooling pump failure or overheating GPU gives operators only minutes to decide which jobs to checkpoint, throttle, or migrate. A bad decision can make things worse by moving workloads onto hardware that's already at risk.

We built DataCenterOS to automate those decisions safely and keep recovery running even if the AI managing it crashes.

What it does

DataCenterOS is an autonomous incident-response system for AI data centers. It monitors a simulated 64-GPU cluster, investigates infrastructure failures, and coordinates recovery using multiple AI agents.

When an incident occurs, it:

Correlates alarms: Groups related failures into one incident rather than treating every alert separately.
Investigates with AI agents: Cooling and Workload specialists analyze different parts of the infrastructure, exchange findings, and report to a Recovery Lead powered by NVIDIA Nemotron.
Simulates before acting: Evaluates recovery options 10 minutes into the future, rejecting unsafe migrations before the AI can choose them.
Executes recovery: Checkpoints, throttles, pauses, or migrates workloads while enforcing safety constraints.
Handles cascading failures: Replans when new hardware failures invalidate previous decisions.
Survives crashes: Resumes interrupted workflows without duplicating actions or losing recovery progress.

The dashboard shows agent conversations, GPU telemetry, decisions, and execution status in real time. It also distinguishes recovered workloads from hardware that still needs a technician.

How we built it

Our guiding principle was simple: AI makes decisions, deterministic code enforces safety, and SpacetimeDB remembers everything.

Infrastructure simulation

We built a GPU cluster simulator modeling cooling capacity, rack temperatures, GPU failures, workload progress, checkpointing, and migration.

GPU temperatures respond to incoming air temperature and utilization. Cooling pumps have limited heat-removal capacity, while GPUs throttle at 85°C and shut down at 92°C. Checkpoints take 30 simulated seconds, and migrating a job takes two minutes.

This creates real tradeoffs: saving one job might consume time another job doesn't have.

We also connected a real NVIDIA L40S GPU through NVIDIA Brev, streaming live temperature, power, and utilization into the same telemetry format as our simulated hardware.

Multi-agent recovery with NVIDIA Nemotron

Our system runs three agents through NVIDIA NIM:

Cooling Agent: Investigates pumps, temperatures, and thermal headroom.
Workload Agent: Investigates GPU health, training jobs, and checkpoints.
Recovery Lead: Combines their findings and selects a recovery plan.

The specialists have separate conversation histories and limited access to evidence. Neither has the complete picture, so they exchange questions before the Lead decides.

We enforce a 30-second planning budget, validate agent responses, and provide deterministic fallbacks when AI calls fail.

Importantly, agents cannot directly modify infrastructure. Our code generates recovery candidates, simulates them 10 minutes ahead, and checks their safety. The Lead can only select from those approved options.

Distributed systems and crash recovery

We designed DataCenterOS as a distributed system first and an AI system second. Agents are replaceable workers; SpacetimeDB is the source of truth.

Each workflow has a 15-second ownership lease. When ownership changes, a monotonic fencing token increments, and database reducers reject writes from outdated owners. This prevents a crashed agent that later wakes up from interfering with its replacement.

Every action also uses a deterministic idempotency key. If an agent crashes after requesting a checkpoint, its replacement can retry without executing that checkpoint twice.

The simulation clock and in-flight actions run inside SpacetimeDB. Even if the agent is killed with SIGKILL, committed work continues independently.

A replacement agent also revalidates the previous plan instead of blindly resuming it. In our live test, it rejected an unsafe destination with DEST_GPU_UNHEALTHY and STALE_DECISION_EPOCH after that GPU failed, then replanned using healthy hardware.

SpacetimeDB and live dashboard

SpacetimeDB stores incidents, recovery workflows, action history, and agent conversations.

Our React dashboard subscribes directly to database tables using SQL subscriptions over WebSockets. No polling or separate message bus is needed.

Agent messages, call statuses, timeouts, and decisions are persisted several times per second, allowing the dashboard to show conversations as they happen rather than replaying them afterward.

Version-checked writes protect completed results from being overwritten by progress updates. We also retain the last four planning rounds, including interrupted conversations.

Every conversation shown on the dashboard is committed database state. When the agent crashes, the dashboard keeps updating because it never depended on the agent process.

Testing and results

We built over 800 automated tests covering the agent and dashboard, plus browser and accessibility tests.

Our end-to-end failure scenario injects a cooling pump failure, destroys the GPU selected for migration, and kills the agent mid-recovery.

The system passed all 18 end-to-end checks in two consecutive runs on committed code. SpacetimeDB completed the in-flight action, a replacement agent resumed recovery, and the workload migrated to healthy GPUs without duplicate actions.

The agents also successfully exchanged and consumed each other's findings in both planning rounds, without requiring fallback coverage. This verified that multi-agent collaboration actually occurred rather than just appearing on the dashboard.

Challenges we faced

Getting agents to actually communicate. Initially, the specialists rarely asked each other questions. We made cross-agent questions part of the planning protocol and improved the evidence available to each agent. Our final tests confirmed successful collaboration in both planning rounds without fallback coverage.

Unpredictable model latency. Identical AI requests sometimes completed in under 6 seconds and other times took over 12. We added strict time budgets, specialist coverage, and deterministic fallbacks so slow models couldn't block recovery.

Preventing stale decisions. A valid recovery plan can become unsafe seconds later. We introduced decision epochs, destination revalidation, and fencing tokens to prevent outdated plans or agents from modifying infrastructure.

Testing real crashes. Our first crash test accidentally killed a helper process instead of the agent. We corrected the process management to demonstrate genuine recovery after SIGKILL.

Integrating real hardware. Streaming a real L40S GPU into SpacetimeDB initially broke tests expecting exactly 64 GPUs. We separated external telemetry from simulated cluster membership while preserving a shared schema.

What we learned

The biggest lesson was that autonomous systems need to remain safe even when their AI is wrong, slow, or completely unavailable.

We learned to separate probabilistic reasoning from deterministic execution, persist agent state before displaying it, and use idempotency and ownership control to make crash recovery reliable.

We also learned that multi-agent collaboration needs to be verifiable, not just visible. Our final runs confirmed that specialists genuinely exchanged and used each other's findings across multiple planning rounds, while our system independently enforced safety.

Passing all 18 checks twice gave us confidence in the tested recovery scenario, although broader benchmarking is still needed to compare multi-agent recovery against a single-agent approach.

Why we fit the NMC² track

The NMC² track focuses on computing cloud infrastructure, and one of the toughest challenges in GPU cloud operations is what happens when hardware degrades mid-workload.

DataCenterOS addresses this problem directly:

Protects expensive compute: Instead of just reporting infrastructure failures, it protects unsaved training progress by deciding which jobs to checkpoint, throttle, or migrate.
Enforces safety before execution: Every recovery option is simulated 10 minutes ahead, with thermal and capacity constraints enforced inside SpacetimeDB. Even an operator cannot bypass these checks.
Survives its own failures: Leases, fencing tokens, and idempotent actions let recovery continue after an agent crashes without duplicating operations.
Keeps operators informed: The system distinguishes verified workload recovery from physical hardware issues requiring a technician.
Integrates real infrastructure: NVIDIA Nemotron through NIM powers reasoning, a real NVIDIA L40S streams live telemetry through Brev, and SpacetimeDB maintains the authoritative state.

For a cloud operator, the potential impact is fewer cascading failures, less compute lost to hardware faults, and technicians spending less time triaging alarms and more time resolving issues that actually require physical intervention.

What's next

We want to expand DataCenterOS to larger clusters, integrate more physical hardware telemetry, and test increasingly complex cascading failures.

We also plan to benchmark multi-agent recovery against a single-agent baseline and eventually connect our recovery engine to real workload orchestration systems.

Our long-term goal is to move data center operations from reactive troubleshooting toward autonomous, verifiable recovery, leaving operators to focus on physical repairs and decisions that require human judgment.