At 2 AM a chiller fails and nobody notices. Ten minutes later a whole row of servers is overheating, and that's when the alarm finally goes off. FaultZero catches that chain the moment it starts, tests a fix in a digital twin, and lets the operator approve it. In our tests it warned a median 12 minutes before the facility's own alarms.
Try it live at faultzero.us, or break it from your phone at faultzero.us/break.
Inspiration
Every app we use runs on a data center that isn't allowed to stop. While reading about outages, we noticed that most of them aren't a single failure. They're a chain: a chiller trips, the cooling units start warming up, and then the racks overheat. The alarms only go off once a temperature limit has already been crossed, so the person on shift is always reacting. More than half of operators say their last major outage cost over $100,000. We wanted to give that operator a way to see the next hour the moment something breaks, and a fix they can trust.
What it does
FaultZero is an operations console for a data hall, built around a live digital twin.
When a piece of equipment fails, the twin copies the facility and runs it an hour into the future, which takes about 9 milliseconds. It tells the operator what will fail next, when, and why, in plain words like "Row C passes 32°C in 11 minutes because Chiller 2 tripped." Instead of a single guess, it runs 24 possible futures and gives a probability, like "88% chance a rack overheats this hour, most likely in 8 to 12 minutes." It doesn't need to be told how bad a failure is either. It works that out from the sensor readings.
Then an NVIDIA Nemotron agent investigates the incident with tools and tries fixes inside the twin. We don't take the model's word for anything. Every plan is simulated again before the operator sees it, so the numbers on the card (12 racks at risk becomes 0, $10,200 at risk becomes $0) come from the twin, not the model. Nothing happens until a person clicks Approve, and approving sends out a work order with a technician, an arrival time and the spare part they need.
It also catches a sensor that's been tampered with to hide a hot rack, lets you scrub through the next hour with or without the fix, writes a handoff for the next shift, and has a QR code so anyone can break the data center from their phone and watch it respond.
How we built it
The backend is Python and FastAPI. We wrote a minute-by-minute model of a data hall with utility power, a generator, two UPS units, four PDUs, three chillers, six cooling units and 24 racks, where failures can be partial or total. The digital twin is a copy of that model stepped forward in time, and the cascade engine walks the power and cooling dependencies to find the root cause of each predicted failure.
On top of that we added a 24-run ensemble for the probabilities, and a calibration step that works backwards from two sensor readings to estimate how badly something has failed. The agent uses NVIDIA Nemotron with tool calling, and if the API is unavailable a scripted planner runs the same tools so the demo never stops.
Every simulated minute of all 41 assets is stored in a Tiger Data (TimescaleDB) hypertable, and clicking a rack shows its history. The frontend is three.js with live power and water flow, updated over Server-Sent Events. The app runs on Render at faultzero.us, a domain we got through MLH and GoDaddy Registry, and 29 automated tests run every time we push.
How we tested it
We didn't want to just claim it works, so we measured it. We ran 200 random failures and made our own twin's job harder on purpose, with different random noise and up to 30% error in how bad it thought each failure was. Of the 141 failures that ended up overheating a rack, a normal threshold alarm warned ahead of time in none of them. A simple trend-line forecast gave a median of 9 minutes of warning. FaultZero gave 12 minutes, and warned at least 10 minutes ahead in 101 of them. It also caught all 15 tampered sensors hiding a hot rack, with no false alarms in 100 normal runs.
We also tested the things we were worried about. When we made the real building 30% worse than the twin assumes, FaultZero still warned a median 15.5 minutes ahead. Over 150 incidents, every time it said something was 90% or more likely, it happened (57 out of 57), and every time it said 10% or less, it didn't (0 out of 72). And over 16 incidents, Nemotron, without being told any strategy, saved every rack in 15 of them and cut the total exposure from $236,170 to $870, matching our hand-written playbook in 15 of 16 cases. The one case where it did worse is exactly why every plan gets checked by the twin first.
Challenges we ran into
Our first agent plans often sounded reasonable and didn't actually work, which is what led to the rule that the twin checks every plan. We also trained a machine-learning forecaster, and it did worse than a straight trend line because it only sees temperatures after they start rising. That pushed us to model the cause instead of the symptom. We worried about what happens if our model of the building is wrong, so we stress-tested it and added calibration, and our tests caught a sign error in that math before it went live. And campus Wi-Fi blocked phones from reaching our laptop, so we deployed the whole thing publicly so judges can break it from anywhere.
Accomplishments that we're proud of
The 12-minute head start is something we measured, not something we're claiming. The probabilities actually hold up. The AI is useful without being in charge. And anyone can pick up their phone and break our data center.
What we learned
Predicting the cause works better than forecasting the symptom. An AI agent is most useful when something checks its work. And being honest about where your project falls short makes it better, not weaker.
How it's different
Tools like Cadence Reality and NVIDIA Omniverse simulate data centers for design and planning, and some systems tune cooling automatically. FaultZero is aimed at a different moment: something has just broken, and a person needs to understand what's coming and decide what to do in seconds.
What's next
We'd like to fit the twin to a real building's sensor data (BACnet or Modbus) using the history we already store in Tiger Data, then add real weather and workload forecasts, send approved work orders straight into maintenance software, and support more than one data hall.
All data in this project is simulated, and dollar figures are illustrative ($1,000 per rack-hour above 27°C).
Built With
- css3
- fastapi
- github-actions
- godaddy-registry
- html5
- javascript
- numpy
- nvidia-nemotron
- postgresql
- pytest
- python
- render
- scikit-learn
- server-sent-events
- three.js
- tiger-data
- timescaledb
- uvicorn