-
-
Login Page
-
Floor Page: Overview of this section of the Floor 1 Block A1 of the Data Center in Spartanburg
-
Floor Page: Shift to Floor 1 Block B2
-
Location Map: Located at the bottom left of the Floor Page
-
Cluster Page: Live placement of assets and assignments in all eight Data Centers
-
Cluster Page: Shows all the services per rack and the remediation runs explaining the fixes
-
Alerts Page: Shows all the alerts sorted out by the Data Center location
-
AI Agent Page: Queries given with real-time data from existing or recent alerts and conditions; User is also changed for user access
-
Shift Handoff Page: Shows the shift reports and handoffs taken place with live telemetric data backing up the status at time of shift
-
Shift Handoff Page: Report can be printed or saved as PDF
-
My Queue Page: Shows if the user is on shift or not, and the existing tasks assigned to the user and the tasks which are open for claiming
-
My Queue Page: User is off shift
-
Log Page: Shows all alerts and events which have transpired through each location and floor
-
Security Page: Intrusion detected via Honeypot mechanism and identification of threat given
Inspiration
Imagine you’re the CEO, and your product goes down in the middle of the night. One DNS failure in a single AWS region did that in October 2025. It hit an estimated 70,000 organizations, cost US companies an estimated $500 to $650 million in direct losses, and took nearly 15 hours to fully recover. fourweekmba hivelocity
Outages like that start as a small warning nobody owned, or a problem lost at shift change. MESA makes sure every warning has an owner, every handoff is accepted, and intruders get caught on their first wrong move, so small problems don’t become expensive ones.
What it does
MESA is a live operations console for a fleet of data centers. Every alert gets an owner, and nothing gets quietly dropped. It cuts the cost of downtime by catching problems earlier, assigning an owner to each one, and leaving a tamper-evident record that shortens post-incident review.
Live telemetry from 8 data centers. GPU and CPU temperatures and power for 288 servers stream in every 10 seconds. It’s real recorded data from the OLCF Summit supercomputer (the ClusterWise dataset), replayed as if it were happening now. Problems open themselves as incidents. If a rack’s hottest GPU stays above 50 °C for 30 seconds, MESA opens an incident. At 55 °C it escalates. It closes only after two minutes back under the limit, so a rack hovering near the line doesn’t spam you. These are fixed rules, not an AI guessing. It notices when data stops. If a data center goes quiet for 30 seconds, that’s an incident too. A dead sensor feed is just as dangerous as a hot rack. A log nobody can quietly edit. Every incident opening, escalation and close is written to a hash-chained event log. The console’s “Verify chain” button recomputes every hash in the browser and points to the exact entry if anything was changed. Reports, not chat. When an incident escalates or closes, an AI agent writes the report. That’s the only thing the database lets the agent do. It can’t acknowledge, close or change anything. A security layer that assumes the attacker is already inside. See below.
Security: catching the intruder on the first wrong move
Attacks cause outages too. Most security tools try to keep attackers out. MESA assumes someone might get in, and makes sure the first wrong move is caught. Because our rules live in the database, so does our security.
Decoy functions no operator ever calls. SpacetimeDB lets any connected client call a reducer unless the module checks who’s asking. We turned that into a tripwire. The module exposes tempting admin functions, like admin_override_alarm and export_operator_tokens, that change nothing real. The moment one is called, the same transaction records the caller’s identity, connection ID, which decoy they hit, and how many times. It then quarantines that identity and writes a critical threat event. The decoys never throw an error, because a rolled-back transaction would erase our own evidence, and to the attacker the call looks like it worked. Containment inside the database. Real reducers check the quarantine table before doing anything. In our scripted four-step intrusion (recon, credential use, privilege grab, pivot), the attacker trips a decoy, then tries a real action and is refused by the database itself, with no human involved. Repeat hits raise a counter and drop severity from critical to high, so a persistent prober doesn’t flood the on-call lead. Least privilege everywhere else. Each of the 8 feeds has its own identity, and the database only accepts a site’s data from the identity that owns it. The AI agent is allow-listed and can only write reports, so even a manipulated agent can’t act. The hash-chained log makes after-the-fact edits detectable. Why a tripwire. Zero-day attacks can’t be matched by signature. A tripwire doesn’t need to know the attack, only that nobody legitimate would ever do that. Honest limits. The honeypot is detection on top of access control, not a replacement for it. Our source is public, so a determined attacker can read which functions are decoys. Role checks are the real barrier, and the honeypot catches blind scanning and reconnaissance.
How we built it
Everything runs on SpacetimeDB. The database is the backend: our TypeScript module holds the tables, and the rules run inside its reducers, in the same transaction as the incoming data. The watchdog that catches silent feeds is a scheduled reducer, so it keeps running even if every browser is closed. Every screen subscribes to the same tables and updates live, with no API layer in between. The honeypot is part of the same module.
Database: SpacetimeDB module on Maincloud Telemetry: a Node replayer on Railway that streams the Summit data as eight separate feeds, each with its own identity Console: React, Vite, Tailwind, shadcn/ui and ReUI, deployed on Vercel. The log keeps a small local snapshot so it loads instantly instead of waiting on the connection Agent reporting: reducers that only an allow-listed agent identity can call, one report per incident status change Security layer: TypeScript decoy reducers, a quarantine table, and a scripted attacker client, all inside the module
Challenges we ran into
No public live data-center telemetry exists. Operators keep it private for security reasons, which makes sense. We pulled one hour of real Summit telemetry over HTTPS (only the columns we needed, out of a 1.6 GB daily file) and replayed it with timestamps shifted to now. The data didn’t have everything we wanted. There’s no inlet temperature, humidity or CPU utilization. We decided not to fake any of it. Utilization is estimated from power draw and labeled “est.”, and anything we don’t have is hidden. Our first rules flapped. One rack sat right around 50 °C and opened and closed four incidents in an hour. Borrowing the stabilization window idea from Kubernetes (only close after two calm minutes) brought it down to one. A reducer that throws erases its own evidence. Our decoys had to be written so they never fail, or the intrusion record would roll back with them. Deploying from a repo we didn’t own. Only the repo owner can connect it to Railway and Vercel, which cost us some time.
Accomplishments that we’re proud of
Real data end to end: 100,674 readings an hour across 288 servers, with zero failed batches in testing. Stopping the stream opened a “feed silent” incident for every site in 34 seconds, and restarting one site closed only that site’s incident. When we edited a past log entry directly in the database to test it, the browser’s Verify button caught it right away. A scripted outsider trips a decoy, gets quarantined by the database, and is refused on their next real action, in about nine seconds and with no human involved. The AI can’t take any action on its own. The rules decide and act, and the AI only writes up what happened.
What we learned
SpacetimeDB changes how you think about a backend. Putting the rules in the database meant no separate service could get out of sync with the data. Fixed thresholds with good hysteresis beat a model for this job. They’re instant, free, and you can explain every decision. “Don’t show numbers you don’t have” is harder than it sounds when the screen design has a box for them. Public source code and security pull against each other. We designed assuming attackers can read our repo.
What’s next
Decoy hits open incidents automatically, so intrusions land in the same queue as a hot rack, with an owner and a deadline A decoy admin login portal and canary credentials, to catch credential stuffing and leaked-credential use Security tables visible only to the lead role, and role checks on every reducer Facility data (cooling power and PUE) from NREL’s open data center dataset Acknowledge and assign buttons, plus escalation to the next person on call A shift handoff that won’t let a shift end until every open incident has been accepted by the next operator Moving workloads off a hot rack in place, instead of spinning up duplicate copies