Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
What's next for Sentinel
SENTINEL — Autonomous AI-Powered Data Center Inspection
Inspiration
As artificial intelligence continues to grow, the data centers powering it are becoming increasingly critical. Thousands of servers, networking devices, and cooling systems operate around the clock, and even a small environmental anomaly can escalate into equipment damage, downtime, or costly failures.
Traditional monitoring systems are good at telling operators when something goes wrong, but they often leave the difficult part to humans: determining what happened, why it matters, and what to do next.
We wanted to go beyond dashboards, static thresholds, and basic alerts.
What if a data center could investigate its own problems?
That question inspired SENTINEL, an autonomous AI-powered inspection system that combines real-time telemetry, multimodal vision, and coordinated AI agents to investigate potential equipment issues and recommend actionable next steps.
Our vision is to eventually deploy SENTINEL on a physical rover capable of navigating data-center environments, inspecting equipment, collecting sensor readings, and serving as a mobile intelligence layer for infrastructure operators.
What It Does
SENTINEL transforms traditional monitoring into an intelligent investigation workflow.
Instead of simply displaying an abnormal temperature or humidity reading, SENTINEL can gather the available evidence, coordinate multiple specialized AI agents, analyze equipment photographs, and generate a structured investigation report.
The system includes:
- Real-time monitoring: A live dashboard displays rack telemetry, environmental conditions, and alerts using SpacetimeDB.
- Multi-agent investigation: Six specialized agents collaborate through LangGraph to analyze anomalies.
- Multimodal visual inspection: NVIDIA Nemotron processes equipment photographs to identify visible indicators, connections, labels, and potential concerns.
- Evidence-based reasoning: Agents use structured tools and telemetry context to investigate possible explanations.
- Verification: A dedicated Verification Agent reviews findings and identifies unsupported conclusions.
- Persistent investigation history: Investigation reports and image provenance are stored in SpacetimeDB and synchronized across connected dashboards.
The current software can operate with explicitly simulated sensor telemetry and reference equipment images while the physical rover integration is developed.
How We Built It
NVIDIA Nemotron — The Intelligence Behind SENTINEL
At the core of SENTINEL is NVIDIA Nemotron, which powers our multi-agent reasoning and multimodal inspection.
Rather than using a single AI prompt to generate a diagnosis, we built an agentic workflow where specialized agents investigate different aspects of the same problem.
Our six-agent architecture consists of:
- Supervisor Agent: Coordinates investigations, delegates tasks, and determines the next steps.
- Telemetry Agent: Examines sensor readings, thresholds, and historical evidence.
- Environment Agent: Investigates environmental conditions and potential abnormalities.
- Documentation Agent: Uses available equipment reference information to contextualize findings.
- Vision Agent: Analyzes real equipment photographs through a vision-capable Nemotron model.
- Verification Agent: Reviews evidence and challenges unsupported findings.
We used LangGraph to orchestrate these agents through a shared investigation state. The Supervisor can dynamically select specialists based on the available evidence instead of always executing a fixed sequence.
This architecture allows SENTINEL to approach an investigation as a coordinated reasoning process rather than a simple chatbot interaction.
SpacetimeDB — Real-Time Infrastructure
We used SpacetimeDB as the real-time backend connecting telemetry, alerts, investigation records, and the monitoring dashboard.
SpacetimeDB allows connected clients to receive synchronized updates without requiring us to build a separate polling infrastructure.
It supports the investigation lifecycle by maintaining telemetry observations, rack information, alerts, and completed reports.
We also integrated image metadata and provenance so the system can distinguish uploaded reference photographs from actual rover-camera captures.
React and TypeScript — Mission Control
We developed a React and TypeScript dashboard that acts as SENTINEL's mission-control interface.
Operators can inspect rack conditions, review alerts, upload equipment photographs, initiate AI investigations, and examine the resulting reports.
The dashboard communicates with the agent service, while SpacetimeDB keeps the underlying monitoring and investigation data synchronized.
Vision-Based Inspection
One of our most exciting capabilities is multimodal inspection.
Using NVIDIA Nemotron's vision capabilities, SENTINEL can examine photographs of networking and server equipment and report visible details such as cables, ports, status indicators, and labels.
We implemented image validation, source attribution, and cryptographic hashing to preserve evidence integrity.
We also designed the system to distinguish between what the model can actually observe and what would require additional measurements or human inspection.
For example, a photograph might show an illuminated indicator, but that alone does not prove the cause of an equipment fault.
Technology Stack
- AI: NVIDIA Nemotron, multimodal vision-language inference
- Agent orchestration: LangGraph
- Backend: Node.js, TypeScript, Express
- Real-time database: SpacetimeDB
- Frontend: React, Vite, TypeScript
- Validation: Zod
- Future hardware integration: Raspberry Pi-based inspection rover, camera, and environmental sensors
Challenges We Faced
1. Building a genuinely agentic system
Our first major challenge was designing an architecture where AI agents perform meaningful specialized work rather than simply passing text between one another.
We needed the Supervisor to make useful delegation decisions, specialist agents to access real tools, and the investigation process to remain understandable and bounded.
LangGraph helped us structure this process, but designing reliable routing, shared state, and termination conditions required careful engineering.
2. Preventing unsupported AI conclusions
AI models can generate convincing explanations even when the evidence is incomplete.
This is especially challenging when interpreting sensor anomalies or equipment photographs.
To address this, we incorporated structured outputs, deterministic telemetry tools, evidence provenance, and a dedicated Verification Agent.
Our testing also exposed important limitations in verification routing and report integrity, reinforcing the need for deterministic safeguards alongside model-based reasoning.
3. Integrating multimodal AI
Visual inspection introduced challenges beyond traditional text-based inference.
We had to validate image inputs, preserve their provenance, construct multimodal model requests, and distinguish genuine visual observations from assumptions.
We also learned that vision-language models can misinterpret small LEDs, port positions, and equipment labels, making uncertainty and verification essential.
4. Connecting AI with real-time data
Integrating an asynchronous multi-step AI investigation with a real-time database required careful handling of investigation state, reports, and synchronization.
We wanted operators to see consistent information across multiple dashboard sessions without losing the context behind an investigation.
5. Working around hardware availability
Because our physical rover hardware was not yet available during software development, we designed SENTINEL so the intelligence layer could be developed and tested independently.
We used clearly identified simulated telemetry and real reference photographs to exercise the software pipeline without misrepresenting those inputs as physical rover observations.
This separation also made the system more modular and easier to integrate with hardware later.
What We Learned
Building SENTINEL taught us that agentic AI is much more than connecting several language models together.
We learned how to:
- Design specialized AI agents with distinct responsibilities.
- Use LangGraph to coordinate dynamic, multi-step workflows.
- Integrate NVIDIA Nemotron with structured tools and multimodal inputs.
- Combine deterministic sensor analysis with AI reasoning.
- Build real-time applications with SpacetimeDB.
- Preserve evidence provenance across multiple system components.
- Evaluate AI outputs critically rather than trusting plausible explanations.
- Design software and hardware components that can be developed independently.
One of our biggest takeaways was that a reliable AI system needs to know not only what it can conclude, but also what it cannot conclude from the available evidence.
What's Next
Our long-term vision is to turn SENTINEL into a fully autonomous physical inspection platform.
Our next steps include:
- Physical rover integration: Connecting the Raspberry Pi, camera, and environmental sensors to the existing software.
- Equipment intelligence: Expanding manufacturer documentation and trend-based early-warning capabilities.
- Multi-rover coordination: Exploring how SpacetimeDB could support a fleet of inspection robots.
- Stronger verification: Improving deterministic evidence checks and investigation reliability.
- Predictive maintenance research: Evaluating whether richer historical data can support validated failure forecasting.
- Autonomous inspection: Enabling the rover to navigate equipment areas and collect observations without constant human direction.
Ultimately, we envision a future where data centers do more than monitor themselves.
They investigate, explain, and help operators respond before small problems become major failures.
SENTINEL is our first step toward that future.
Built With
- agentic
- ai
- api
- computervision
- express.js
- javascript
- langchain
- langgraph
- multimodal
- nemotron
- node.js
- nvidia
- react
- spacetimedb
- typescript
- vite
- zod
