SentryNode

• 

  dashboard SentreNode Nematron isolate agent

• 

  dashboard SentreNode

Inspiration

SentryNode was born on a rainy night in early September while analyzing NASA turbofan jet engines and exploring how mathematical algorithms could predict equipment wear and tear. This sparked a critical question: What if we could apply this approach to GPU clusters, which suffer from aggressive hardware wear that can cost operators thousands of dollars in unexpected downtime? We wanted to build more than just a predictive engine—we wanted to integrate NVIDIA Nemotron to translate complex mathematical variants into human-readable insights, empowering operators to catch failures before the entire system collapses. However, we realized we needed something more: an autonomous agent capable of taking immediate defensive action to protect the system while the GPU awaited human intervention.

What it does

SentryNode is a real-time telemetry monitoring and diagnostic agent for GPU clusters. It tracks critical node health metrics—such as temperature, VRAM usage, PCIe/NVLink error rates, and network latency. When anomalies are detected, it leverages NVIDIA Nemotron (via Nebius Token Factory) to generate clear analytical explanations, and features an autonomous response agent that isolates faulty nodes to prevent cascading failures.

How we built it

• Backend & Core Engine: Developed using Python and FastAPI for fast async telemetry processing and anomaly detection.
• Machine Learning & AI: Implemented an unsupervised anomaly detection pipeline and connected to NVIDIA Nemotron through Nebius Token Factory for natural language diagnostic generation.
• Frontend: Built a lightweight HTML/CSS dashboard for live cluster visualization.

Challenges we ran into

One of our biggest hurdles was finding the right balance for anomaly detection. We initially wanted to use a standard Isolation Forest model, but it didn't quite meet our rigorous telemetry demands. We had to heavily modify and adapt the model's architecture and parameters to match our high performance and accuracy expectations for cluster-wide hardware monitoring.

What we learned

We gained deep insights into predictive maintenance, unsupervised machine learning adaptation, and how to seamlessly bridge raw hardware telemetry with large language model agents to create truly resilient infrastructure.

Built With

• fastapi
• html
• nebius
• nvidia-nemotron
• python
• scikit-learn