Inspiration

Modern IT infrastructure can experience unexpected failures, including high CPU utilization, memory exhaustion, increased latency, service degradation, and deployment-related issues. Detecting these problems is important, but identifying their root causes and recovering services safely is equally challenging.

We wanted to explore how Artificial Intelligence could help automate IT incident management while maintaining strict safety controls. This inspired us to build an autonomous IT reliability system that combines NVIDIA Nemotron's reasoning capabilities with deterministic monitoring, policy enforcement, and independent recovery verification.

Our central idea was simple: AI should help make decisions, but it should never have unrestricted authority to execute recovery operations.

What it does

Autonomous IT Reliability & Incident Recovery System demonstrates the complete infrastructure incident lifecycle:

Detect → Diagnose → Decide → Execute → Verify → Audit

The system:

  • Simulates healthy and faulty infrastructure conditions using metrics, logs, and deployment events.
  • Detects anomalies using configurable, deterministic thresholds.
  • Generates structured incidents containing severity, metrics, logs, and supporting evidence.
  • Uses NVIDIA Nemotron through Nebius Token Factory for AI-assisted root-cause diagnosis and recovery recommendations.
  • Applies an independent policy gate to validate proposed recovery actions against predefined rules and allowlists.
  • Executes approved actions inside a simulated infrastructure environment.
  • Independently verifies whether the service has actually recovered.
  • Maintains an audit trail of incidents, recommendations, policy decisions, recovery actions, and verification results.

The system also includes deterministic fallback mechanisms to reduce dependence on external AI availability.

How we built it

We developed the project using Python, FastAPI, Pydantic, and NVIDIA Nemotron through Nebius Token Factory.

We divided development into three main responsibilities:

  • Monitoring & Detection: Built deterministic telemetry scenarios, configurable thresholds, severity classification, structured incident schemas, and automated tests.
  • AI Diagnosis: Integrated NVIDIA Nemotron to analyze incident information and generate structured root-cause explanations and recommendations.
  • Orchestration, Policy Enforcement & Recovery: Developed the orchestration workflow, policy gate, simulated recovery executor, independent verification, and audit logging.

The backend uses structured data models to communicate between modules. The monitoring engine identifies incidents, the AI component provides recommendations, and the policy layer independently determines whether a proposed action is allowed.

Recovery actions operate on simulated infrastructure rather than real production systems.

Challenges we ran into

One of our biggest challenges was integrating independently developed modules while maintaining consistent data contracts between monitoring, diagnosis, and orchestration.

We also faced the challenge

Built With

  • artificial-intelligence
  • automation
  • fastapi
  • httpx
  • infrastructure
  • machine-learning
  • nebius-token-factory
  • nvidia-nemotron
  • pydantic
  • pytest
  • python
  • rest-api
  • uvicorn
Share this project:

Updates

Submission history