FailureDNA X

• 

  FailureDNA X — AI-powered GPU investigation dashboard using NVIDIA Nemotron to analyze simulated GPU failures.

• 

  GPU investigation results showing diagnostic status, three completed tests, simulated cost, and AI-generated diagnosis.

• 

  AI Investigation Timeline showing NVIDIA Nemotron’s diagnostic test selections, reasoning, and simulated GPU measurements.

• 

  GPU diagnostic timeline showing normal memory and network test results, with AI-generated explanations and simulated measurements.

• 

  Inject a hidden GPU fault with one click

• 

  Nemotron suspects overheating and runs the cheapest test first: a 1-unit thermal check

• 

  A memory check finds GPU3 at 97.4% memory, and memory pressure is confirmed for a test cost of 4 out of 18.

• 

  One click hides a GPU fault in a simulated 4-GPU node. The agent sees only the symptom.

Inspiration: GPU infrastructure powers AI training, machine learning, and high-performance computing. However, diagnosing GPU performance problems can be complicated and time-consuming. Engineers often run multiple diagnostic tests before identifying the actual cause.

We wanted to explore a smarter approach: what if an AI agent could investigate GPU failures, select the most relevant diagnostic tests, and explain its decisions? That idea inspired FailureDNA X, an autonomous GPU failure investigation prototype built for the NVIDIA × Nebius Global AI Hackathon.

What it does: FailureDNA X uses NVIDIA Nemotron through Nebius Token Factory to investigate simulated GPU performance issues.

Instead of blindly following a fixed diagnostic checklist, the system uses AI-guided reasoning to choose diagnostic tests, examine evidence, and identify potential root causes. Our platform includes two investigation experiences:

1. GPU Investigation Dashboard Users can run a simulated GPU investigation and view the GPU ID, investigation status, completed tests, diagnostic cost, suspected causes, and a step-by-step investigation timeline.
2. Live Hidden-Fault Investigation The system injects a simulated GPU fault without revealing its identity to the AI agent. NVIDIA Nemotron investigates the symptoms, selects diagnostic tools, and evaluates the resulting evidence. The dashboard visualizes how the probabilities of different root causes change throughout the investigation. At the end, it reveals the hidden fault and compares the AI-guided investigation with a traditional fixed checklist.

How we built it: We developed FailureDNA X using a React frontend and a Python FastAPI backend.

The frontend provides interactive dashboards, diagnostic timelines, charts, and investigation summaries. The backend manages simulated GPU failures, diagnostic tools, investigation workflows, and evidence tracking. We integrated NVIDIA Nemotron through Nebius Token Factory using an OpenAI-compatible API client. The AI agent selects diagnostic actions based on available symptoms and evidence. We also implemented fixed-checklist and information-gain approaches for comparison. Our technology stack includes React, JavaScript, Vite, Tailwind CSS, Python, FastAPI, Pydantic, Pytest, NVIDIA Nemotron, and Nebius Token Factory.

Challenges we ran into: One of our biggest challenges was integrating the frontend and backend after developing different parts of the application in parallel.

We encountered API configuration issues, mismatched backend ports, CORS restrictions, and authentication errors while connecting to Nebius Token Factory. We resolved these problems by aligning the API configurations, updating CORS settings, correcting environment-variable loading, and improving backend error logging. Another challenge was designing an investigation workflow that could select diagnostic tests dynamically while keeping the results understandable and explainable.

Accomplishments that we're proud of: We successfully developed and integrated two GPU investigation dashboards.

Our backend passed 24 automated tests, and our frontend production build completed successfully. We also demonstrated a live NVIDIA Nemotron investigation that correctly identified a simulated memory-pressure fault. In one example, the AI-guided investigation completed five diagnostic tests at a simulated cost of 10 units, compared with six tests and a cost of 18 units for the fixed checklist. That represents approximately 44% lower simulated diagnostic cost in that particular run. This is a demonstration result, not a statistically validated benchmark. We are especially proud that the system explains which tests were selected, why they were chosen, and what evidence supports its diagnosis.

What we learned: This project helped us understand how to build AI-powered investigation workflows that combine language-model reasoning with structured diagnostic tools.

We gained hands-on experience integrating NVIDIA Nemotron, working with Nebius Token Factory, building FastAPI services, developing React dashboards, and coordinating development through GitHub. We also learned the importance of transparent AI decisions, evidence-based diagnosis, and reliable API integration.

What's next for FailureDNA X: Our next goal is to extend FailureDNA X beyond simulation by integrating real GPU telemetry using tools such as NVIDIA DCGM.

We also want to support multi-GPU clusters, historical incident analysis, automated incident reports, and production monitoring integrations. Our long-term vision is to make GPU infrastructure troubleshooting faster, more explainable, and more efficient. FailureDNA X - Investigate smarter. Diagnose with evidence.

Built With

• fastapi
• github
• javascript
• nebius-token-factory
• nvidia-nemotron
• pydantic
• pytest
• python
• react
• tailwind-css
• vite