OutageDNA

Inspiration

Major outages happen every week across some of the most sophisticated engineering organizations in the world. After each incident, teams publish postmortems explaining what failed, why it failed, and what they changed.

But those lessons usually stay trapped inside documents.

Another engineering team may have the exact same failure pattern in its own codebase and never realize it until the same outage happens again.

We built OutageDNA around a simple question:

What if software teams could learn from someone else’s outage before repeating it themselves?

OutageDNA turns real production incidents into reusable reliability intelligence. It studies what caused an outage, extracts the underlying technical failure pattern, checks whether similar risk exists in a codebase, proves the vulnerability with a resilience test, recommends a mitigation, and verifies that the mitigation actually improves the result.

Our goal is to move postmortems from something teams read after failure into something they can use before failure.

What it does

OutageDNA follows a six-stage workflow:

Incident → Failure DNA → Exposure → Resilience Test → Mitigation → Verified

For our working demo, OutageDNA analyzes GitHub’s real August 17, 2026 outage, where client retry behavior amplified traffic while services were already degraded.

1. Discover a real incident

OutageDNA uses Tavily to retrieve the official public post-incident report.

Instead of starting from a manually written summary, the agent works from a real source describing an actual production failure.

2. Extract the Failure DNA

The incident is passed to NVIDIA Nemotron through Nebius Token Factory.

Nemotron extracts the reusable technical mechanism behind the outage, including:

• the trigger
• the failure mechanism
• observable signals
• the recommended resilience test

In the GitHub example, it identifies a retry amplification storm.

3. Find exposure in the codebase

OutageDNA then compares the extracted Failure DNA against vulnerable application code.

Nemotron identifies a 95% mechanism match and points directly to the retry logic responsible for the exposure.

Instead of simply saying that a codebase "may be vulnerable," OutageDNA shows the exact code that creates the risk.

4. Prove the vulnerability

OutageDNA then runs an incident-specific resilience test.

In our demo:

• 100 initial requests
• become 900 upstream requests
• producing 9× request amplification
• resulting in a FAIL

This demonstrates that the same type of mechanism seen in the real outage can be reproduced in the target code.

5. Generate mitigation reasoning

Nemotron then proposes a concrete mitigation: introduce a global retry budget so repeated failures cannot create unbounded amplification.

6. Verify the mitigation

The same failure scenario is executed against a controlled patched implementation using that retry-budget strategy.

The result drops from: 9× amplification → 3× amplification

The resilience test now passes.

This means OutageDNA does not stop at:

"Here is a possible problem."

It tries to answer the more useful engineering question:

"Can we reproduce the risk, mitigate it, and verify the improvement?"

How we built it

OutageDNA is built as a lightweight AI reliability agent with a web interface designed around an evidence-first workflow.

AI and agent reasoning

We use:

• NVIDIA Nemotron 3 Super 120B A12B for incident analysis, Failure DNA extraction, code comparison, and mitigation reasoning
• Nebius Token Factory for model access
• Tavily for live discovery of real-world incident sources

The agent performs a sequence of structured reliability tasks rather than making a single generic LLM call.

Its workflow is:

1. retrieve a real incident source
2. extract the underlying failure mechanism
3. compare that mechanism against source code
4. execute the vulnerable scenario
5. generate mitigation reasoning
6. rerun the same scenario against the patched implementation
7. return the full before-and-after result to the UI

Application stack

The web application is built with:

• Next.js
• TypeScript
• React
• Tailwind CSS
• Vercel

The frontend is intentionally designed like a reliability engineering tool rather than a generic AI chat interface.

It exposes the evidence behind each conclusion:

• original incident source
• extracted Failure DNA
• exact vulnerable file
• relevant source line
• match confidence
• resilience-test metrics
• mitigation strategy
• verified before-and-after results

Reproducible verification

For the web demo, Nemotron generates the mitigation reasoning used by the agent.

We then verify that strategy against a controlled pre-authored patched implementation of the same retry-storm scenario.

We chose this approach deliberately so the demo remains deterministic, reproducible, and easy for judges to inspect while still demonstrating the full reasoning and verification loop.

Challenges we ran into

One of the biggest challenges was deciding how far the AI agent should go automatically.

It would have been easy to let the model generate arbitrary code patches and then present the result as though every patch were automatically safe. We felt that would make the demo less trustworthy.

Instead, we separated AI reasoning from deterministic verification.

Nemotron analyzes the incident, finds the exposure, and recommends the mitigation strategy. The system then tests that strategy against a controlled implementation using the same failure scenario.

Another challenge was making the result feel like an engineering tool rather than an AI-generated dashboard.

We repeatedly removed anything that implied capabilities the prototype did not actually have, including fake integrations, fake run history, dead controls, and misleading progress states.

We wanted every visible claim in OutageDNA to correspond to something the system actually performs.

Finally, translating a real-world postmortem into something testable was difficult. Production incidents are written for humans, while software verification requires structured, executable hypotheses.

Creating the concept of Failure DNA helped bridge that gap.

Accomplishments that we're proud of

The part we are most proud of is that OutageDNA completes the full loop from a real public incident to a verified engineering outcome.

The demo does not stop at summarizing an outage.

It:

• discovers a real GitHub incident through Tavily
• extracts its failure mechanism with NVIDIA Nemotron
• finds the same mechanism in code
• identifies the exact vulnerable line
• reproduces the failure experimentally
• measures 9× amplification
• proposes a concrete mitigation
• verifies the same scenario again
• reduces amplification to 3×
• produces a passing resilience result

We are also proud that the product remains transparent about what is live, what is AI-generated, and what is deliberately controlled for reproducibility.

That distinction matters for a reliability tool.

What we learned

The biggest lesson was that incident postmortems contain far more reusable engineering knowledge than we initially realized.

A postmortem is not just a historical explanation of what happened to one company.

Inside it are patterns:

• retry storms
• dependency saturation
• cascading failures
• bad fallback behavior
• overloaded queues
• configuration mistakes
• cache failures
• timeout amplification
• partial dependency failure

These mechanisms can often appear in completely unrelated software systems.

We also learned that AI becomes much more useful for reliability engineering when it is paired with executable evidence.

A model saying:

"This code looks risky"

is helpful.

But showing:

100 requests → 900 upstream calls → 9× amplification → FAIL

is much more convincing.

That changed how we designed the product.

OutageDNA became less about AI-generated explanations and more about AI-guided verification.

What's next for OutageDNA

Our long-term vision is to build a collective memory for software reliability.

Instead of analyzing one incident at a time, OutageDNA could continuously learn from thousands of public incidents across major technology companies and open-source systems.

Future versions could include:

• automatic monitoring of new public postmortems
• a growing library of reusable Failure DNA patterns
• repository integrations
• automatic pull-request analysis
• CI/CD resilience checks
• codebase-wide exposure scanning
• generated chaos and resilience tests
• organization-specific incident memory
• historical risk tracking
• verification of proposed fixes inside isolated environments

Eventually, an engineering team could ask:

"Has this type of code caused a real outage somewhere before?"

And OutageDNA could answer with the incident, the matching mechanism, the exposed code, the test that proves it, and the mitigation that prevents it.

Our goal is simple: Learn from their outage before it becomes yours.

Built With

• nebius-token-factory
• next.js
• node.js
• nvidia-nemotron
• react
• tailwind-css
• tavily
• typescript
• vercel