Inspiration
AI agents are getting better at reasoning, using tools, calling APIs, and completing multi-step tasks — but production environments are messy. APIs return 429s, databases slow down, MCP servers fail, schemas change, context windows grow, and dependencies disappear.
We realized that most agent demos answer one question:
Can the agent complete the task when everything works?
We wanted to answer a harder one:
Will the agent still work when production starts failing?
Traditional software has SRE, chaos engineering, observability, regression testing, and incident response. Agentic systems need the same reliability discipline, but adapted for systems that reason, call tools, accumulate context, and make decisions dynamically.
That inspired FaultForge Agent SRE — a crash-test and reliability engineering platform for AI agents.
Instead of inventing arbitrary failures, we also wanted FaultForge to learn from problems developers are experiencing in the real world. That led to Reality Replay™, where Tavily provides live failure intelligence and NVIDIA Nemotron converts that evidence into reproducible reliability tests.
Our goal became simple:
Find the failure. Reproduce it. Diagnose it. Repair it. Replay it. Prove the agent got better.
What it does
FaultForge is an SRE platform for AI agents.
It tests whether an agent can survive operational failures such as:
- API rate limits and HTTP 429 storms
- dependency outages
- slow or degraded APIs
- schema changes
- stale or malformed tool responses
- context/token exhaustion
- database failures
- fallback failures
- repeated retry loops
The core workflow is:
Connect / Analyze → Attack → Diagnose → Repair → Replay → Prove
Reality Replay™
FaultForge uses Tavily Search and Extract to discover current reliability problems from documentation, issues, release notes, and other technical sources.
That evidence is passed to NVIDIA Nemotron running through Nebius Token Factory, which interprets the failure and converts it into a bounded reliability scenario.
Instead of using Tavily simply to answer a question, FaultForge uses the live web as a source of executable software tests.
Nemotron Autopsy
After an agent fails, Nemotron analyzes the execution trace and identifies the likely root cause.
It then generates a structured resilience policy that can include:
- retry limits
- exponential backoff
- timeout changes
- circuit-breaker thresholds
- fallback behavior
- schema validation
- freshness checks
- context/token limits
FaultForge applies that generated policy to the repaired runtime.
Ghost Replay
We then replay the same failure against both the original and repaired configurations.
Rather than showing one successful example, FaultForge can run repeated paired trials and compare:
- survival rate
- p50 and p95 latency
- retries
- token usage
- cost per successful task
- recovery behavior
We also run benign regression checks because a reliability fix is not useful if it breaks normal workloads.
The final result is a Reliability Matrix showing what the agent survived before and after the repair.
How we built it
FaultForge is built as a full-stack application with a React + TypeScript + Vite frontend and a FastAPI backend.
NVIDIA Nemotron + Nebius
We use NVIDIA Nemotron models through Nebius Token Factory's OpenAI-compatible API.
Different reasoning workloads can be routed to different Nemotron models:
- Nemotron Lightning for fast classification and triage
- Nemotron Super for root-cause analysis and repair generation
- Nemotron Ultra for harder multi-component reasoning and escalation
Nemotron is not used as a decorative chatbot. It is part of the reliability loop: interpreting failures, diagnosing traces, generating repair policies, and helping compile regression scenarios.
Tavily
Tavily provides FaultForge's live failure intelligence layer.
We use Tavily to search for relevant real-world failures and extract supporting technical evidence. FaultForge then uses that evidence to create Reality Replay scenarios.
This gives each sponsor technology a clear role:
- NVIDIA Nemotron: understands failures and engineers repairs
- Nebius: provides the inference infrastructure for the reasoning layer
- Tavily: grounds the system in current real-world failure intelligence
- FaultForge: executes the reliability loop and proves the result
Reliability engine
We built a controlled fault-execution harness so failures can be reproduced safely without disrupting third-party services.
FaultForge supports failure classes including rate limiting, dependency death, latency degradation, schema mutation, context pressure, and unreliable tool output.
The generated runtime policy is then applied and the same scenario is replayed repeatedly for measurable before/after comparison.
Product experience
We designed FaultForge as a coherent product rather than a collection of developer scripts.
The UI includes:
- Dashboard
- Survival Arena
- Reality Replay
- Reliability Matrix
- Flight Recorder
- Reports
- Integration Settings
- collapsible navigation
- authentication support
- light/dark product styling
We also maintain a deterministic Demo Mode so the product remains demonstrable even if an external API is temporarily unavailable, while Live Mode connects directly to Nebius and Tavily.
Challenges we ran into
One of our biggest challenges was making sure FaultForge was more than a visually impressive simulation.
Early versions could compare a predefined baseline policy with a predefined hardened policy. That looked convincing, but it was not enough. We redesigned the repair flow so the policy generated by Nemotron could become the policy used during replay.
Another challenge was balancing real-world evidence with reproducibility. We wanted Tavily to find current incidents, but we did not want our demo to depend on a third-party service actually being down. Our solution was to use live evidence to define a bounded local failure scenario, then reproduce that behavior safely inside FaultForge.
We also discovered that a simple "break an agent and add a circuit breaker" story was not sufficiently differentiated. Existing agent infrastructure already includes security hardening, retries, and observability. We therefore focused FaultForge specifically on operational reliability:
not "Is my agent safe from a jailbreak?"
but "Will my agent stay alive when its dependencies begin failing?"
Another challenge was making reliability metrics meaningful. A repair can improve failure recovery while accidentally damaging normal behavior, so we added benign regression testing alongside failure survival.
Finally, we had to make the product both technically credible and understandable within a short hackathon demo. That led us to simplify the experience around one clear loop:
Attack → Diagnose → Repair → Replay → Prove.
Accomplishments that we're proud of
We are most proud that FaultForge evolved from a chaos-testing idea into a complete reliability workflow for agentic systems.
Some of the accomplishments we are particularly proud of are:
- Building a complete crash-test, diagnosis, repair, and replay loop
- Using NVIDIA Nemotron as the decision and repair engine rather than simply adding an LLM chat feature
- Creating Reality Replay™, where Tavily turns real-world technical evidence into reliability testing inputs
- Applying generated resilience policies during replay
- Running repeated before/after trials rather than relying on a single successful execution
- Measuring both failure survival and benign-task regression
- Building a Flight Recorder that makes agent failures understandable
- Creating a Reliability Matrix that makes improvement visible to both engineers and non-technical judges
- Supporting both deterministic Demo Mode and live Nebius/Tavily integrations
- Designing the project as a developer product that could eventually become part of CI/CD workflows
The result feels less like a hackathon dashboard and more like the beginning of an SRE platform for AI agents.
What we learned
The biggest lesson we learned is that making agents smarter is only part of the production problem.
Agent systems introduce new reliability challenges because failures can propagate through models, tools, APIs, databases, retrieval systems, and context windows. A tool timeout does not just create a timeout — it can trigger retries, increase token consumption, increase latency, exhaust context, and cause downstream decisions to fail.
We also learned that reliability must be measured, not assumed.
A system should not say:
"The repair looks better."
It should be able to say:
"We reproduced the same failure repeatedly, measured the original behavior, applied the generated repair, and measured the improvement."
We also learned how powerful sponsor technologies become when each one solves an essential part of the architecture instead of being added just to satisfy a requirement.
Tavily became more valuable when we stopped thinking of it as "search for an answer" and started thinking of it as live failure intelligence.
Nemotron became more valuable when we stopped treating it as a chatbot and used it as a reliability engineer.
Nebius Token Factory became the inference infrastructure powering our Nemotron reasoning workloads, while giving FaultForge a clear path toward larger-scale evaluation on Nebius infrastructure.
What's next for FaultForge
The next step is to turn FaultForge from a hackathon reliability lab into a production developer platform.
We plan to add:
- Nebius Serverless Jobs for large-scale parallel replay and evaluation
- deeper NeMo Relay / ATIF / ATOF trajectory ingestion
- first-class LangGraph, MCP, OpenAI-compatible, and custom HTTP agent adapters
- secure isolated sandboxes for executing customer-owned agents
- persistent reliability history using PostgreSQL and object storage
- OpenTelemetry export for existing observability platforms
- automated GitHub pull requests containing generated fixes and regression tests
- CI/CD reliability gates that can block a merge when an agent's reliability score regresses
- team workspaces, authentication, RBAC, and enterprise governance
- a continuously updated Failure Radar built from live ecosystem incidents
Our long-term vision is that every production agent should have a reliability gate just like traditional software has unit tests, integration tests, and deployment checks.
A developer should be able to push a change and automatically receive:
Reliability improved: 94 → 97 — safe to deploy
or:
Reliability regression detected — merge blocked
Everyone is building smarter agents.
FaultForge is building the system that makes those agents reliable enough to trust.
Log in or sign up for Devpost to join the conversation.