ARE - (Autonomous recovery engine)

**Autonomous Recovery Engine — An AI-powered SRE agent that detects failures, diagnoses root causes, safely verifies fixes, and automatically opens recovery PRs.**

Comment

Inspiration

Software failures happen constantly. A broken test, unexpected exception, dependency issue, or configuration error can stop a development pipeline and require an engineer to manually investigate logs, find the root cause, write a fix, test it, and create a Pull Request.

We asked: What if an AI agent could handle this recovery loop autonomously while keeping humans in control?

That question led us to build ARE — Autonomous Recovery Engine, an AI-powered SRE agent designed to detect, diagnose, verify, and recover from software failures.

What it does

ARE monitors GitHub-based software workflows and responds when failures occur.

It:

  • Detects a CI/test failure.
  • Collects the relevant incident and code context.
  • Uses NVIDIA Nemotron to analyze the failure and determine the likely root cause.
  • Generates a potential code fix.
  • Performs an AI-powered risk assessment before execution.
  • Tests the proposed fix inside an isolated Docker sandbox.
  • Retries failed recovery attempts within controlled limits.
  • Creates a GitHub Pull Request when a fix is successfully verified.
  • Keeps a human in control of the final merge decision.

The core loop is:

Detect → Reason → Assess Risk → Verify → Recover

How we built it

ARE is built as an agentic recovery pipeline using:

  • NVIDIA Nemotron via Nebius Token Factory — reasoning, root-cause analysis, and patch generation
  • LangGraph — orchestrating the recovery state machine
  • FastAPI + Python — backend and workflow orchestration
  • GitHub Webhooks + PyGithub — repository and Pull Request integration
  • Docker + Pytest — isolated verification of generated fixes
  • PostgreSQL — incident and recovery state
  • Next.js — Mission Control interface

We designed the system so that an AI-generated patch does not directly modify the production codebase. It must first pass through the risk and verification stages before a recovery PR is created.

Challenges we ran into

The biggest challenge was making the system work with real repositories rather than a hard-coded demo.

We had to maintain repository, branch, commit, file, and incident context across multiple stages of the agent workflow.

Another major challenge was trust and safety. An AI-generated fix can look correct while still introducing another problem. We therefore added risk assessment and Docker-based sandbox verification before allowing a recovery PR to be created.

We also had to handle failed fixes, retries, authentication, GitHub integration, and keeping the recovery state consistent across the application.

Accomplishments that we're proud of

We're proud that ARE goes beyond simply using an LLM to generate code.

We built an end-to-end recovery pipeline connecting:

Real GitHub failures → AI reasoning → risk assessment → isolated execution → automated verification → Pull Request

We are especially proud of the human-in-the-loop recovery model. The system can autonomously investigate and verify a solution, but it does not blindly merge AI-generated code.

We also built a Mission Control interface that makes the recovery process visible instead of hiding the AI's actions behind a single button.

What we learned

Our biggest lesson was that building an autonomous AI engineer is a systems problem, not just an LLM problem.

We learned how to connect AI reasoning to real developer workflows, maintain state across an agentic system, safely execute AI-generated code, evaluate risk, handle failed attempts, and design human approval into an autonomous workflow.

Most importantly, we learned that successful AI automation requires verification and controlled execution, not just generation.

What's next for ARE - (Autonomous Recovery Engine)

The next step is to make ARE a true always-on AI SRE agent.

We want ARE to:

  • Monitor repositories continuously.
  • Learn from previously successful recoveries.
  • Detect recurring failure patterns.
  • Handle more classes of infrastructure and production incidents.
  • Improve its patch-generation and verification strategies over time.
  • Build a persistent recovery knowledge base.
  • Expand from CI failures toward broader production incident recovery.
  • Provide deeper observability, metrics, and recovery analytics.

Our long-term vision is simple:

When software breaks, ARE shouldn't just tell an engineer that something is wrong — it should investigate, prove a safe solution, and prepare the recovery.

Detect. Reason. Verify. Recover.

Built With

Share this project:

Updates

Submission history