RESCUE AI

RESCUE is an AI Production Engineer that connects incident evidence, remembers past failures, validates fixes, and safely recovers production with human approval.

Comment

Inspiration

Production incidents rarely have a single obvious cause. An API changes, a deployment happens, an error rate increases, and engineers have to manually connect code, logs, APIs, tests, and previous incidents to understand what went wrong.

We wanted to build an AI engineer that could do more than simply alert developers. RESCUE was inspired by the idea of an AI Production Engineer that can connect the evidence, understand the failure, propose a fix, validate it, and keep a human in control before any action is taken.

What it does

RESCUE continuously analyzes production signals and connects them with operational knowledge.

When an incident occurs, RESCUE:

  • Detects abnormal API or service behavior.
  • Retrieves relevant code, API contracts, runbooks, tests, configurations, and incident evidence using Moss.
  • Identifies the likely root cause and affected services.
  • Determines the blast radius of the change.
  • Searches its persistent memory for similar historical incidents.
  • Generates a minimal proposed code/configuration fix.
  • Validates the proposed fix with automated tests.
  • Shows the evidence and reasoning to the engineer.
  • Requires human approval before applying the fix.
  • Simulates staging deployment and verifies recovery.

For example, when an API changes customer_id to customerId, RESCUE can connect that breaking contract change to affected services, identify the outdated API client, propose the migration, validate the change, and guide the engineer through approval and recovery.

How we built it

RESCUE is built as a full-stack application using:

  • .NET 10 / ASP.NET Core for the backend.
  • Clean Architecture for separation of domain, application, infrastructure, and API layers.
  • React 19 + TypeScript + Vite for the command-center UI.
  • SQLite + EF Core for persistent incident memory and application data.
  • SignalR for real-time incident and recovery updates.
  • Moss as the low-latency retrieval layer for operational evidence.
  • A local retrieval fallback for reliable demo and development operation when cloud retrieval is unavailable.
  • Roslyn-based analysis and automated validation for safer proposed fixes.

The system is designed around a human-in-the-loop workflow rather than allowing an AI agent to make uncontrolled production changes.

Challenges we ran into

The biggest challenge was designing an AI workflow that is useful during an incident without giving an AI unrestricted control over production systems.

We had to make the system explainable: every proposed action should be backed by evidence rather than simply generated by an LLM.

Another challenge was combining low-latency retrieval with persistent incident memory. Retrieval answers the question "What evidence is relevant now?", while memory answers "Have we seen and solved something like this before?"

We also had to handle unreliable or unavailable external services gracefully, which led us to implement a fallback retrieval path while keeping the architecture ready for the cloud retrieval layer.

Accomplishments that we're proud of

We are proud that RESCUE demonstrates a complete incident-to-recovery workflow rather than being just an AI chatbot.

The system can connect:

API change → affected code → production failure → historical incident → root cause → proposed fix → validation → human approval → staging verification.

We also built a persistent incident memory system, real-time command-center experience, evidence chain, approval gate, automated validation, and low-latency retrieval pipeline.

Most importantly, RESCUE is designed with safety in mind. The AI recommends and prepares changes, but a human remains responsible for approving the action.

What we learned

We learned that building a reliable AI engineering system is not only about generating a good answer.

The AI needs access to the right evidence, the ability to connect multiple signals, persistent memory of previous outcomes, validation before action, and clear boundaries around what it is allowed to do.

We also learned that latency matters in developer workflows. Fast retrieval makes it practical to use AI during an active production investigation rather than waiting for a long analysis cycle.

Most importantly, we learned that human approval, evidence, and validation are essential when AI is working around production systems.

What's next for RESCUE AI

Our next goal is to move RESCUE from a powerful incident simulator into a production-ready AI reliability platform.

We plan to add:

  • Real integrations with GitHub, CI/CD, monitoring, and observability platforms.
  • Real-time production telemetry ingestion.
  • Automated API contract and breaking-change detection.
  • Deeper dependency and blast-radius analysis.
  • More sophisticated incident memory and learning.
  • Automated staging deployments with real verification.
  • GitHub pull request creation with evidence attached.
  • Configurable approval policies for different risk levels.
  • Multi-service and multi-environment support.

The long-term vision is simple:

RESCUE should become the AI Production Engineer that helps developers go from "Something is broken" to "We know why, we know the fix, and we have verified it" as quickly and safely as possible.

Built With

Share this project:

Updates

Submission history