Inspiration

Modern software systems are becoming increasingly complex, and when something goes wrong, engineers often have to investigate multiple sources of information—logs, metrics, traces, deployment history, and infrastructure health—to identify the root cause.

This process is time-consuming, especially during incidents when every minute matters.

We wanted to explore a different approach: What if an AI agent could act as an autonomous reliability engineer, investigating incidents, gathering evidence, identifying root causes, and safely coordinating remediation?

That idea led to Sentinel: an AI-powered reliability engineer designed to reduce the time between detecting an incident and restoring a system to a healthy state.

What it does

Sentinel is an AI-powered incident investigation and response system that helps identify, diagnose, and resolve application reliability issues.

Its workflow includes:

  • Incident detection: Identifies abnormal system behavior using monitoring signals.
  • Autonomous investigation: Uses an AI agent to dynamically select investigation tools and gather relevant information.
  • Evidence-based diagnosis: Correlates logs, metrics, service health, and configuration changes to identify likely root causes.
  • Risk-aware remediation: Evaluates proposed actions using a policy engine before allowing execution.
  • Human-in-the-loop approval: Requires human approval for actions that carry additional risk.
  • Automated recovery verification: Checks fresh telemetry after remediation to determine whether the system has actually recovered.
  • Incident history: Records investigation steps, evidence, decisions, actions, and outcomes for future reference.

Sentinel is designed around a key principle: AI should not just explain what went wrong; it should help safely restore the system.

How we built it

We built Sentinel using a modular architecture that separates incident detection, investigation, decision-making, policy enforcement, and remediation.

Backend and infrastructure

  • FastAPI for backend APIs.
  • PostgreSQL for persistent incident and investigation data.
  • SQLAlchemy and Alembic for database management.
  • Redis for supporting caching and coordination where needed.

AI agent

  • NVIDIA Nemotron through Nebius Token Factory for AI-powered investigation.
  • Tool-calling architecture that allows the agent to choose relevant investigation tools dynamically.
  • Structured evidence collection to ground diagnoses in actual observations.

Safety and remediation

  • A policy engine that evaluates remediation actions based on risk, confidence, and evidence.
  • An allowlisted action registry instead of unrestricted command execution.
  • Human approval for actions requiring additional authorization.
  • Post-remediation verification before an incident can be considered resolved.

Frontend

  • React, TypeScript, and Vite.
  • An incident-focused interface for viewing investigations, evidence, tool executions, remediation approvals, and recovery status.
  • Server-Sent Events (SSE) for real-time workflow updates.

Development and deployment

  • Docker and Docker Compose for running the application and its dependencies.
  • A deterministic simulation environment for reproducible incident testing.

The architecture is designed to support real monitoring integrations, allowing Sentinel to investigate telemetry from monitored applications rather than depending exclusively on simulated data.

Challenges we ran into

One of our biggest challenges was designing an AI agent that could investigate incidents rather than simply generate explanations.

We needed the agent to decide which tools to call, understand their results, correlate different observations, and produce a diagnosis supported by evidence.

Another major challenge was safety. Giving an AI agent the ability to execute remediation actions introduces serious risks. We addressed this by separating AI decision-making from policy enforcement and execution. The agent can recommend actions, but the backend independently validates them against predefined rules.

We also had to think carefully about recovery verification. Executing a remediation does not necessarily mean an incident is resolved. Sentinel needs fresh observations to confirm that the system has returned to a healthy state.

Finally, designing a modular architecture that supports both deterministic simulation and real monitoring environments required careful separation of telemetry providers, investigation tools, and remediation logic.

Accomplishments that we're proud of

  • Built an end-to-end incident response architecture rather than a simple AI-powered monitoring chatbot.
  • Implemented a tool-calling agent architecture for investigation.
  • Designed an evidence-tracking system to connect diagnoses with collected observations.
  • Built a risk-aware remediation workflow with human approval.
  • Implemented post-remediation verification so successful execution is not automatically treated as successful recovery.
  • Created a deterministic simulation environment for reproducing and testing incidents.
  • Built a real-time incident investigation interface.
  • Designed the system with modular telemetry providers and controlled remediation interfaces, making future integrations possible.

We are particularly proud of the separation between AI reasoning and deterministic safety enforcement. It allows Sentinel to use AI for investigation and decision-making without giving the model unrestricted control over the infrastructure.

What we learned

We learned that building an autonomous reliability engineer involves much more than connecting an LLM to monitoring data.

An effective system needs reliable telemetry, well-designed investigation tools, structured evidence, bounded agent execution, deterministic policy enforcement, and independent recovery verification.

We also learned that AI-generated diagnoses must be treated as hypotheses until supported by actual observations.

Most importantly, autonomy and safety should be designed together. A useful reliability agent should not only know what action might fix an incident; it should also understand when that action is unsafe, when human approval is necessary, and how to verify the outcome.

What's next for Sentinel

Our next goal is to move Sentinel closer to real-world infrastructure monitoring.

We plan to:

  • Integrate Prometheus for real application and infrastructure metrics.
  • Integrate Grafana Loki for centralized application log collection.
  • Expand telemetry support through OpenTelemetry and distributed tracing.
  • Strengthen Nemotron-powered dynamic investigation and evidence correlation.
  • Add more incident scenarios, including database failures, worker failures, resource exhaustion, and service degradation.
  • Introduce more sophisticated remediation strategies with preconditions, rollback mechanisms, and verification.
  • Support integrations with deployment platforms and CI/CD systems.
  • Explore historical incident analysis to help identify recurring failures and improve future investigations.

Our long-term vision is to make Sentinel an AI reliability engineer that works alongside development and infrastructure teams: continuously monitoring systems, investigating incidents, recommending safe actions, and eventually resolving well-understood problems autonomously under clearly defined policies.

Sentinel aims to move incident response from reactive troubleshooting toward evidence-driven, safe, and increasingly autonomous reliability engineering.

Built With

Share this project:

Updates

Submission history