Sentinel

Evidence-backed incident response using CockroachDB vector memory and Amazon Bedrock AI

Comment

Sentinel — Evidence-Backed Incident Response

Inspiration

When an infrastructure incident occurs, engineers often have to search through old incidents, logs, runbooks, and previous fixes before deciding what to do. Traditional AI assistants can generate plausible recommendations, but they may not explain why a recommendation was made or whether it is supported by previous incidents.

We built Sentinel to explore a different approach:

What if an AI incident-response agent could remember previous incidents and use that memory as evidence before making a diagnosis?

Sentinel combines CockroachDB's vector capabilities with Amazon Bedrock to create an evidence-backed incident-response workflow.

What Sentinel Does

A user submits an infrastructure incident containing:

  • Incident title
  • Symptoms
  • Severity
  • Service
  • Region

Sentinel then:

  1. Converts the incident into an embedding using Amazon Titan Text Embeddings V2.
  2. Stores the incident and embedding in CockroachDB.
  3. Performs vector similarity search against historical incidents.
  4. Retrieves the most relevant historical incidents.
  5. Provides that historical context to Amazon Nova 2 Lite.
  6. Generates a diagnosis and proposed resolution grounded in the retrieved evidence.
  7. Persists the diagnosis, proposed resolution, and selected historical evidence back into CockroachDB.

The result is not simply an AI-generated answer. It is a diagnosis connected to the historical incidents that influenced it.

Architecture

                    ┌──────────────────────┐
                    │   React / Vite UI    │
                    └──────────┬───────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │ API Gateway / Lambda │
                    │      FastAPI          │
                    └──────────┬───────────┘
                               │
                 ┌─────────────┴─────────────┐
                 │                           │
                 ▼                           ▼
      ┌────────────────────┐       ┌────────────────────┐
      │ Amazon Bedrock     │       │    CockroachDB     │
      │                    │       │                    │
      │ Titan Embeddings   │       │ Incident Memory    │
      │ Nova 2 Lite        │       │ VECTOR(1024)       │
      └─────────┬──────────┘       │ Vector Search      │
                │                  └─────────┬──────────┘
                │                            │
                └────────────┬───────────────┘
                             ▼
                   ┌────────────────────┐
                   │ Evidence-backed    │
                   │ Diagnosis          │
                   │ + Resolution       │
                   └────────────────────┘

CockroachDB as Agent Memory

CockroachDB is the persistent memory layer of Sentinel.

Each incident contains an embedding stored as:

VECTOR(1024)

When a new incident arrives, Sentinel uses vector similarity search with CockroachDB's <-> distance operator to retrieve the closest historical incidents.

For example:

New Incident
     │
     ▼
Generate Embedding
     │
     ▼
CockroachDB Vector Search
     │
     ├── Historical Incident A
     ├── Historical Incident B
     ├── Historical Incident C
     ├── Historical Incident D
     └── Historical Incident E
     │
     ▼
Evidence supplied to diagnosis agent

This allows Sentinel to use past operational experience as agent memory.

The retrieved incidents are also persisted as evidence associated with the new diagnosis, allowing the user to understand where the recommendation came from.

Agentic Workflow

The core runtime workflow is:

Incident
   ↓
Embedding
   ↓
Historical Memory Retrieval
   ↓
Evidence Selection
   ↓
AI Diagnosis
   ↓
Proposed Resolution
   ↓
Persist Diagnosis + Evidence

The important distinction is that the AI does not operate only on the current incident.

It receives relevant historical memory retrieved from CockroachDB before producing its diagnosis.

AWS Services Used

Amazon Bedrock

Sentinel uses Amazon Bedrock for two AI operations:

Amazon Titan Text Embeddings V2

Converts incident descriptions into 1024-dimensional vectors for semantic retrieval.

Amazon Nova 2 Lite

Analyzes the current incident together with retrieved historical evidence and generates:

  • Diagnosis
  • Reasoning
  • Proposed resolution
  • Confidence
  • Evidence references

AWS Lambda

The FastAPI application is deployed as a Python 3.11 Lambda function.

API Gateway

Provides the public HTTP API used by the frontend and external clients.

Amazon S3

AWS SAM uses an S3 deployment bucket to package and deploy the Lambda application.

CockroachDB Tools Used

Distributed Vector Indexing

CockroachDB's vector capabilities are used as the runtime memory system.

Sentinel performs nearest-neighbor searches against historical incident embeddings and retrieves the most relevant operational memories.

CockroachDB Cloud MCP Server

The CockroachDB MCP server can be used by an MCP-capable agent/operator for governed cluster and schema inspection.

The MCP workflow is intentionally separate from the application's transactional runtime path.

The application uses direct SQL/psycopg2 for deterministic incident persistence and vector retrieval, while MCP provides an agent-friendly interface for database inspection and operational investigation.

Example Incident

A user can submit:

{
  "title": "API timeout errors",
  "symptoms": "Users getting 500 errors during peak traffic",
  "severity": "high",
  "service": "payment-api",
  "region": "us-east-1"
}

Sentinel embeds the incident and searches historical memory.

The diagnosis agent can then use those retrieved incidents to produce a response such as:

Diagnosis:
Database connection pooling issue.

Recommended actions:
1. Increase the connection pool.
2. Add retry logic.
3. Monitor database connection saturation with CloudWatch.

The important part is that the recommendation is associated with retrieved historical evidence rather than being generated from the incident description alone.

API

The deployed Sentinel API exposes:

POST /incidents

Example request:

{
  "title": "Database connection failure",
  "symptoms": "Unable to connect to database after deployment",
  "severity": "critical",
  "service": "database-service",
  "region": "us-east-1"
}

The API returns the resulting incident, diagnosis, confidence, proposed resolution, and evidence.

Additional endpoints expose incident details and historical matches.

Why This Matters

A conventional AI incident assistant might produce:

"The database may have a connection pooling problem."

Sentinel aims to produce:

"Database connection pooling is likely responsible."

Evidence:
→ Similar historical incident
→ Previous diagnosis
→ Previous resolution
→ Outcome of the previous resolution

This creates a more transparent incident-response workflow where the operator can inspect the memory behind the recommendation.

Challenges

One of the biggest challenges was connecting a serverless AWS runtime to CockroachDB Cloud securely.

The project required:

  • CockroachDB TLS certificate configuration
  • Secure connection-string handling
  • Lambda deployment through AWS SAM
  • IAM permissions for Bedrock
  • Vector dimension consistency
  • API Gateway and Lambda integration
  • CORS configuration for the frontend
  • Persisting both AI results and evidence

Another challenge was making sure that historical memory was not merely retrieved but actually became part of the diagnosis workflow.

What We Learned

Building Sentinel demonstrated how database memory and generative AI can work together rather than treating the database as simple storage.

The key lesson was:

Retrieval quality directly affects agent quality.

A powerful model can still produce a weak operational recommendation if the context supplied to it is poor.

CockroachDB's vector capabilities provide the persistent semantic memory layer, while Bedrock provides the reasoning and generation layer.

Together they create a workflow where the agent can learn from previously recorded operational experience without requiring the model itself to permanently memorize those incidents.

Try It

API:

https://6m30yzqjq4.execute-api.us-east-1.amazonaws.com

Main endpoint:

POST /incidents

Source code:

https://github.com/sohail9972/Sentinel

The repository contains the backend, frontend, database migrations, deployment configuration, tests, example configuration, and setup instructions.

Built With

Share this project:

Updates

Submission history