Vitals Incident Room

AI agents and engineers investigate, remediate, and verify production incidents together.

Comment

Inspiration

Vitals started with a simple frustration: when a production incident happens, engineers rarely investigate it in one place. They jump between dashboards, logs, traces, deployment history, alerts, and remediation tools, manually connecting the dots under pressure.

While exploring WebMCP, I saw an opportunity to rethink that workflow.

AI agents are increasingly capable of reasoning about complex engineering problems, but traditionally they have had to interact with web applications through the UI or through completely separate integrations. That creates a gap between what an agent can reason about and what it can actually do.

So I asked:

What if the observability application itself could expose its capabilities directly to an AI agent, while the engineer and agent remained in the same workspace?

That became Vitals Incident Room, an experiment in what an agent-native incident-response workflow could look like.

I wanted to go beyond "add a chatbot to a dashboard." The goal was to make the agent a genuine participant in the incident lifecycle while keeping the human in control of consequential decisions.

What it does

Vitals Incident Room is a WebMCP-powered incident-response workspace where engineers and AI agents investigate, remediate, and verify production incidents together.

For demonstration, a checkout-api service experiences a severe latency and error-rate regression shortly after deployment v2.8.3.

The agent can discover and invoke structured WebMCP tools to:

  • inspect service health,
  • query metrics,
  • compare deployments,
  • inspect distributed traces,
  • search operational logs,
  • analyze the gathered evidence,
  • prepare a remediation,
  • apply an approved remediation,
  • verify recovery.

The investigation can look like:

get_service_health
        ↓
query_metrics
        ↓
compare_deployments
        ↓
inspect_trace
        ↓
search_logs
        ↓
analyze_incident

The agent correlates those signals and identifies the most likely cause: a database query regression introduced in v2.8.3.

It can then prepare a rollback to the last known-good version.

However, the agent cannot authorize its own action.

prepare_remediation
        ↓
HUMAN APPROVAL
        ↓
apply_remediation
        ↓
verify_recovery
        ↓
RESOLVED

The remediation changes the application's simulated system state, and the final verification checks that both latency and error rate have actually returned to acceptable levels.

The result is a shared workflow rather than a chatbot sitting beside an observability dashboard.

How I built it

Vitals Incident Room extends existing Vitals observability work with a focused browser-based Incident Room designed specifically around WebMCP.

The application is built with TypeScript and React, with a deterministic simulated observability environment so the complete workflow can be reproduced without requiring real production credentials or infrastructure.

The architecture separates the WebMCP layer from the application domain:

Architecture

Layer Component Responsibility
Human Engineer Reviews evidence and authorizes consequential actions
Web App Incident Room Presents shared incident state and workflow
Agent Interface WebMCP Tool Surface Exposes structured capabilities to the AI agent
Agent AI Agent Discovers tools, gathers evidence, analyzes the incident, and executes approved actions
Observability Metrics · Logs · Traces · Deployments Provides the signals used for diagnosis
Analysis Evidence Correlation Combines signals into a ranked root-cause hypothesis
Remediation Prepare → Approve → Execute Proposes and safely applies the remediation
Verification Recovery Checks Confirms latency and error-rate recovery before resolution

Core flow:

Engineer → Incident Room → WebMCP → AI Agent → Observability Data → Evidence Analysis → Remediation Proposal → Human Approval → Execution → Verification

The WebMCP layer exposes nine focused tools with explicit schemas and annotations. Read-only investigation capabilities are separated from state-changing operations, and operational logs are explicitly treated as untrusted content.

I also made the demo stateful. A rollback doesn't simply change a label from "failed" to "resolved"; it changes the simulated deployment, latency, error rate, and database-query performance. Verification then reads that new state.

This gives us a deterministic cause-and-effect loop:

v2.8.3
   ↓
latency regression
   ↓
agent investigation
   ↓
rollback to v2.8.2
   ↓
metrics recover
   ↓
incident resolved

Most importantly, I designed the human approval boundary outside the WebMCP action surface. The agent can prepare a remediation, but only the engineer can authorize it.

Challenges I ran into

The biggest challenge was deciding how much of Vitals to expose.

Vitals already covers a broad observability workflow, and my first instinct was to make everything agent-accessible. That quickly became the wrong direction. A large collection of loosely connected tools would demonstrate API coverage, but not why WebMCP fundamentally changes the product.

I narrowed the project to one complete incident-response journey and treated everything else as secondary.

Another challenge was making an agent-driven workflow believable without depending on real production infrastructure. I solved that with deterministic metrics, logs, traces, deployment history, and mutable runtime state. This lets us make the entire experience reproducible while preserving realistic relationships between the signals.

I also had to think carefully about authority and safety. An agent that can investigate an incident and then immediately authorize its own rollback creates a dangerous feedback loop. I separated proposal, approval, execution, and verification so the human remains responsible for the consequential decision.

Finally, WebMCP is an emerging browser capability, so I had to pay close attention to tool schemas, browser support, registration behavior, tool annotations, and the difference between a WebMCP-capable webpage and the agent consuming those tools.

Accomplishments that I'm proud of

I'm most proud that the project evolved beyond a conventional "AI + dashboard" demo.

The agent can actually compose multiple capabilities exposed by the webpage:

Health
  + Metrics
  + Deployment history
  + Traces
  + Logs
        ↓
Evidence-based diagnosis

I'm also proud of the human-in-the-loop safety model. A consequential action cannot be executed until it has been explicitly authorized by the engineer, and recovery is not declared until the post-action state passes verification.

Another accomplishment is that the demo is completely deterministic. I can reset the environment and reproduce the same incident, investigation, remediation, and recovery sequence without relying on external production systems.

Incident Room also deliberately treat operational log data as untrusted content. That distinction matters in an agentic environment because data discovered by a tool should not automatically become an instruction to the agent.

Most importantly, Vitals Incident Room gave us a concrete answer to a question the project started with:

What does an observability application look like when an AI agent is a first-class participant rather than an add-on chatbot?

What I learned

My biggest lesson was that WebMCP is most interesting when it changes the workflow, not when it merely exposes functions.

A collection of tools is easy to demonstrate. Designing a workflow in which an agent can discover, compose, and act on those tools while sharing state with a human is much more interesting.

I also learned that tool design is effectively part of product design. Names, descriptions, schemas, output structures, side-effect annotations, and error behavior all influence how reliably an agent can use an application.

Another important lesson was to separate:

Evidence
   ↓
Analysis
   ↓
Authorization
   ↓
Action
   ↓
Verification

That separation made the application easier to reason about and gave us a much clearer safety model.

Finally, I learned that the best agent experiences are not necessarily fully autonomous. In many engineering workflows, the strongest design is agent speed + human judgment.

What's next for Vitals Incident Room

This challenge version is intentionally focused on one incident-response journey. The next step is to connect that workflow to real observability infrastructure.

Future versions could support real Prometheus metrics, OpenTelemetry traces, production log providers, deployment systems, and runbooks while preserving the same WebMCP interaction model.

I also want to explore richer agent collaboration, including:

  • incident timelines that agents can continuously update,
  • safer remediation strategies such as canary changes,
  • richer evidence graphs and dependency analysis,
  • post-incident report generation,
  • agent permissions based on risk and environment,
  • multiple engineers collaborating with agents in the same incident workspace.

The larger direction is not simply to make Vitals "AI-powered."

It is to make observability agent-native:

Humans define intent and retain authority. Agents gather evidence, coordinate workflows, execute approved operations, and verify the result.

Built With

  • agentic-ai
  • ai-agents
  • artificial-intelligence
  • developer-tools
  • devops
  • human-in-the-loop
  • incident-response
  • mcp
  • observability
  • react
  • site-reliability-engineering
  • typescript
  • vite
  • web-applications
  • webmcp
Share this project:

Updates

Submission history