Data Incident Action Agent

Turns raw data incidents into AI-assisted diagnosis, business impact, and owner-assigned recovery plans with human approval.

Comment

Inspiration

Data incidents rarely end when an alert fires. Once a pipeline fails, someone still has to understand what broke, determine the downstream impact, decide what needs to happen first, identify who should own each action, and communicate the incident safely.

I wanted to explore whether an AI agent could go beyond simply explaining an error and instead help coordinate the decision-making around incident recovery.

That led to Data Incident Action Agent: an AI-assisted workflow that turns raw incident evidence into technical diagnosis, business impact, prioritised recovery actions, owner assignments, success checks, and a stakeholder-ready incident brief.

What it does

The user provides a data incident report. The agent combines that report with available dependency evidence and produces a structured incident-response plan.

The built-in demo simulates schema drift in CUSTOMER_ORDERS:

  • customer_id is renamed to customer_key
  • order_total changes from NUMERIC to STRING
  • the downstream fct_customer_orders transformation fails
  • executive and operational dashboards become stale

From this evidence, the agent:

  1. Classifies the incident severity
  2. Diagnoses the technical failure and likely root cause
  3. Identifies impacted data assets
  4. Explains the business impact
  5. Generates prioritised P0/P1/P2 recovery actions
  6. Assigns an owner and success check to each action
  7. Recommends containment and prevention measures
  8. Generates an incident brief and stakeholder notification draft
  9. Requires human approval before the communication is considered reviewed

The application also exposes operational AI metrics including the Gemini model used, response latency, input tokens, output tokens, and estimated inference cost.

The prototype deliberately does not send stakeholder messages or modify production systems. Human approval is an explicit boundary between AI recommendations and consequential actions.

How I built it

The application is built in Python 3.12 with Streamlit for the interactive interface.

The reasoning layer uses Gemini 3.7 Flash through Vertex AI, accessed with the Google GenAI SDK (google-genai).

The workflow is:

Incident report + dependency evidence → Gemini → structured response validation → diagnosis → business impact → recovery plan → incident brief → human approval

Gemini receives the incident information and explicitly supplied dependency metadata. The generated response is validated using Pydantic structured-output models before the application renders it.

The application is containerised with Docker and deployed as a public hackathon demo on Google Cloud Run.

I also capture Gemini usage metadata so each run can display latency and token consumption. Configurable Gemini token prices are then used to calculate an estimated model cost.

Challenges I faced

Keeping the agent grounded

One of the biggest challenges was making the agent useful without allowing it to invent incident evidence.

I therefore separated supplied evidence from AI reasoning. The demo dependency metadata is explicitly labelled, and Gemini is instructed to reason from the incident report and the evidence provided to it.

Producing operational output instead of generic AI text

A long explanation is not enough during an incident.

I designed structured output contracts so the model must return actionable information such as severity, impacted assets, prioritised actions, owners, and measurable success checks.

Designing a safe human-in-the-loop boundary

Incident-response automation can become risky if an AI system automatically communicates incorrect information or changes production systems.

For this prototype, the agent prepares the response, but a human remains responsible for approval. Even after approval, the application records the review rather than actually sending the message.

Deploying Gemini securely on Google Cloud

Moving from a local prototype to Cloud Run required configuring the application's runtime identity and Vertex AI permissions correctly.

This helped me understand the difference between development credentials and the service identity used by a deployed Cloud Run application.

Making AI usage measurable

I also wanted the prototype to show that AI inference has operational characteristics of its own.

The application therefore surfaces model latency, input/output token usage, and an estimated cost for each analysis.

What I learned

The biggest lesson was that an agent becomes much more useful when it produces decisions and next actions rather than only answers.

Adding priorities, responsible teams, and success criteria transformed the output from an AI explanation into something closer to an incident-response workflow.

I also learned how important structured output is when integrating an LLM into an application. Pydantic validation gives the rest of the application a predictable contract instead of depending on arbitrary natural-language responses.

Another important learning was that human approval can be part of the architecture, rather than an interface feature added at the end.

Finally, deploying the application gave me practical experience connecting Gemini through Vertex AI to a containerised application running on Google Cloud Run, including runtime IAM configuration and model usage measurement.

What's next

The current hackathon version intentionally uses clearly labelled demo dependency metadata so that it is reproducible without access to private production systems.

The next step would be to replace that demo evidence with integrations to real data engineering and observability systems, allowing the agent to retrieve lineage, pipeline status, ownership, and incident context dynamically.

Future versions could also:

  • create incident tickets after human approval
  • monitor recovery checks
  • maintain incident history
  • integrate with data catalogs and orchestration platforms
  • notify stakeholders through approved enterprise channels
  • compare actual recovery outcomes against the agent's recommendations

The goal is to evolve the prototype from an AI incident analyser into an agentic incident-response coordinator for data engineering teams.

Built With

  • agentic
  • agents
  • api
  • cloud
  • data
  • docker
  • engineering
  • gemini
  • genai
  • generative
  • google
  • human-in-the-loop
  • llm
  • pydantic
  • python
  • run
  • sdk
  • streamlit
  • vertex
Share this project:

Updates

Submission history