Data Incident Commander

An evidence-first AI incident response system for DataHub that uses MCP-verified investigation, requires human approval before write-back, and independently verifies every recorded incident.

Comment

Inspiration

Modern data incidents are rarely caused by a single broken dataset.

Engineers must investigate metadata, lineage, ownership, freshness, quality signals, and downstream impact across multiple disconnected tools. Even with AI assistance, one critical question remains:

Can we trust the evidence behind the recommendation?

We built Data Incident Commander to answer that question.

Instead of allowing AI to guess, our goal was to create an evidence-first workflow where every conclusion is grounded in verified DataHub metadata, every mutation requires explicit human approval, and every write-back is independently verified before an incident is considered complete.

What it does

Data Incident Commander is an evidence-first AI incident response workspace built on DataHub.

An investigation starts with a factual incident draft containing only the observed issue and affected asset. No recommendations or assumptions are generated at this stage.

The investigation then moves through a complete lifecycle:

Draft → Investigated → Awaiting Approval → Approved → Writeback Pending → Recorded

During investigation, the system collects verified evidence from DataHub through the mandatory MCP workflow, including:

  • Metadata
  • Ownership
  • Lineage
  • Freshness
  • Quality information
  • Direct and downstream impact

Using that evidence, the platform produces:

  • Evidence ledger
  • Blast-radius analysis
  • Severity assessment
  • Confidence score
  • Responsible-owner identification
  • Evidence-backed recommendation

Before any metadata can be changed, a human reviewer must explicitly approve the investigation.

Only then is a controlled incident tag written back to DataHub, followed immediately by an independent read-back verification.

The incident becomes Recorded only after successful verification.

Every step is preserved in a complete audit timeline.

How we built it

The backend is built with Python and FastAPI.

The frontend is built with React and TypeScript.

The investigation workflow is powered by DataHub OSS together with mcp-server-datahub.

Before the investigation begins, the MCP runtime validates its available tools and compatible schemas to ensure evidence collection is trustworthy.

Verified MCP tools include:

  • search
  • get_entities
  • get_lineage
  • get_lineage_paths_between

MCP performs investigation reads, while a separately gated DataHub GMS integration performs the approved write-back and independent verification.

The deployed demo uses:

  • Docker
  • Nginx
  • systemd
  • DataHub OSS

Challenges we ran into

The biggest challenge was preventing AI from appearing more certain than the available evidence.

Instead of allowing the application to continue with incomplete information, we implemented a fail-closed design.

If verified evidence cannot be collected, the investigation simply stops.

Another challenge was separating investigation from mutation.

Collecting evidence must never automatically modify metadata.

Human approval became a hard technical boundary, not simply a user-interface button.

Finally, we wanted to prove that metadata had actually changed.

Instead of trusting a successful write request, the system independently reads the updated metadata back from DataHub before recording the incident.

Accomplishments that we're proud of

We successfully demonstrated the complete investigation lifecycle:

Draft → Investigated → Awaiting Approval → Approved → Writeback Pending → Recorded

The demonstrated investigation includes:

  • Verified evidence collection
  • Ownership discovery
  • Lineage analysis
  • Blast-radius calculation
  • Severity assessment
  • Confidence scoring
  • Human approval
  • Verified DataHub write-back
  • Independent verification
  • Complete investigation timeline

We also completed:

  • Public live deployment
  • Judge-ready documentation
  • Frontend production build
  • 423 passing backend tests

What we learned

Building trustworthy AI is not about producing better answers.

It is about producing answers that can be verified.

Throughout this project we learned that operational AI systems need clear boundaries between:

  • Observed facts
  • Verified evidence
  • AI reasoning
  • Human decisions
  • External mutations
  • Verification receipts

Those boundaries make AI significantly more trustworthy for enterprise data operations.

What's next

The current incident repository is intentionally stored in memory for the hackathon demonstration.

Next steps include:

  • Persistent incident storage
  • Authentication and reviewer roles
  • Additional DataHub write-back actions
  • Notification workflows
  • Historical incident comparison
  • Operational analytics
  • Expanded governance policies

Conclusion

Data Incident Commander demonstrates that AI can accelerate incident response without sacrificing trust.

By combining verified DataHub evidence, mandatory MCP investigation, human approval, approval-gated write-back, and independent verification, the platform transforms fragmented investigations into one auditable workflow.

Evidence first. Human approved. AI assisted.

Built With

Share this project:

Updates