Autonomous Data Incident Investigator (ADII)

Perceptix (ADII): an autonomous data reliability agent that detects pipeline incidents, explains root cause with Gemini, verifies evidence, and escalates alerts with approval-gated remediation.

Comment

Inspiration

Modern data stacks fail in quiet, expensive ways: schema drift, freshness gaps, broken orchestration runs, and upstream changes often surface only after dashboards or downstream apps break. We wanted an MVP that behaves like an on-call data investigator, not just a monitor.

What it does

Autonomous Data Incident Investigator (ADII) continuously observes pipeline and table signals, detects incidents, and produces an incident report with:

  1. Triage and prioritization
  2. Root-cause hypotheses (Gemini-powered reasoning)
  3. Investigation steps and collected evidence
  4. Verification with confidence scoring
  5. Persistent incident history and audit events
  6. Slack/email escalation
  7. Optional remediation with approval gates and risk assessment

It also ships a dashboard for live status, incidents, actions (archive/delete/bulk), and an incident detail view with a full reasoning trace.

How we built it

  • Backend: FastAPI + Gunicorn/Uvicorn with a modular agent architecture (Observer, Reasoner, Investigator, Verifier, Escalator, Historian).
  • AI: Google Gemini (gemini-3-pro-preview) used for structured JSON decisions across triage, planning, verification, policy suggestion, and remediation risk assessment.
  • Storage: SQLite persistence for incidents, metrics, audit events, and tenant isolation (separate DB per tenant).
  • Real-time UX: WebSocket incident stream for live activity and UI updates.
  • Frontend: React + TypeScript + MUI dashboard consuming secure APIs.
  • Ops: Nginx reverse proxy + TLS, systemd service, production logging.

Challenges we ran into

  • Production deployment on a multi-app server (Nginx routing, upstream configuration, and avoiding conflicts).
  • Ensuring long-running cycles don’t block requests (background tasks + WebSocket notifications).
  • Making Gemini calls reliable and safe: budgets, caching, strict JSON parsing, and fallbacks.
  • Getting alerting (Slack/email) working end-to-end and verifying deliverability.
  • Keeping demo flow smooth while maintaining real-world architecture.

Accomplishments that we're proud of

  • A working end-to-end autonomous loop: Observe → Reason (Gemini) → Investigate → Verify → Persist → Escalate → (Optional) Remediate.
  • A “Gemini proof” endpoint that exposes safe runtime metadata for judges.
  • A visible reasoning trace/decision log on each incident, showing what the system decided and why.
  • Multi-tenant foundations with isolated storage per tenant plus policy and approval endpoints.
  • A production deployment with TLS, secure API access, and persistent incident history.

What we learned

  • “Agentic” systems need guardrails: deterministic fallbacks, strict schemas, and budget controls are mandatory for reliability.
  • Demos succeed when the system exposes internal reasoning clearly (traceability beats raw complexity).
  • Productionizing isn’t just code: deployment hygiene, routing, and observability make or break the experience.

What's next for Autonomous Data Incident Investigator (ADII)

  • Add a full UI for tenant/policy/approval administration (currently available via API).
  • Expand native connectors (Airflow auth, Datadog, Prometheus, dbt, etc.) and richer telemetry ingestion.
  • More advanced evidence tracing and root-cause graphing in incident detail views.
  • Stronger RBAC, audit trails, and organization-level configuration.
  • Smarter remediation orchestration (playbook selection, dynamic safeguards, and cost-aware model routing).

Built With

  • audit-events
  • build
  • built-with-python-3.12-fastapi
  • gunicorn-google-gemini-api-(google-genai-sdk
  • incident/activity
  • isolation)
  • live
  • management)
  • material
  • metrics
  • model:-`gemini-3-pro-preview`)-sqlite-(persistent-incidents
  • mui)
  • nginx
  • production
  • proxy
  • pytest
  • react
  • reverse
  • service
  • stream)
  • systemd
  • tenant
  • tls)
  • tooling)
  • tsc
  • typescript
  • ui
  • uvicorn
  • vite
  • websockets
Share this project:

Updates