DriftRadar

DriftRadar's multi-agent architecture: Detect → Diagnose → Explain → Plan (Gemini) → Execute/Escalate → Verify → Remember

Inspiration

A model's accuracy can quietly degrade for weeks before anyone notices. And once someone does notice, answering "which upstream data change actually caused this?" today means manually tracing lineage across wikis, Slack threads, and tribal knowledge — often taking days. We wanted to build an agent that closes that entire loop autonomously.

What it does

DriftRadar watches a model's live prediction distribution and runs a full closed-loop investigation with no human driving it:

Detects drift statistically — KS-test, PSI, embedding centroid drift — not "does this look different to an LLM."
Traces backward through a data lineage graph and runs an intervention-style check on every upstream node to isolate the genuine cause, distinguishing it from nodes that merely drifted around the same time.
Explains the finding in plain language via an LLM reasoning over the statistical evidence — never inventing a cause on its own.
Plans a concrete remediation action using Gemini, scored with a confidence and risk level.
Acts autonomously when confidence is high and risk is low/medium — triggers a retrain, rolls back a model version, quarantines a bad data source, or opens a ticket. Otherwise it escalates to a human instead of guessing.
Verifies the action actually worked by re-checking drift, rather than assuming success.
Remembers every incident — outcome included — so future decisions are informed by what was tried before.
Writes back the diagnosis as a structured incident and opens a GitHub issue with a concrete suggested fix.

The full loop: Detect → Diagnose → Explain → Plan → [Execute → Verify | Escalate] → Report → Remember.

Autonomy safety gate: the Executor never acts blind. A plan only auto-executes when confidence ≥ 0.75 AND risk level isn't high — enforced in code, not just prompted for. A hallucinated "high confidence" from the LLM alone cannot authorize a real action.

How we built it

Detector — pure statistics (scipy: KS-test, PSI, cosine drift), deliberately not an LLM call — the right-sized tool for the job.
Diagnoser — causal graph traversal over a lineage DAG with bootstrap intervention testing, to isolate genuine causes from mere correlation.
Explainer — Groq/LLaMA-70B, grounded so it can't report a root cause the algorithm didn't isolate.
Planner — Gemini 3.5 Flash decides the remediation action, with the same grounding discipline: a proposed target outside the trace's actual isolated root causes is rejected and forced to a safe no-action plan.
Executor / Verifier / Memory / Coordinator — a multi-agent layer where the Executor carries out actions and tracks rollback references, the Verifier re-checks drift post-action rather than assuming success, Memory persists incident history (local JSON or Firestore), and the Coordinator orchestrates the full loop end-to-end.
Backend: FastAPI, deployed on Google Cloud Run.
Frontend: React + Vite, deployed on Vercel, with a live Agent Action view, confidence meter, and incident history.

Challenges we ran into

Keeping the Planner from ever auto-executing on a hallucinated high-confidence claim — solved by enforcing the confidence/risk gate in code rather than trusting the model's own stated confidence.
Concurrency bugs in the Memory layer: concurrent incident saves were racing and silently losing data; fixed with an asyncio lock and atomic writes.
A double-approval race where two concurrent "Approve & execute" clicks could both pass the guard and both fire a real remediation action — fixed with an atomic claim-based approval flow.
A slow/hung Gemini call could block the entire event loop since it was awaited synchronously — moved off the event loop with a hard timeout.

Accomplishments that we're proud of

A real, tested safety architecture around autonomous action — not just a demo that always "succeeds." The agent escalates to a human exactly when it should, and every reversible action carries a working rollback path.

What we learned

Autonomy is mostly a trust problem, not a capability problem — the hard part wasn't getting an LLM to propose an action, it was building the guardrails so a wrong or overconfident proposal can't cause real damage.

What's next for DriftRadar

Wiring the currently-simulated actions (retrain, rollback, quarantine) to real orchestrators — Vertex AI Pipelines, a model registry — and connecting to a live DataHub instance instead of the mock lineage graph.