Inspiration
Monday, 9am. Conversion drops 3.2%. The PM pings Slack: "What happened?" The data team spends the next 4 hours slicing dashboards — segment by segment, device by device. A data engineer queries the warehouse. A data analyst builds breakdowns in a notebook. Eventually — maybe — they find a culprit. Or maybe the drop is systemic and they just burned $800 of salary chasing noise. At $200/hour for a data engineer + analyst pair, that's a $3,200/week tax on ambiguity. And the bigger problem? Every AI agent built to "solve" this will always produce a confident-sounding explanation — whether the data supports one or not. A false diagnosis that triggers a rollback, pauses a campaign, or wastes another 4 hours of debugging is worse than no diagnosis at all. We built prove-or-abstain to break both problems at once: eliminate the manual investigation cost and eliminate the false-diagnosis liability. It's an agent that investigates metric anomalies the way a careful analyst would — and refuses to act when the evidence is insufficient, naming exactly why.
What it does
Prove-or-abstain takes two snapshots of a business metric (baseline vs. current, segmented by dimensions like device, country, plan, channel) and returns one of two verdicts in under 2 seconds:
- ASSERT — cause found and statistically proven. If confidence ≥ 70% and autopilot is ON, the agent executes the remediation directly (pauses the campaign, posts to Slack, fires a webhook). Every action is logged to an audit trail with a SHA256-hashed replayable trace.
- ABSTAIN — no single cause isolates. The drop is real but systemic or
diffuse. The agent escalates to a human with the exact failing gate named —
not "I don't know," but "interaction share 0.63 > 0.50 — rate and mix effects
are entangled; this needs a human."
This isn't a prompt-based heuristic. It's a four-gate pipeline where each gate
is a hard mathematical check, not an LLM opinion:
| Gate | Condition | What it prevents |
| :--- | :--- | :--- |
| Material | |\Delta R|/R_0 ≥ 2% | Investigating noise |
| Localized | concentration ≥ 0.55 | Blaming one segment for a diffuse drop |
| Significant | z-test p ≤ 0.01 (or n ≥ 1000 for sum metrics) | Acting on sampling noise |
| Clean | interaction share ≤ 0.50 | Confusing rate/mix entanglement with a cause |
The ABSTAIN gate is the cost-saving innovation. On real-world data, 7 out
of 10 investigations produce ABSTAIN — because most metric shifts are systemic,
not localizable. An agent that fabricates a cause on those 7 creates downstream
cost (wrong rollback, wasted debugging). Prove-or-abstain saves the
investigation and prevents the false action.
What it replaces:
| Before (human) | After (prove-or-abstain) |
| :--- | :--- |
| Data engineer queries warehouse: 30 min | /investigate POST: 2 sec |
| Data analyst builds segment breakdowns: 2h | 4 gates, automated: 2 sec |
| Analyst evaluates significance: 30 min | Two-proportion z-test: 2 sec |
| Team discusses confidence: 1h | Confidence score + calibrated ECE: 2 sec |
| Decision: act or escalate: 30 min | Autopilot executes or escalates: 2 sec |
| Total: 4.5 hours, ~$900 | Total: 2 seconds, $0.004 |
| Multiple times/week | 24/7, no fatigue, no phantom diagnoses |
Track: 4 — Autopilot Agent. The agent handles ambiguous inputs
(Qwen routes free-text questions), invokes external tools (SQL, Google Sheets,
CSV upload, time series), includes human-in-the-loop checkpoints (ABSTAIN
always escalates, autopilot requires confidence ≥ 0.70), and is
production-ready (Docker, CI, 105 tests, persistent state, SSE streaming).
## How we built it
Math first, gates before agent. The exact attribution decomposition was
built and validated against a hand-derived oracle before any LangGraph loop
existed. The ASSERT/ABSTAIN decision was tuned and verified against three
calibrated scenarios independently of the orchestration — the safety property
was proven before the agent was wired around it.
rate = w0(r1 - r0)
mix = r0(w1 - w0)
interaction = (w1 - w0)(r1 - r0)
contribution = rate + mix + interaction
residual = 0
Two orchestration modes, same verdict. A LangGraph state machine
(
detector → hypothesizer → investigator → verifier, bounded loop) provides a fixed, auditable path. An alternative agent mode lets Qwen drive the investigation via tool calls — a determinism guard (_finalize_verdict) checks every dimension Qwen skipped, so the LLM can never cause a false ABSTAIN. The 20-scenario benchmark proves both modes produce identical ASSERT/ABSTAIN outcomes. A 20-scenario benchmark with meaningful real data. 10 synthetic scenarios (one per gate edge case, ground truth derived from panel construction, not pipeline output) plus 10 public real-world datasets (seaborn, vega-datasets, UCI, fivethirtyeight): | Metric | Value | | :--- | :--- | | Accuracy | 100% (20/20) | | False-ASSERT | 0% | | False-ABSTAIN | 0% | | ECE (calibration) | 0.41 (conservative — under-confident, safe direction) | | Per-investigation cost | ~4,500 tokens, $0.004 (qwen-turbo) | The hard LLM boundary — Qwen drives the path, math decides the outcome. Qwen (via DashScope on Alibaba Cloud) does four things and nothing else:plan_dimensions()orders candidate dimensions to find the cause faster,write_report()phrases a conclusion from pre-computed numbers,route_query()maps free-text questions to the right investigation, andmap_schema()reshapes unfamiliar CSV columns — the one documented exception where Qwen's judgment matters, with a self-verification pass that catches its own misreads. Every number in the output comes from pandas/numpy. Run withQWEN_MOCK=1and every verdict is bit-identical — the LLM is provably separable from the outcome. Production-ready stack. FastAPI with 16 endpoints (CSV upload, SQL queries, Google Sheets, time series, natural language, MCP server), SQLite-backed persistent memory with thread-safe locking, Docker image (non-root user, HEALTHCHECK), SSE streaming for live trace display, webhook notifications (Slack/Discord/Teams auto-detected on EXECUTE), and an MCP server exposing 7 tools that Qwen Cloud agents consume directly. ## Why Qwen Cloud We chose Qwen on Alibaba Cloud DashScope for three concrete reasons, not defaults: - Tool-calling reliability. Qwen drives the agent loop by calling
test_dimension,drill, andfinalizein a bounded tool-use loop. 100% benchmark accuracy in agent mode proves the tool calls are consistent and predictable — Qwen never invents a dimension name or overrides a gate verdict. - Model interchangeability without verdict drift. Because the gates
decide the outcome (not the LLM),
qwen-turbo,qwen-plus, andqwen-maxall produce identical ASSERT/ABSTAIN decisions. Cross-model evaluation confirms this — run the cheapest model, get the same correctness. At $0.40/M input tokens and $1.20/M output tokens for qwen-turbo, a full investigation costs $0.004. - MCP native integration.
mcp_server.pyexposes the full pipeline (investigate, autonomous check, dashboard, alert resolution, gate descriptions) as MCP tools over stdio or SSE transport. A Qwen Cloud agent becomes the outer orchestrator — it decides when to investigate based on user context, calls the pipeline, interprets results, and generates a human-readable response with embedded trace. Where Qwen earns its keep in the pipeline: map_schema()— the one place judgment matters. When a raw CSV has unfamiliar column names, Qwen maps them to[metric, dims..., n, c]. Unlike every other function in the codebase, mock and real mode can genuinely disagree here — so real mode runs a self-verification pass where Qwen re-examines its own answer.suggest_setup()— classifying unfamiliar metric names as rate vs. sum from the name alone (a genuine text-understanding task).- Routing free-text questions ("why did conversion drop?") to the right investigation, extracting segment filters from follow-ups.
- Ordering dimensions in spaces wider than 2 candidates (finds the cause in
1 iteration vs. 3).
## Challenges we ran into
Making diffuse scenarios genuinely hard. A naive uniform shift is too
easy to reject — every gate fails cleanly. We built a
mixshiftscenario where composition and rate move simultaneously, producing a non-trivial interaction term. That's what actually stress-tests the "clean mechanism" gate. Rate limiting vs. autonomous monitoring. DashScope enforces 60 req/min. The agent loop makes 4-8 tool calls per investigation, so continuous autopilot surveillance can trip the limit. We added a sliding-window rate limiter that queues investigations without dropping alerts — a broken feed never kills the loop, and a throttled source never starves others. Schema mapping — the one honest LLM dependency.map_schema()is the single place in the entire codebase where mock and real mode can produce different results — interpreting ambiguous column names has no mechanically deducible answer. Rather than hide this behind a human-confirmation gate (which makes the LLM decorative), we built a two-pass system: Qwen proposes a mapping, then self-verifies it against the sample rows before it's acted on. A single-cell collapse is ambiguous by construction. One cell (paid×mobile) collapsing concentrates 100% on both its defining dimensions. Mock mode always testeddevicefirst and never revealed the ambiguity; a live Qwen run reordered them and our own benchmark flagged it as wrong. The drill-down recovers the other dimension either way — the full diagnosis was never lost — so we fixed the benchmark to credit either field and documented the nuance instead of silently special-casing it. Persistent baseline without losing history. An earlier version of continuous monitoring kept its baseline in an in-process dict — lost on restart. Moving to a real SQLite table with a pooled reference window (same summed-counts algebra as the time-series baseline, keeping the z-test valid) made the autopilot durable across restarts. ## What we learned - Silence is more valuable than noise. 7 out of 10 real-world datasets produce ABSTAIN. An agent that invents a cause on those 7 creates churn: the PM rolls back a deploy, the engineer reverts a feature flag, the analyst builds a dashboard for a phantom problem. Prove-or-abstain saves the investigation and prevents the downstream cost of a false positive. The ABSTAIN rate is the feature.
- Qwen + deterministic gates is the right architecture split. Let the LLM drive exploration (tool-choice ordering), let the math decide the verdict. The two layers are independently testable — mock mode proves verdict independence, benchmark proves math correctness, cross-model eval proves LLM interchangeability.
- Cost calibration is a competitive moat. Every investigation is metered: tokens consumed, cost computed, latency logged. The qwen-turbo → qwen-max cost spectrum lets operators trade speed for thoroughness without impacting correctness. At $0.004/investigation, the ROI against a $200/hr data analyst is 50,000:1 per investigation.
- Real data surfaces genuine insights that synthetic benchmarks miss.
The Titanic passenger manifest confirms "women and children first" — the
agent finds
sex=female(p=0.0018), not the popularpclassexplanation that fails significance. College major data reveals the STEM employment gap concentrates in majority-women fields — a genuine, unplanted finding with 1.4% confidence, correctly staying a RECOMMEND not an auto-EXECUTE. - An MCP server changes the relationship. When Qwen Cloud agents can
call prove-or-abstain as a tool, the integration flips: Qwen becomes the
primary orchestrator, prove-or-abstain becomes its skill. A user asks
Qwen "why did revenue dip?" — Qwen calls
investigate_scenario, reads the gates, interprets the result, and generates a response. The separation of concerns is preserved, the value is multiplied. ## What's next - Closed control loop. Beyond one-shot EXECUTE: the agent observes the effect of its own action, re-investigates, and confirms or rolls back — the full OODA loop a human team runs today.
- OAuth-native connectors — Stripe, GA4, Amplitude — removing the manual export step for the most common SaaS metrics.
- Seasonality-aware baselines for time series, and adaptive pooling for long-running watched sources (today defaults to all history).
- Deeper drill-down beyond one level.
- Multi-turn conversational interface beyond today's single filtered follow-up — real dialogue with Qwen maintaining investigation context.
- evidence.py's embedded event table replaced by a real calendar/deploy-log/ticketing integration, grounding ASSERT speculation in actual operational events. --- Built with: Python · FastAPI · LangGraph · Qwen via Alibaba Cloud DashScope · pandas · numpy · Docker · SQLite · MCP (Model Context Protocol) · Alibaba Cloud Function Compute Repository: github.com/Demba09/prove-or-abstain License: MIT
Log in or sign up for Devpost to join the conversation.