Liquet

A dispute arrives by email. Liquet's 3-model AI pipeline reads the evidence, cross-examines claims, stress-tests its own verdict — then either resolves autonomously or escalates to a human.

Comment

LIQUET — Autonomous Marketplace Dispute Arbitrator

Inspiration

A buyer in São Paulo files a dispute at 11 PM. The product arrived, but the box was empty. The seller's automated system responds within thirty seconds — denied. The buyer re-escalates. Now it sits in a queue: behind 847 other tickets, behind a weekend, behind a public holiday, behind an agent who skim-reads six words and clicks "denied" because the queue just hit 900. Five days later, the buyer receives a form email that doesn't mention a single fact from their case.

That is not an edge case. That is Tuesday.

Online marketplaces collectively handle hundreds of millions of disputes every year. Most platforms resolve them the same way: keyword filters, denial-rate targets, and a human review queue that no one can actually keep up with. The systems that call themselves "AI-powered" are almost always just rule engines with a confidence score hardcoded to sound decisive. They don't build in doubt. They don't know when to stop. They optimize for throughput, not correctness — and on genuinely ambiguous cases, they guess.

The problem gnawed at us because it has a name from Roman law. When a jury had heard all the evidence and still could not decide, a juror was entitled to return a verdict of non liquet — "it is not clear" — rather than force a guess. That option, the structured right to abstain, is what makes justice honest rather than merely fast. A system that knows it cannot know is categorically different from a system that is wrong with high confidence.

Every existing dispute-automation tool we looked at had the same flaw: no principled abstention. The moment we articulated that gap, the project had a name and a north star: build an agent that resolves disputes it can resolve, and says so clearly when it cannot.

What it does

Liquet is a fully autonomous marketplace dispute arbitration agent. A dispute enters the system — through email, API, or the web dashboard — and Liquet runs a complete evidence-gathering and adjudication pipeline, stress-tests its own verdict against a built-in devil's advocate, runs adjudication three times to check that the reasoning is stable, and then makes a binary, non-negotiable decision: LIQUET (resolve autonomously) or NON LIQUET (escalate to a human reviewer). It never auto-resolves when it isn't sure, and it never pretends to be sure when it isn't.

Capability What it looks like Where it happens
Three-model pipeline qwen3.6-flash triages the incoming dispute and scores complexity before spending premium tokens; qwen3.6-plus visually inspects product photos and shipping screenshots for physical evidence; qwen3.7-max reasons over the assembled case file and produces the verdict services/orchestrator.py, mcp_servers/vision_intake/, services/adjudicator.py
7 MCP tool servers Independent tool servers for order records, logistics tracking, listing data, buyer/seller communications, visual evidence intake, policy engine, and resolution execution — the orchestrator calls only what each case actually needs mcp_servers/
Evidence reliability hierarchy Carrier scan (95%) > order record (90%) > listing data (85%) > photo (70%) > message thread (40%) > unverified claim (20%). Missing evidence lowers confidence deterministically; it never crashes a run or silently inflates a score core/models.py, services/adjudicator.py
LIQUET gate A hard-rule decision: effective confidence ≥ 0.80 AND order value < $500 AND no hard contradictions AND verdict stable across three independent runs → LIQUET (auto-resolve). Any single condition failing → NON LIQUET regardless of confidence score services/liquet_gate.py
Hard contradiction detection If the order record says "delivered" and the carrier scan says "returned to sender," both at ≥70% reliability, no score can override this; the case is escalated rather than decided wrong services/liquet_gate.py
Stability scoring The adjudicator runs three times on the same case file. If the verdicts disagree — different resolutions, or the same resolution but confidence variance > 0.08 — the effective confidence is penalised. A system that contradicts itself is not 87% confident; it is unreliable services/orchestrator.py
Skeptic model After the primary verdict, a second qwen3.7-max pass receives only the losing party's narrative and the primary verdict, and is instructed to generate the strongest possible rebuttal. A rebuttal that raises a hard contradiction the primary pass missed escalates the case services/orchestrator.py
Ghost case injection Before adjudication, historical cases with similar dispute category, order value bracket, and evidence pattern are retrieved and injected into the case file as precedent. The adjudicator is grounded in what the platform has actually decided before, not reasoning from scratch each time services/ghost_cases.py
NON LIQUET queue Human reviewers see a one-screen decision brief: both narratives, the complete evidence map with reliability scores, the agent's lean, the stability scores across three runs, and precisely why the case abstained. Every piece of information a reviewer needs to decide is pre-surfaced pages/NonLiquetQueue.jsx
Full audit trail Every orchestrator step, every MCP tool call, every LLM call, every gate decision, and every human override is logged immutably with timestamps and actor attribution — citable in any subsequent review repositories/dispute_repo.py, pages/CaseDetail.jsx
Email intake An IMAP poller monitors a Gmail inbox. Any structured dispute email — buyer complaint, seller response, or support escalation — is parsed by qwen3.7-max and automatically submitted as a new case, closing the intake loop with zero human intervention services/email_intake.py
Resolution webhooks LIQUET resolutions fire a signed POST to a configurable webhook URL. NON LIQUET escalations fire a separate endpoint. External systems — order management, payment processing, CRM — can react programmatically to every Liquet decision services/orchestrator.py
One-click human approval NON LIQUET cases generate HMAC-SHA256 signed approval links emailed to the designated reviewer. A single authenticated click approves or overrides the agent's recommendation and updates the case status — no login required backend/api/approvals.py
ReasoningGlass Streams qwen3.7-max extended-thinking tokens live in the UI as the adjudicator works — a dark terminal with typewriter effect showing the full reasoning chain before the verdict lands. Judges and reviewers can watch the agent think in real time pages/CaseDetail.jsx
VerdictNarrator cosyvoice-v3-plus TTS reads the verdict rationale aloud with an animated waveform. The verdict is not just a screen update — it's delivered services/narrator.py
SceneReconstruction wan2.6-t2i generates a visual reconstruction of the dispute scenario from the assembled facts so reviewers can immediately picture the conflict before reading the evidence map services/scene_reconstruction.py
Confidence Breakdown UI Raw adjudicator confidence, stability penalty, effective confidence, and the specific evidence items that moved the score — all shown with animated fill bars so the reviewer immediately understands why a case landed where it did components/ConfidenceBreakdown.jsx
Calibrated abstention The system's ECE (Expected Calibration Error) is 0.015 against a naive threshold baseline of 0.142. When Liquet says 83%, it is meaningfully more likely to be correct than when it says 71% — the scores are honest eval/run_eval.py

The LIQUET gate is deterministic Python, not a probabilistic model output. The gate cannot be hallucinated past. An LLM that generates a verdict of 91% confidence does not get to resolve a case where stability scoring shows 0.61 agreement across three runs. The math overrules the prose.

How we built it

Orchestrator. The core of Liquet is a single async orchestrator (services/orchestrator.py) that plans and executes the full investigation. It holds a concurrency semaphore (max 3 simultaneous full investigations) so the production instance doesn't saturate the QwenCloud rate limits under load, and it calls MCP tool servers in dependency order — fetching order data before vision intake, fetching the communications thread before the skeptic pass — so each model call is informed by the evidence that actually matters for it.

Three-model routing, matched to task. qwen3.6-flash handles fast triage and pre-classification via a dedicated /api/triage endpoint — it pre-scores complexity, priority, and initial lean before the orchestrator commits any premium tokens to a full investigation. qwen3.6-plus handles every call that carries images or screenshots — the vision intake MCP server passes evidence photos directly to the model for damage detection, label verification, and screenshot analysis. qwen3.7-max handles adjudication, the skeptic pass, email parsing, and the ReasoningGlass streaming — the calls where extended reasoning chain actually improves correctness. No single model handles everything. Each one handles only what it is best at.

7 MCP tool servers. Seven independent FastMCP servers — order_service, logistics_service, listing_service, comms_service, vision_intake, policy_engine, resolution_service — run in-process in the current deployment for latency, but each is a genuine standalone MCP server with its own mcp.run(transport="stdio") entrypoint. An external MCP-compatible client can connect to any of them over the actual protocol; the tool definitions and database access are identical whether called in-process or over stdio.

LIQUET gate. The gate is entirely deterministic Python (services/liquet_gate.py). It receives the primary verdict, stability scores, contradiction flags, and order metadata and applies four hard conditions in sequence. The code is readable, testable, and impossible to override through model output — if effective_confidence < 0.80, the case escalates, full stop.

Stability scoring. The orchestrator calls the adjudicator three times on the same assembled case file, with temperature varied slightly across runs to avoid identical sampling. It computes stability_score = fraction of runs that agree on the same resolution and effective_confidence = raw_confidence × stability_score. A verdict that holds across all three runs has full score; one that splits two-to-one is penalised; one that produces three different resolutions cannot pass the gate regardless of the nominal confidence figure.

Ghost case injection. Before the primary adjudication pass, services/ghost_cases.py queries the local case database for historical disputes with matching dispute_category, overlapping order_value_bracket, and the same set of evidence_types_present. Up to five matching cases, with their final resolution and confidence, are injected into the adjudicator's system prompt as precedent examples. The model reasons over real history, not a blank slate.

Email intake. An async IMAP poller (services/email_intake.py) monitors a Gmail inbox. Inbound emails matching dispute patterns are parsed by qwen3.7-max into a structured DisputeCreateRequest and submitted automatically. The full loop — buyer sends complaint email, Liquet parses it, investigates, and emails back an auto-resolution or escalation notice — runs with zero human involvement on LIQUET cases.

Webhooks and one-click approval. LIQUET resolutions POST a signed JSON payload to a configurable WEBHOOK_URL. NON LIQUET escalations generate a per-case HMAC-SHA256 token and email the reviewer a signed approve/override URL; clicking it validates the token, applies the decision, and closes the case — no separate login flow required.

Backend. FastAPI, async throughout via asyncio and aiosqlite. Structured JSON logging via structlog with per-request dispute_id binding for trace-level debugging. Pydantic v2 for all request/response validation. SQLAlchemy async ORM with a clean repository layer so the storage backend is swappable without touching service logic.

Frontend. React 18 + Vite + Tailwind CSS. A fixed sidebar navigation replaces the traditional top bar so the dashboard feels like an operator tool, not a marketing page. A full CSS design token system (--bg-base, --accent, --text-1, etc.) in :root means the dark theme is globally consistent including on pages not explicitly rewritten. Route-change animations (key={location.pathname}), animated confidence bars (CSS transition from 0 to target on first paint), count-up stat tiles, and pulsing status indicators give the UI a live quality that matches the real-time nature of the system.

Infrastructure. Docker Compose on Alibaba Cloud ECS Singapore — the backend FastAPI service behind the frontend nginx container on port 80. Evidence images on Alibaba Cloud OSS; CI/CD via GitHub Actions that deploys on every push to main.

Challenges we ran into

Calibrating the LIQUET gate without labelled ground truth. The gate thresholds — 0.80 confidence, $500 order value, stability requirement — sound like they could be arbitrary. Getting them to actually mean something required building an eval harness (eval/run_eval.py) with 10 hand-labelled cases covering the full range of dispute types, running the full pipeline against each, and checking that the gate produced zero false-PROCEED outcomes (cases that should have escalated but didn't). The final calibration target was ECE ≤ 0.02 against labelled outcomes, meaning the confidence scores needed to be genuinely predictive, not just high-looking numbers.

The skeptic model almost always agreed with the primary verdict. In early versions, the skeptic pass was given the full case file — the same context the adjudicator saw — and reliably found minor points to note without ever changing the outcome. The fix was to deliberately withhold the adjudicator's reasoning from the skeptic and give it only the losing party's narrative and the verdict. Forced adversarial asymmetry produces real rebuttals; symmetric access produces a second opinion that sounds critical but effectively endorses the first.

Stability scoring exposed a model caching problem. When the orchestrator called the adjudicator three times, early runs showed near-perfect stability — not because the reasoning was robust but because the model was returning nearly identical outputs on identical inputs at temperature 0. The fix was to vary temperature slightly across the three runs (0.0, 0.3, 0.5) and to shuffle the order of evidence items in the system prompt between passes. Real stability now means the verdict holds across genuinely different sampling conditions, not just three identical API calls.

Ghost case retrieval poisoned the adjudicator on unrepresented dispute types. When ghost case injection was first added, cases with no close historical matches retrieved the five globally most common historical cases regardless of relevance. An unrelated "wrong item" precedent injected into a "never arrived" case confused the adjudicator into applying the wrong policy section. The fix was a minimum similarity threshold — below it, the case adjudicates without precedent rather than with irrelevant precedent.

ReasoningGlass streaming and rate limits collided. Streaming qwen3.7-max extended-thinking tokens live while simultaneously running the stability scorer's three concurrent adjudication passes hit QwenCloud rate limits in testing. The resolution was to make the ReasoningGlass stream the final adjudication pass (the third, canonical run) rather than a separate fourth call, and to apply the orchestrator semaphore before starting any pass — so the live streaming and the stability runs share the same token budget and the same concurrency limit.

Email parsing ambiguity. Real dispute emails don't arrive formatted as JSON. A buyer saying "I got the wrong thing it looks nothing like the picture" needs to be mapped to dispute_category: item_not_as_described, an order_id extracted from a subject line or forwarded order confirmation, and a seller_response that may or may not be present. The qwen3.7-max email parser needed several iterations of prompt refinement before it consistently handled partial information, missing order IDs (which become None rather than hallucinated values), and multi-party email threads where buyer and seller messages are interleaved.

Accomplishments that we're proud of

  • Zero false-PROCEED rate on the labelled eval harness. The single most dangerous failure mode — a case that should have escalated but was auto-resolved instead — is checked explicitly and hard-fails the eval run if it is ever non-zero. We have not seen it.
  • A skeptic model that generates real adversarial rebuttals, not just mild caveats. The forced asymmetry (withholding the adjudicator's reasoning, giving only the losing narrative) produces rebuttals that occasionally identify evidence the primary pass underweighted — and those cases correctly escalate.
  • Ghost case injection that grounds the adjudicator in the platform's actual history, not just general LLM priors. When precedent exists, the model reasons over it; when it doesn't, the system says so and adjudicates on first principles.
  • A fully end-to-end closed loop: dispute arrives by email → Liquet parses, investigates, adjudicates → auto-resolution email sent to buyer and seller → webhook fires to external systems → all within minutes, with zero human involvement on LIQUET cases.
  • MCP integration that is architecturally real. Each of the 7 tool servers has a genuine mcp.run(transport="stdio") entrypoint. The in-process calling and the protocol-level calling hit the same handler functions and the same database.
  • An abstention mechanism that is honest rather than just conservative. The NON LIQUET path is not a fallback; it is the intended outcome for a specific class of cases. The system is designed to escalate ~25% of cases to human review — that is not a failure rate, that is the calibration target.
  • A one-click human approval flow that is secure. HMAC-SHA256 per-case signed tokens mean each approval link is single-use and case-specific; replaying a token from a resolved case cannot affect any other case.

What we learned

  • Deterministic gates outperform probabilistic thresholds for safety-critical decisions. Having liquet_gate.py be plain Python with explicit conditions means the gate is testable, auditable, and immune to model hallucination. The LLM provides the confidence score; the Python decides whether that score is sufficient. Keeping those two concerns cleanly separated was the single best architectural decision we made.
  • Self-doubt is a feature, not a bug. The stability scorer, the skeptic pass, and the ECE calibration all exist to surface uncertainty rather than hide it. Every time we were tempted to smooth over a confidence number or suppress a conflicting stability run, making it visible produced a better system — both in accuracy and in trustworthiness.
  • Model tier selection matters more than always using the biggest model. qwen3.6-flash for triage costs a fraction of a qwen3.7-max call and produces pre-classification that is good enough to route cases correctly. Using the flagship model for every step would make the system 4-5x more expensive and no more accurate on the simple cases that represent ~60% of volume.
  • MCP's real value is composability, not just organization. Running 7 independent tool servers means the orchestrator can call any combination of them in any order — and a future agent with different reasoning needs can call the same tools without touching the backend. The tool definitions are the stable interface; the orchestrator calling them is just one consumer.
  • End-to-end automation requires closing every loop. A system that auto-resolves 75% of disputes but requires a human to click "send email" on every resolution is not actually automated. Closing the intake loop (email parsing), the resolution loop (verdict notification), and the escalation loop (signed one-click approval links) is what makes the difference between an autopilot and a sophisticated drafting tool.

What's next for Liquet

  • Cross-platform evidence verification — integrating with carrier APIs (FedEx, DHL, local last-mile) to pull authoritative tracking data directly rather than relying on seller-submitted screenshots, raising the reliability ceiling on logistics evidence to near-certain.
  • Multi-currency and jurisdiction-aware policy engine — right now the policy engine reasons over a single policy document. A production deployment needs policy variant selection by seller country, buyer country, and dispute category, with the adjudicator grounded in the correct legal jurisdiction.
  • Longitudinal seller risk scoring — tracking LIQUET/NON LIQUET rates, auto-resolution outcomes, and human-override patterns per seller over time to surface systemic bad actors before individual disputes escalate. A seller whose disputes consistently hit the NON LIQUET path is a different risk profile from one who has a single edge-case claim.
  • PostgreSQL migration — the data layer is already async SQLAlchemy with a clean repository interface; moving off SQLite for production concurrent load is a DATABASE_URL change, not a rewrite.
  • Generalising the pattern to adjacent verticals — the same architecture (fast triage → deep evidence analysis → adversarial stress test → deterministic gate → human escalation with brief) applies directly to insurance claim routing, vendor invoice disputes, and content moderation appeals. The domain models and policy document change; the orchestration and gate logic do not.

Built With

  • aiosqlite
  • alibabacloudecs
  • alibabacloudoss
  • cosyvoice-v3-plus
  • docker
  • fastapi
  • fastmcp
  • githubactions
  • nginx
  • pydantic
  • python
  • qwen3.6-flash
  • qwen3.6-plus
  • qwen3.7-max
  • react
  • sqlalchemy
  • structlog
  • tailwindcss
  • tenacity
  • vite
  • wan2.6-t2i
Share this project:

Updates