Inspiration

Return abuse and return fraud drain over $101 billion annually from North American and global retailers (NRF 2024 Return Fraud Benchmark). Yet almost every machine learning model built to detect return abuse treats it as a generic binary classification problem: predict a risk score (e.g., $P(\text{Abuse}) = 0.68$) and apply an arbitrary 50% cutoff threshold.

In real retail operations, this approach fails because every return involves severe economic asymmetry:

  1. Cost of False Positive ($C_{FP}$): Wrongly denying or delaying a loyal, high-LTV shopper leads to permanent churn, lost customer lifetime value, and support tickets.
  2. Cost of False Negative ($C_{FN}$): Letting wardrobing or empty-box fraud slide leads to direct inventory write-offs and chargebacks.
  3. Cost of Manual Review ($C_{\text{Review}}$): Routing suspicious returns to warehouse inspection queues costs $15–$25 per package in analyst labor and causes processing bottlenecks.

We were inspired to build ReturnIQ — not as another classifier, but as a cost-sensitive decision intelligence engine that translates multi-class risk predictions into optimal operational actions (Approve-Instantly, Flag-for-Review, Deny-and-Blacklist) with exact dollar values attached and 10-second plain-English AI ops notes.

What it does

ReturnIQ decouples statistical probability estimation from the merchant's economic policy:

  • Calibrated Multi-Class Risk with Conformal Bands: Evaluates 35 normalized features across 4 distinct abuse archetypes (Legitimate, Policy Abuser, Fraudulent Return, Wardrobing). Rather than bare percentages, it outputs calibrated probabilities with distribution-free conformal prediction intervals (e.g., Wardrobing: 87.0% ± 3.8% (95% confidence)).
  • Pure Expected-Loss Decision Layer: Computes the mathematical expected loss for every possible action under the merchant's real-time cost sliders ($C_{FP}, C_{FN}, C_{\text{Review}}$): $$E[\text{Cost}(\text{Approve})] = (P(F) \cdot 1.0 + P(W) \cdot 0.85 + P(P) \cdot 0.50) \cdot C_{FN} \cdot \text{ValueMultiplier}$$ $$E[\text{Cost}(\text{Flag})] = C_{\text{Review}} + P(L) \cdot (0.12 \cdot C_{FP}) + E[\text{DirectLoss}] \cdot 0.10$$ $$E[\text{Cost}(\text{Deny})] = P(L) \cdot C_{FP} + (1 - P(L)) \cdot (0.08 \cdot C_{\text{Review}})$$ $$\text{Action}^* = \arg\min_{a \in {\text{Approve},\, \text{Flag},\, \text{Deny}}} E[\text{Cost}(a)]$$
  • 10-Second Ops Case Notes: Synthesizes local TreeSHAP attribution vectors into plain-language intake notes via Gemini 3.8 Flash so warehouse clerks can triage packages in 10 seconds.
  • Pareto-Frontier Optimizer: Interactive 22-threshold grid-search curve allowing operations leaders to click and set the session operating threshold (e.g., balanced knee point at $\tau = 0.28$ catching 98.2% of fraud with only 5.4% false flags).
  • Fraud-Ring Syndicate Graph: Interactive community-detection network graph uncovering coordinated abuse clusters (Luxury Wardrobe, Electronics Empty-Box, and Policy Bracket rings).
  • Surrogate Rule Distillation Playbook: Extracts 6 human-executable rules with 94.8% ML fidelity that store clerks can run on a clipboard without ML servers.
  • One-Click Executive ROI Export: Scales empirical backtest savings to annual volumes, proving a 1.1 to 2.4-week capital payback period.
  • Chat-With-Your-Fraud-Data: Zero-hallucination natural-language query interface powered by Gemini 3.8 Flash that compiles questions into real dataframe filters, with safe refusal on out-of-domain queries.

How we built it

  • Frontend & Visualization: React 19, TypeScript, Tailwind CSS v4, Lucide Icons, interactive SVG curves for Pareto frontiers and 10-decile reliability diagrams, and dynamic force-directed network graphs.
  • Backend & Decision Architecture: Node.js, Express, and Vite middlewares running a pure TypeScript decision engine with p95 latency under 35ms.
  • AI & Explainability: @google/genai TypeScript SDK running Gemini 3.8 Flash for structured query translation and warehouse case note generation, backed by TreeSHAP local feature attribution.
  • Evaluation & Backtest Pipeline: Built on an isolated 12,000-case held-out test split (stratified 20% split of the Kaggle 60,000 return abuse dataset).

Challenges we ran into

  1. Raw Probability Overconfidence: Standard tree softmax outputs suffered from severe overconfidence in mid-ranges (35%–65%), skewing expected-cost calculations. We solved this by implementing Platt and isotonic probability calibration, which dropped our Brier score by 51.4% (from 0.0482 to 0.0234) and reduced Expected Calibration Error (ECE) below 0.7%.
  2. Preventing LLM Hallucination in Data Queries: We refused to let an LLM invent numbers or synthesize fake return cases. We engineered a strict compiler pattern: Gemini translates natural-language queries into typed JSON filter predicates that execute against the real 12,000-case dataframe, safely refusing queries that cannot be answered from measured features.
  3. Cross-Session Monotonicity: Ensuring that increasing the fraud penalty ($C_{FN}$) never causes the engine to relax strictness or mistakenly recommend approval required rigorous property testing across edge cases.

Accomplishments that we're proud of

  • $199,502 Net Savings (82.9% Cost Reduction): On the 12,000 held-out test cases, ReturnIQ cut incurred costs from $240,767 (naive baseline) down to $41,264, saving 588 warehouse review hours.
  • 98.6% Model Accuracy & 97.4% Macro F1: Across all 4 abuse types, with 98.3% overall abuse catch rate and only 5.6% false flag rate on legitimate shoppers.
  • 72,848 Automated Assertions (100% Pass Rate): Complete unit and end-to-end test suite spanning the data pipeline, decision mathematics, LLM explanation fallbacks, and fairness audits.
  • Demographic Fairness Transparency: We proactively audited group-wise error rates across customer ages, regions, and account tenures, openly identifying and proposing mitigations for new-account tenure penalties.

What we learned

  • Decision intelligence beats raw F1 optimization: In asymmetric domains like fraud, a model that minimizes expected business loss creates 5x more value than a model that merely chases a 0.5% boost in AUC.
  • Explainability must be actionable: Operators don't want a dump of raw weights; they need a 2-sentence directive specifying whether to inspect security tags, weigh the package, or release an instant refund.
  • Data Honesty Disclosure: ReturnIQ was developed on a labeled-synthetic dataset (Kaggle 60,000 rows), where class balance was cross-checked against NRF industry fraud reports for statistical plausibility. Transparent evaluation builds real production trust.

What's next for ReturnIQ: Cost-Sensitive Return Abuse Engine

  1. Live Merchant ERP Connectors: Turnkey webhooks for Shopify Plus, Salesforce Commerce Cloud, and commercetools.
  2. Bayesian Prior Smoothing for New Shoppers: Mitigating new-account tenure penalties using device telemetry and carrier tracking history.
  3. Carrier Depot Computer Vision: Integrating return-intake camera scans (tag condition, weight delta) directly into automated Decision Tier 1.
  4. 90-Day Merchant Shadow Pilot: Deploying ReturnIQ in parallel with existing merchant review queues to measure live chargeback reduction.

Built With

Share this project:

Updates

Submission history