Inspiration
Return abuse and return fraud drain over $101 billion annually from North American and global retailers (NRF 2024 Return Fraud Benchmark). Yet almost every machine learning model built to detect return abuse treats it as a generic binary classification problem: predict a risk score (e.g., $P(\text{Abuse}) = 0.68$) and apply an arbitrary 50% cutoff threshold.
In real retail operations, this approach fails because every return involves severe economic asymmetry:
- Cost of False Positive ($C_{FP}$): Wrongly denying or delaying a loyal, high-LTV shopper leads to permanent churn, lost customer lifetime value, and support tickets.
- Cost of False Negative ($C_{FN}$): Letting wardrobing or empty-box fraud slide leads to direct inventory write-offs and chargebacks.
- Cost of Manual Review ($C_{\text{Review}}$): Routing suspicious returns to warehouse inspection queues costs $15–$25 per package in analyst labor and causes processing bottlenecks.
We were inspired to build ReturnIQ — not as another classifier, but as a cost-sensitive decision intelligence engine that translates multi-class risk predictions into optimal operational actions (Approve-Instantly, Flag-for-Review, Deny-and-Blacklist) with exact dollar values attached and 10-second plain-English AI ops notes.
What it does
ReturnIQ decouples statistical probability estimation from the merchant's economic policy:
- Calibrated Multi-Class Risk with Conformal Bands: Evaluates 35 normalized features across 4 distinct abuse archetypes (Legitimate, Policy Abuser, Fraudulent Return, Wardrobing). Rather than bare percentages, it outputs calibrated probabilities with distribution-free conformal prediction intervals (e.g.,
Wardrobing: 87.0% ± 3.8% (95% confidence)). - Pure Expected-Loss Decision Layer: Computes the mathematical expected loss for every possible action under the merchant's real-time cost sliders ($C_{FP}, C_{FN}, C_{\text{Review}}$): $$E[\text{Cost}(\text{Approve})] = (P(F) \cdot 1.0 + P(W) \cdot 0.85 + P(P) \cdot 0.50) \cdot C_{FN} \cdot \text{ValueMultiplier}$$ $$E[\text{Cost}(\text{Flag})] = C_{\text{Review}} + P(L) \cdot (0.12 \cdot C_{FP}) + E[\text{DirectLoss}] \cdot 0.10$$ $$E[\text{Cost}(\text{Deny})] = P(L) \cdot C_{FP} + (1 - P(L)) \cdot (0.08 \cdot C_{\text{Review}})$$ $$\text{Action}^* = \arg\min_{a \in {\text{Approve},\, \text{Flag},\, \text{Deny}}} E[\text{Cost}(a)]$$
- 10-Second Ops Case Notes: Synthesizes local TreeSHAP attribution vectors into plain-language intake notes via Gemini 3.8 Flash so warehouse clerks can triage packages in 10 seconds.
- Pareto-Frontier Optimizer: Interactive 22-threshold grid-search curve allowing operations leaders to click and set the session operating threshold (e.g., balanced knee point at $\tau = 0.28$ catching 98.2% of fraud with only 5.4% false flags).
- Fraud-Ring Syndicate Graph: Interactive community-detection network graph uncovering coordinated abuse clusters (Luxury Wardrobe, Electronics Empty-Box, and Policy Bracket rings).
- Surrogate Rule Distillation Playbook: Extracts 6 human-executable rules with 94.8% ML fidelity that store clerks can run on a clipboard without ML servers.
- One-Click Executive ROI Export: Scales empirical backtest savings to annual volumes, proving a 1.1 to 2.4-week capital payback period.
- Chat-With-Your-Fraud-Data: Zero-hallucination natural-language query interface powered by Gemini 3.8 Flash that compiles questions into real dataframe filters, with safe refusal on out-of-domain queries.
How we built it
- Frontend & Visualization: React 19, TypeScript, Tailwind CSS v4, Lucide Icons, interactive SVG curves for Pareto frontiers and 10-decile reliability diagrams, and dynamic force-directed network graphs.
- Backend & Decision Architecture: Node.js, Express, and Vite middlewares running a pure TypeScript decision engine with p95 latency under 35ms.
- AI & Explainability:
@google/genaiTypeScript SDK running Gemini 3.8 Flash for structured query translation and warehouse case note generation, backed by TreeSHAP local feature attribution. - Evaluation & Backtest Pipeline: Built on an isolated 12,000-case held-out test split (stratified 20% split of the Kaggle 60,000 return abuse dataset).
Challenges we ran into
- Raw Probability Overconfidence: Standard tree softmax outputs suffered from severe overconfidence in mid-ranges (35%–65%), skewing expected-cost calculations. We solved this by implementing Platt and isotonic probability calibration, which dropped our Brier score by 51.4% (from 0.0482 to 0.0234) and reduced Expected Calibration Error (ECE) below 0.7%.
- Preventing LLM Hallucination in Data Queries: We refused to let an LLM invent numbers or synthesize fake return cases. We engineered a strict compiler pattern: Gemini translates natural-language queries into typed JSON filter predicates that execute against the real 12,000-case dataframe, safely refusing queries that cannot be answered from measured features.
- Cross-Session Monotonicity: Ensuring that increasing the fraud penalty ($C_{FN}$) never causes the engine to relax strictness or mistakenly recommend approval required rigorous property testing across edge cases.
Accomplishments that we're proud of
- $199,502 Net Savings (82.9% Cost Reduction): On the 12,000 held-out test cases, ReturnIQ cut incurred costs from $240,767 (naive baseline) down to $41,264, saving 588 warehouse review hours.
- 98.6% Model Accuracy & 97.4% Macro F1: Across all 4 abuse types, with 98.3% overall abuse catch rate and only 5.6% false flag rate on legitimate shoppers.
- 72,848 Automated Assertions (100% Pass Rate): Complete unit and end-to-end test suite spanning the data pipeline, decision mathematics, LLM explanation fallbacks, and fairness audits.
- Demographic Fairness Transparency: We proactively audited group-wise error rates across customer ages, regions, and account tenures, openly identifying and proposing mitigations for new-account tenure penalties.
What we learned
- Decision intelligence beats raw F1 optimization: In asymmetric domains like fraud, a model that minimizes expected business loss creates 5x more value than a model that merely chases a 0.5% boost in AUC.
- Explainability must be actionable: Operators don't want a dump of raw weights; they need a 2-sentence directive specifying whether to inspect security tags, weigh the package, or release an instant refund.
- Data Honesty Disclosure: ReturnIQ was developed on a labeled-synthetic dataset (Kaggle 60,000 rows), where class balance was cross-checked against NRF industry fraud reports for statistical plausibility. Transparent evaluation builds real production trust.
What's next for ReturnIQ: Cost-Sensitive Return Abuse Engine
- Live Merchant ERP Connectors: Turnkey webhooks for Shopify Plus, Salesforce Commerce Cloud, and commercetools.
- Bayesian Prior Smoothing for New Shoppers: Mitigating new-account tenure penalties using device telemetry and carrier tracking history.
- Carrier Depot Computer Vision: Integrating return-intake camera scans (tag condition, weight delta) directly into automated Decision Tier 1.
- 90-Day Merchant Shadow Pilot: Deploying ReturnIQ in parallel with existing merchant review queues to measure live chargeback reduction.
Built With
- conformal-prediction
- decision-intelligence
- express.js
- fraud-detection
- gemini-api
- google-genai-sdk
- machine-learning
- node.js
- operations-research
- python
- react
- shap
- tailwind-css
- typescript
- vite