Refract

Autonomous return fraud decisioning agent that evaluates risk with conformal AI and auto-executes PayPal settlements.

Comment

Inspiration

E-commerce return fraud and abuse cost merchants over $100 billion annually. For online retailers, handling returns is a lose-lose balancing act:

  • Block aggressively, and you insult high-value, loyal customers.
  • Approve blindly, and you bleed margin to serial abusers and organized fraud rings.

Most fraud solutions stop at giving a static risk score and leaving the rest to overloaded human ops teams. We built Refract to bridge this gap: an autonomous return-risk decisioning agent that doesn't just predict risk, but safely executes the appropriate financial action in PayPal (instant refunds, dispute fraud-flagging, customer restrictions, or human escalations).

What It Does

When an e-commerce return request arrives, Refract runs a multi-stage decision pipeline in under 50ms:

  1. 3-Class Risk Classification: Instead of binary "fraud or not", Refract classifies returns into Legitimate, Abuse (policy gaming / serial wardrobing), and Fraud (stolen devices, identity rings, synthetic accounts).
  2. Conformal Prediction (RAPS): Machine learning models often produce overconfident, hallucinated probabilities. Refract uses Split Conformal Prediction (MAPIE) to output mathematically guaranteed prediction sets. If an edge case is ambiguous (e.g., set contains both Abuse and Fraud), Refract quantifies the epistemic uncertainty rather than guessing.
  3. CLV-Weighted Cost Decision Engine: The agent computes the expected financial cost of each action weighted by the customer's Customer Lifetime Value (CLV). A VIP customer with a high CLV is handled differently than an anonymous burner account on a shared IP subnet.
  4. Bounded Gemini AI Critic: A Gemini agent acts as an automated second opinion. It reviews the structured return signals, CLV trade-offs, and top SHAP feature drivers. If the statistical model recommends an aggressive action that conflicts with merchant context, Gemini can escalate the case to human review—with a strict guardrail preventing it from ever downgrading hard fraud blocks.
  5. Autonomous PayPal Execution: Once decided, Refract calls PayPal's APIs directly:
    • APPROVE -> Executes instant refund via PayPal Payments/Refunds API.
    • VERIFY_AND_BLOCK -> Accepts/flags claim via PayPal Disputes API.
    • HUMAN_REVIEW -> Routes into an actionable Dispute Inbox with full AI reasoning and SHAP attributions.

Workflow (Image Added in the Media Section)

  1. Ingestion & Feature Engineering: Live return requests or PayPal webhook events are ingested by FastAPI and transformed into 33 behavioral features (purchase velocity, delivery-to-return delta, device sharing, and lifetime return rates).
  2. Calibrated Machine Learning: A 3-class CatBoost model evaluates multi-class risk (Legitimate, Abuse, Fraud), followed by Temperature Scaling to align raw logits with empirical probabilities.
  3. Uncertainty Quantification: MAPIE Conformal Prediction (RAPS) computes statistical prediction sets with a 90% coverage guarantee. If an edge case is ambiguous (e.g., set contains both Abuse and Fraud), it mathematically flags epistemic uncertainty rather than guessing.
  4. Economic Decision Engine & Gemini Critic: Decisions are passed through a Customer Lifetime Value (CLV) cost matrix. Concurrently, Google Gemini reviews the local SHAP attribution drivers and CLV trade-offs as a bounded second opinion, producing plain-English reasoning for the merchant and escalating high-uncertainty returns.
  5. Autonomous PayPal Execution: Safe returns trigger instant refunds via the PayPal Payments API (/v2/payments/captures/.../refund), confirmed fraud triggers dispute flags via the PayPal Disputes API (/v1/customer/disputes), and escalated cases are held in the interactive Dispute Inbox for one-click human resolution.
  6. Cloud Infrastructure: The entire pipeline runs cloud-natively on Render via an automated Blueprint (render.yaml) with durable retry workflows and MongoDB Atlas for full audit logging.

How I Built It

  • Machine Learning & Uncertainty: Python, CatBoost (3-class classifier), Temperature Scaling for probability calibration, and MAPIE for RAPS conformal prediction sets.
  • Weak Supervision: Snorkel-based programmatic labeling functions, eliminating the need for expensive manual labeling.
  • Explainability: SHAP tree explainers for local feature attribution paired with Gemini 1.5 Flash for natural-language merchant narratives.
  • Backend & Commerce Engine: FastAPI, Motor (async MongoDB Atlas driver), and a native Python PayPal REST SDK integrating Sandbox Refunds, Disputes, and Webhooks.
  • Cloud Architecture & Workflows (Render):
    • Render Blueprint (render.yaml): Fully automated, declarative Infrastructure-as-Code deploying Refract's services for $0/mo.
    • Render Workflows (api/workflow.py): Multi-step durable execution pattern with exponential backoff retries for safe, idempotent PayPal settlement.
    • Zero-Downtime Health Probing: Built-in health check probes on /health.
  • Frontend Dashboard: A standalone, responsive dashboard featuring a real-time WebGL/Three.js data visualization, Server-Sent Events (SSE) live stream, Pareto risk-appetite simulation, and an interactive Dispute Inbox.

Challenges I Faced

  1. Class Overlap & Concept Drift: Real return abuse looks remarkably similar to legitimate returns until purchase velocity and delivery timing are analyzed across historical windows. We injected realistic concept drift into our test data and designed calibrated labeling functions to separate behavioral abuse from technical fraud.
  2. Preventing Autonomous Model Failures: Handing monetary decisions to an AI model is dangerous if the model is overconfident. Conformal prediction solved this: when the model's confidence drops below the error budget, the prediction set widens, penalizing the automated actions and cleanly routing the case to the Human-in-the-Loop Dispute Inbox.
  3. Idempotency in Agentic Commerce: Ensuring that network retries or webhook duplicates would never trigger duplicate refunds in PayPal required strict transaction-keyed tracking across MongoDB and workflow task states.

What I Learned

  • Accuracy is the wrong metric in fraud: An 85% accurate model with an economic cost matrix that protects merchant CLV dramatically outperforms a 95% model that blindly blocks loyal VIPs.
  • LLMs shine as bounded critics, not unconstrained deciders: Using CatBoost + Conformal Prediction for the statistical heavy-lifting and placing Gemini as a bounded critic provided the ideal combination of mathematical guarantees and contextual business reasoning.

What's Next for Refract

  • Direct Shopify and WooCommerce app extensions to plug into merchant checkouts.
  • Online adaptive retraining on verified human resolution labels from the Dispute Inbox.
  • Support for automated partial refunds and return shipping label generation via PayPal Partner APIs.

Built With

Share this project:

Updates

Submission history