Inspiration

A returned package does not always contain the same product that was shipped. For small businesses, checking returns manually takes time, while mistakes can mean lost inventory or unnecessary friction for genuine customers.

SwapShield AI explores a practical question: can AI help merchants compare shipping and return evidence while keeping people responsible for the final decision?

What it does

SwapShield compares dispatch and return photos and provides a risk estimate, visual observations, and a recommendation for a merchant reviewer.

The workflow supports three outcomes: recommend approval, request human review, or ask for clearer photos when evidence is insufficient. It does not automatically reject refunds or accuse customers of fraud.

The application includes a review dashboard, a live image-upload workflow, and a printable reviewer report. Seeded examples let visitors explore the interface separately from live model inference.

How we built it

We combined pretrained vision models with a smaller classifier trained for the verification task:

  • DINOv2-small compares visual representations of the product images.
  • Qwen3-VL-4B extracts structured observations about visible similarities and discrepancies.
  • A logistic fusion classifier with Platt calibration combines signals into a calibrated risk estimate.
  • Quality checks request new photos when the evidence is insufficient.

The interface uses React and TypeScript, while a Python FastAPI service runs local inference. An optional API-based LLM auditor checks evidence consistency without changing the classifier’s score or recommendation.

Challenges we faced

Different camera angles can make matching products look different, while similar products can hide meaningful differences. Running the vision-language model within an 8 GB GPU memory budget also required quantization, image-size limits, and controlled inference.

Evaluation introduced another challenge: keeping product identities separate across training, validation, and testing to avoid overly optimistic results.

What we learned

A useful AI system needs more than a confident prediction. Evidence quality, uncertainty, calibration, and clear human controls matter just as much.

We also learned that requesting better evidence has a cost: it can protect against unsupported decisions while adding work for genuine customers.

Accomplishments

The locked evaluation reports 0.929 F1 on 30 held-out furniture image pairs, with two missed substitutions. However, 40% of pairs requested recapture. These are proof-of-concept results on a small benchmark, not evidence of production performance.

What’s next

We want to evaluate real merchant photos, improve difficult product categories, reduce recapture requests, and shorten inference time while preserving human oversight.

Built With

Share this project:

Updates

Submission history