-
-
Classify a garment: your image through all three models, routed by the simulated sorting gate (current app).
-
Live bench: untouched test images, truth vs all three models, with an abstain slider (current app).
-
Evidence: measured results on the untouched 10,000-image test set, confusion matrix and per-class F1 (current app).
Inspiration
Most first vision models get shipped on one number: accuracy on a clean test set. But a model that is 91% accurate can still be confidently wrong exactly when the input shifts a little. I wanted a tool that makes that visible instead of hiding it - something a curator or researcher can use to decide whether a clothing-image classifier deserves trust, and to see precisely when it does not.
What it does
FashionShift is a browser-local workbench for three models I trained on Fashion-MNIST: logistic regression, a small CNN, and the same CNN trained with augmentation. The models drive the main workflow end to end - classify a garment and route it through a simulated sorting gate - and every audit tool stays one tab away.
- Classify a garment: drop in any PNG or JPG, watch the exact preprocessing the models see, get predictions from all three, then route the garment through a simulated sorting gate - auto-sort when confidence clears your threshold, manual review otherwise, with per-model routing shown. Batch-import a CSV of 784-pixel rows with optional true labels. Session history stays in your browser.
- Live bench: random images from the untouched 10,000-image official test set, all three models predicting in your browser. An abstain slider shows coverage vs risk - grey out the low-confidence calls and watch the error rate change.
- Evidence: clean accuracy, macro-F1, per-class F1, confusion matrix, calibration (reliability curve, ECE, Brier, NLL), and a failure gallery of the worst confident mistakes.
- Stress lab: blur, occlusion and translation applied to inputs. Augmentation helps a lot under translation (+22 points) but does NOT help under blur or occlusion - the workbench shows that honestly instead of cherry-picking.
- Model card: export model_card.md and metrics.json.
- The app tells you up front what it cannot do: it knows ten classes of 28x28 grayscale catalog photos. A phone photo or a sketch is out of distribution, and the app shows the models producing confident nonsense there - that exposure is the point.
How I built it
Models trained in PyTorch (seeded, 50k/10k split of the official training set; the official 10k test set stayed untouched until evaluation). The app is a single self-contained HTML file with hand-written JavaScript inference - I exported the trained weights and re-implemented forward passes in JS, with an automated parity test proving the browser math matches PyTorch to 1e-6 across 1,440 probabilities. No server, no account, no telemetry.
Challenges
Making in-browser inference bit-faithful to PyTorch (softmax numerics, layout, preprocessing order), and designing stress tests that reveal real weaknesses instead of confirming what I hoped - the augmentation result genuinely surprised me.
Accomplishments
CNN reaches 91.1% on the untouched test set; the whole audit - metrics, failures, stress, live inference - runs offline in one file; JS/PyTorch parity is machine-checked; the negative results (augmentation failing under blur/occlusion, 0.60-0.75 confidence on wrong calls) are displayed, not buried.
What I learned
Calibration and abstention change how a model should be used far more than a point of accuracy, and data augmentation is not a free robustness win - it trades clean accuracy for shift-specific gains.
What's next
Richer stress families (rotation, lighting), per-class abstention thresholds, and import-time distribution-shift scoring.
AI disclosure
Built with AI assistance (design iteration, code generation, demo narration) directed and verified by me. All reported numbers come from the actual trained models and are reproducible from the repo.
Demo video
The demo video below was recorded for the original submission and shows its earlier interface; the gallery screenshots show the current app. The sorting-gate rework kept the same trained models, data and metrics.
Built With
- computer-vision
- javascript
- machine-learning
- python
- pytorch