SplitShield

Duplicate images leaking between train and test make model accuracy look better than it is. SplitShield finds the leakage, shows the evidence, repairs the split, and measures the difference.

Comment

Inspiration

Every ML practitioner has seen a model score suspiciously well — only to discover the test set contained images the model had already trained on. Sometimes byte-identical, sometimes a recompressed or cropped copy. Tools exist to find duplicates, but the workflow that matters — reviewing the evidence, repairing the split without wrecking the test set, and honestly quantifying what the leakage was associated with — is usually a pile of ad-hoc notebooks. We built that workflow as a product.

What it does

Upload an image-classification dataset as a ZIP (YOLO-style detection layouts work too) and SplitShield:

  • Detects leakage three ways — exact duplicates (SHA-256), near duplicates (64-bit DCT perceptual hash, exhaustive Hamming search), and appearance-similar pairs (embeddings + cosine top-k search) — each with split-aware severity and deliberately cautious confidence language ("Confirmed exact duplicate" vs "Requires human review").
  • Explains the evidence — an Evidence Explorer shows every suspicious pair side by side with its similarity metrics; a human confirms, clears, or flags each finding, and those decisions persist without ever overwriting the algorithmic evidence.
  • Scores integrity transparently — a documented, deterministic 0–100 Dataset Integrity Score with a full penalty breakdown. No LLM, no black box.
  • Repairs the split — a proposed manifest that consolidates duplicate groups into single splits, protects the test set, explains every move, and exports as CSV/JSON. Original files are never modified.
  • Quantifies the damage — a seeded diagnostic classifier is scored on the original vs leakage-quarantined test set, reporting the Observed Evaluation Gap with bootstrap confidence intervals and explicit anti-overclaiming warnings.
  • Exports a reproducible audit — printable report with dataset fingerprint, seed, config, and package versions, so anyone can re-run and verify.

How we built it

FastAPI backend running a staged pipeline (validate → index → exact → perceptual → embedding → score → evaluate → report) with every stage persisted to SQLite; Next.js frontend renders dashboards, evidence review, repair, and reports from the computed results. Ingestion is hardened: file-signature validation, zip-traversal and zip-bomb guards, random internal IDs, configurable retention with one-click deletion. Everything runs on CPU — no GPU required.

Challenges

  • No pretrained weights in our build environment — the network blocked model downloads, so we designed a pluggable embedding backend: MobileNetV3 when available, otherwise a classical appearance descriptor (contrast-normalised structure + illumination-invariant chromaticity + gradient histograms) that we validated against transformed-duplicate cases — and the UI always discloses which backend ran. No fake "semantic AI" claims.
  • Honest metrics are hard — we deliberately renamed our headline metric to "Observed Evaluation Gap" and gate the experiment behind eligibility checks, because removing test samples changes the population and correlation isn't causation.
  • Real datasets are messy — supporting detection-style layouts, conflicting labels, corrupt files, and 700 MB uploads without crashing took as much engineering as the detection itself.

Accomplishments & what we learned

A deterministic synthetic demo dataset with a ground-truth manifest verifies the detectors end to end (89 automated tests + a browser E2E test). The whole system is deployed and public. Biggest lesson: the credibility of a data-quality tool lives in its language — saying exactly what the evidence supports, and no more.

What's next

Video frame sampling, filename/CSV label mapping, FAISS for very large datasets, and side-by-side audits of popular public datasets — several of which are known to contain real leakage.

Existing tools, acknowledged

Duplicate detection itself isn't new — FiftyOne, CleanVision, and Cleanlab do it well. SplitShield's contribution is the complete detect → explain → review → repair → quantify → export workflow in one auditable tool.

Built With

Share this project:

Updates