Cold Case

Inspiration

Give a damaged old photo to an AI restorer and it comes back perfect. The torn eye is sharp again, the missing cheek is filled in. But part of that face is gone forever: the AI didn't recover it, it invented it, and it never tells you where.

For a family album that's a curiosity. For a historical record it's a falsified one. The people who keep those records are already warning about it:

Genealogist Steve Little, republished by the Santa Barbara County Genealogical Society, showed a "restored" ancestor photo where the tool "generated a new, convincing-but-fake person." His verdict: "AI's version is a deepfake."
BYU's Family History Library teaches that "AI tools used for 'photo restoration' often fabricate facial features."
In August 2026, Indonesia's National Archives warned that AI-polished archival photos can amount to falsification.

And it's happening at scale: one consumer AI enhancer ran 1,000,000 family photos in its first 10 days (MyHeritage, 2020). Once an invented face is published, it becomes history. So we asked: what if a restorer had to tell you exactly which parts it made up?

Who it's for

An old photo is often the only record of a person, a place or an event. If a restorer invents part of it and nobody can tell, the record is changed for good. We built Cold Case for the people who can't afford that, and gave each of them a feature built for their job:

Who	How many	The risk today	What Cold Case gives them
Historical societies & archives	21,588 US history organizations, mostly volunteer-run (AASLH census, 2022)	A restored photo goes into the catalog and is quietly treated as the original	Batch mode: restore a box of scans at once, sorted riskiest (most invented) first, with a CSV catalog and XMP/JSON sidecars so each file's labels travel with it
Families & genealogists	1,000,000 photos through one AI enhancer in 10 days (MyHeritage, 2020)	An invented face is passed down as great-grandmother's real one	Every invented area is marked, and the likeness check compares a restored face against another photo of the same person
Newsrooms & fact-checkers	437 active fact-checking projects worldwide (Duke Reporters' Lab, June 2026)	A viral "restored" archive photo can't be traced back to what the scan really showed	Verify: drop in any file; it reads the signed receipt, catches pixels edited after signing, and finds the original case by pixel hash even if the receipt was stripped
Museums & libraries	35,000 active US museums (IMLS, 2014)	Visitors can't tell which parts of an exhibited restoration are history and which are a guess	Show the restoration beside its evidence map; document mode never generates a letter, so a repaired letter can't say something new
Courts, lawyers & insurers	Rule 707, a proposed US federal rule for machine-generated evidence (Aug 2025)	An AI-cleaned exhibit is challenged and nobody can say what the software changed	A full audit trail signed inside the file: every model, seed and setting, and 0 undamaged pixels changed, tested

And anyone with a shoebox of old photos.

Our first user is the volunteer digitizer at a small historical society: a box of donated photos, a flatbed scanner, and a shared library computer where installing software is blocked. That's why Cold Case is a free, open-source website: nothing to install, it works in any browser, and each volunteer keeps their own cases in an account.

What it does

Cold Case restores damaged historical photos and gives every pixel a verdict:

Real: untouched original, copied from the scan byte for byte
Rebuilt: repaired, and every repair attempt agreed
Invented: repaired, but the attempts disagreed, so this part is a guess (always striped, never shown by color alone)

Upload a scan, or open one of our eight public-domain cases. Our trained detector proposes where the damage is, and you check it in the mask editor:
- Click to fix selects a whole missed crack in one click.
- Standard / More / Most settings catch 68% / 83% / 90% of real damage.
- The receipt records whether a person edited the mask.
See the restoration on a before/after slider. Only damaged pixels change.
Switch on the evidence. Every pixel is labeled, with a live share of each. Hover any region to see its confidence and which stage changed it.
Ask "why is this Invented?" Click any red spot and Cold Case shows all eight repair attempts for that spot side by side, so you can see the AI disagreeing with itself.
See the faces that could have been. Where a face had to be invented, the most different versions are shown together instead of quietly picking one. Add a reference photo and the likeness check scores each one. Face fidelity has three steps, from "restore more" to "preserve", and the labels update with it.
Compare with a typical restorer on the same photo, with every undamaged pixel it changed highlighted in red.
Download the receipt: every stage, setting and region, signed into the image as a C2PA Content Credential. Anyone can check it later on the Verify page.

It handles three kinds of material with the same honesty:

Photo mode: restore, then label.
Document mode: no generative model ever touches a document. Paper is repaired from its surroundings, and any gap that touched writing is flagged as lost text. 0 letters generated.
Color mode: two independently trained colorization models paint the photo; wherever they disagree, the color is labeled a guess. The scan's own brightness is kept, and no color is ever labeled Real.

The site also has a 60-second tour, a "take it apart" view that splits one restoration into its five layers of evidence, and an Evidence page with every number below.

How we built it

The core idea: eight witnesses

A diffusion model paints from random noise, so each run (each seed) comes out different. We repair the same damaged region 8 times with 8 seeds. Where the surrounding photo strongly constrains the hole, all eight answers land in the same place. Where it doesn't, like a missing eye, they scatter. That disagreement is our signal.

For every repaired pixel, we measure how far the eight results spread in perceptual lightness (L* in CIELAB, a color space where equal distances look equally different to a person). Then:

Outside the damage mask → Real (never touched)
Inside the mask, spread ≤ 5 → Rebuilt
Inside the mask, spread > 5 → Invented

The threshold of 5 was calibrated on photos where we knew the true original.

The pipeline (five stages, each written into the receipt)

Detect damage. A U-Net with a ResNet-34 encoder, trained by us in PyTorch. We trained it first on our own procedural damage generator, which draws rips, crackle, flaking, scratches, foxing, creases and blur onto 617 clean Library of Congress photos. Then we fine-tuned it on 90 real damaged photos with expert masks (ARTeFACT dataset). Its mask is combined with Microsoft's old-photo scratch detector.
Restore × 8. SDXL Inpainting 0.1 on Apple silicon (MPS), cropped around each damaged region, with OpenCV Telea for hairline cracks and seamless blending. Every pixel outside the mask is copied straight from the scan.
Measure disagreement. Per-pixel spread across the eight results in CIELAB, measured before face restoration.
Label every pixel. The calibrated threshold turns spread into Real / Rebuilt / Invented. Faces also run through CodeFormer once per seed, with its fidelity weight exposed (0.3 / 0.7 / 1.0).
Sign the receipt. A manifest of inputs, mask source, seeds, settings and pixel counts, embedded in the PNG as a C2PA Content Credential with c2pa-python.

The website

A React + Vite + TypeScript + Tailwind front end over a Python API with a GPU job queue. Results are cached by a hash of the image and mask. The label map is drawn on a canvas over the compare slider, so hovering any pixel shows its label, confidence and stage. It has accounts, batch mode with CSV and XMP export, a Verify page, rate limits, security headers and a strict content security policy. The site runs on Render and keeps every saved case available even when our GPU is offline.

What we built vs. what we used

Part	Built or used
Damage segmenter, its damage generator, and its real-data fine-tune	Built
Click to fix and Standard / More / Most detection levels	Built
8-seed restoration, cropping, compositing, untouched-pixel guarantee	Built around SDXL inpainting
Disagreement map, calibrated threshold, region confidence, attempts viewer	Built
Faces that could have been, likeness check flow, typical-restorer comparison	Built
Receipt format, Verify, pixel-hash lookup	Built; signing via c2pa-python
Document mode and color-disagreement labels	Built
Website, API, job queue, batch mode, accounts	Built
SDXL Inpainting 0.1, CodeFormer, OpenCV Telea, Microsoft scratch detector, two colorizers (ECCV 2016, SIGGRAPH 2017), ARTeFACT data	Used

Results

Every number below is on the site's Evidence page and came from a script in our repo.

Measure	Result	How we measured it
Undamaged pixels changed	0 (a typical restorer: 29.7–61.1%)	Automated test with every model swapped for random noise (8 / 8 pass). The typical restore recipe was run on our 6 demo photos.
A barely damaged photo	Typical restorer repainted 61.1% of it; Cold Case 0	Tennis group, Funchal, c. 1905
Real damage found	61% (from 3%), IoU 0.426	40 real damaged photos with expert masks, never used for training. 59% of what it flags is real damage.
Catching more damage	Standard 68% · More 83% · Most 90%	Levels picked on 20 unseen real photos, scored on 20 others. The cost is a bigger mask (1.2×, 2.1×, 3.5× the true damage).
Do the labels mean something?	Invented pixels are 1.8× further from the truth	13.4 vs 7.3 ΔE, on photos with known originals
Documents: real damage found	55% (from 16%), IoU 0.467	46 real damaged documents held out from training. 0 letters ever generated.
Color mode	Brightness within 0.1 of 100; Lincoln 33% marked a guess	Lab lightness vs the restoration, and the share where the two color models disagree
The answer key	3 of 3 guesses marked Invented	We erased part of Lincoln's eye ourselves so the truth is known. The AI drew three confident eyes, none of them his.

The "typical restorer" is a representative recipe, not a named product: denoise and sharpen the whole frame, fill the damage, and run a face restorer on every face.

Challenges we ran into

The synthetic-to-real gap. Trained only on our synthetic damage, the detector found 3% of real damage. It had learned what our fake damage looks like. We fine-tuned it on 90 real expert-labeled photos and combined it with Microsoft's scratch detector: 61% found on 40 photos it never saw. That still misses about 4 in 10 damaged pixels at the Standard setting. So the mask is a proposal a person checks, the More and Most settings catch up to 90%, and the receipt records who drew the mask.
Disagreement is a signal, not a guarantee. Invented pixels carry 1.8× the error of Rebuilt ones, but the per-pixel correlation is modest (Spearman ρ = 0.29). "Rebuilt" means the attempts agreed, not that they're right, and the receipt itself says so.
Thin cracks are the hardest damage to detect and where the detector misses most.
Speed on one laptop. Eight SDXL runs per region is expensive. We crop around each region, use 20 diffusion steps (20, 30 and 40 were under 1.2% apart), run 8 seeds for demo cases and 4 for live uploads, and cache everything.
Proving a negative. "Never touches undamaged pixels" had to be tested, not promised.

Accomplishments that we're proud of

An honest restorer: every pixel has a verdict, and the verdict is measured, not decorative.
The answer key: we damaged a photo where we know the truth, and Cold Case caught every invented eye.
Beating a typical restorer where it matters most to an archive: 0 undamaged pixels changed, against 29.7–61.1%, even on a barely damaged photo.
Taking real-damage detection from 3% to 61% with real labeled data instead of hiding the number, and up to 90% with a person in the loop.
A feature for each user: batch for archives, likeness check for families, Verify for fact-checkers, an audit trail for courts.

What we learned

Synthetic training data can score beautifully and still fail on the real thing; ninety real labels mattered more than thousands of fake ones.
Diffusion randomness, usually treated as noise, is a useful measurement: where seeds disagree, the model is guessing.
Honesty is a product feature. Showing the weak numbers makes the strong ones believable.

What's next for Cold Case

Pilot with a local historical society: run a real donated box through batch mode and measure how much reviewer time "riskiest first" saves.
More real labels, starting with thin cracks, and (with consent) turning every human-corrected mask into training data.
Reference-guided faces: use a second photo of the same person to guide the restoration, not just check it.
An always-on host for document and color mode, which need no GPU.
A trusted signing certificate, so receipts show as verified in C2PA viewers (ours uses a test certificate today), and a commercially licensed face model to replace CodeFormer, which is non-commercial.

All demo photos are public domain or have no known copyright restrictions; credits are listed on each case in the app.