Robust Detection of AI-Generated Images Under Transforms

Inspiration

A detector is only worth its accuracy on generators it has never seen.

Every shortcut available to us would have inflated the headline number and destroyed exactly that. An AIGC corpus assembled from public datasets is usually separable without looking at the picture at all — real photographs arrive as JPEGs, generated images as PNGs, and a model can score 97% by reading the container. So we spent the first day making our own data harder instead of easier, and the last day looking for the generators our own detector cannot see.

We found them. That negative result is the most useful thing in this submission.

What it does

It decides whether an image is AI-generated, and keeps deciding correctly after the image has been JPEG-compressed, blurred, resized, noised, colour-jittered or cropped — the transforms every upload and re-share applies.

On the organiser's demonstration benchmark — COCO val2017 reals against DALL·E 3 Advanced fakes:

condition	bAcc	AUROC	FPR@95TPR	ECE
clean	0.9891	0.9998	0.000	0.013
JPEG q30	0.9643	0.9931	0.024	0.025
blur σ2.0	0.9667	0.9967	0.012	0.035
resize 0.25×	0.9584	0.9909	0.032	0.029
noise σ0.10	0.9614	0.9938	0.026	0.032

Mean bAcc 0.968 · worst case 0.9584 · maximum degradation from clean 3.1 points. n = 13,843 per condition.

That benchmark is DALL·E 3, and the entire DALL·E family — 64,482 images — is denylisted from our training data by four independent gates. Those numbers come from a generator architecture the model has never seen.

The number we think matters most is not in that table. It is the gap between our in-distribution dev split, which shares all five generator families with training, and the unseen-generator benchmark: 1.0–2.5 points. A detector that had memorised "what FLUX and SD decoders look like" would fall off a cliff there.

Output is a calibrated probability, sigmoid((z + α) / T), with T, α and the decision threshold frozen before inference and carried inside the checkpoint. A decode failure is written to an error log and announced on stderr — never silently scored 0.5.

How we built it

Backbone. DINOv3-L/16 (facebook/dinov3-vitl16-pretrain-lvd1689m), 303.1M parameters — 15% of the 2B budget — fully fine-tuned end to end at 448 px. DINOv2-with-registers is wired as an ungated fallback so a licence-gate delay could never block the critical path.

Why end-to-end rather than a linear probe, measured on our own data: a frozen backbone with a trained head reaches 0.737 worst-case balanced accuracy; full fine-tuning reaches 0.9685. +23.2 points. The published figure we were working from predicted +26.7, so this settles the question empirically rather than by argument.

Crop, never resize. Training reads random 448 px crops at native resolution. Downscaling an image to fit a model destroys exactly the high-frequency evidence a detector depends on.

Distortion sampler. 30% of training images clean, 55% one corruption, 15% two, applied identically to both classes: JPEG, resize across four kernels, blur, noise, colour jitter, crop — each drawn slightly past test severity. MixUp, CutMix and hue jitter are banned by design; published evidence has them degrading this task badly.

Model selection optimises the worst case, not the average. The checkpoint is chosen by the minimum balanced accuracy across five conditions. A model that scores 0.999 clean and 0.85 under noise cannot win — which is exactly the model a mean-based metric would pick.

Development tools. VS Code, Claude Code and Codex CLI for implementation; git with CI on every push (331 tests); R with ggplot2 and dcurves for the calibration and decision-curve analysis.

Training ran on a single RTX 5070 Ti with 16 GB — a full end-to-end fine-tune of a 303M-parameter ViT-L at 448 px, 1.53 s/step, 14.4 of 16.3 GiB resident with about 1.9 GiB of headroom. 16 GB is the floor for this configuration rather than a comfortable fit, and we report the measured figure because it is what someone reproducing this needs to know. SLURM scripts for a university L40S cluster are in the repository as a fallback we planned for and did not need.

Models. DINOv3-L/16 as above; DINOv2-with-registers as the ungated fallback; Stable Diffusion 1.5 and SDXL VAEs used offline to build 49,986 reconstruction pairs from our own real images — a hard negative class that isolates decoder artifacts from semantic content.

APIs. Hugging Face Hub for pretrained weights. Replicate (Flux Dev) and Google / Azure image-generation APIs to generate the set B evaluation images. No inference-time API calls — the submitted model runs locally from a checkpoint.

Libraries and frameworks. PyTorch and torchvision; Hugging Face Transformers (pinned >=5.15.1,<5.16) and Diffusers; NumPy, pandas, PyArrow; Pillow and OpenCV; scikit-learn; imagehash for perceptual-duplicate screening; pytest; R with ggplot2 and dcurves.

Datasets and assets.

Role	Source	Rows
real	COCO train2017	118,287
real	WildFake (afhq, celebahq, church, ffhq, imagenet, laion5b)	48,000
real	SID_Set	39,821
fake	WildFake non-DALL·E families, 20k per family	119,999
fake	SID_Set FLUX	40,000
fake	VAE reconstructions of our own COCO reals (SD1.5 + SDXL)	49,986

Plus 200 images we created ourselves for error analysis: 100 platform-licensed real photographs (Unsplash) curated into three false-positive-magnet categories, and 100 generated images across gpt-image-2, Gemini 3 Pro, Gemini 3.1 Flash and Flux Dev, with a provenance manifest recording generator, prompt, and declared watermark status for every one.

Excluded by our own rule: COCO val2017 and all 64,482 WildFake DALL·E images are never training, model-selection, or calibration data. Four independent gates enforce it — generator family, source dataset, path key, and content/perceptual hash — so a renamed file cannot slip through. Manifest building refuses protected rows outright; training aborts if the manifest trips any gate; the evaluation CLI demands an explicit flag and refuses to repeat a completed run.

The rules do not require this. The benchmark does not count toward the score, and nothing forbids reporting results on it. We imposed it because a number obtained by tuning against the benchmark measures nothing — and we disclose in the repository that two exploratory reads overran our own one-run budget, rather than deleting the artifacts before submission.

Challenges we ran into

The shortcut we had to disarm before training. Our corpus was separable without looking at the picture: a probe reading only the JPEG quantization table scored 0.974 balanced accuracy. A model trained on that data would learn "PNG means fake", report 97% on its own test split, and collapse on anything real. We equalise container statistics with a seeded plan that never sees the label, then equalise file size by stratified selection, and we gated training on four shortcut probes all scoring below 0.60:

probe	raw corpus	after treatment
JPEG quality	0.974	0.500
dimensions	0.732	0.500
file size	0.589	0.497

One residual is documented rather than hidden: a real photograph arrives already JPEG-compressed and that cannot be undone, so total compression history and final-encode quality cannot both be equalised. We equalise the final encode, because that is what the probe reads.

A second training epoch made the model worse, and there was a real bug behind it. Epoch 2 improved every in-distribution condition — worst-case +0.60 on dev, 14 wins and 1 tie across 15 transforms — and lost 4.5 to 11.6 points on held-out generator families. Following our own written selection metric would have shipped the worse model.

The bug: the augmentation RNG had no epoch term, so epoch 2 replayed epoch 1's byte-identical crops and distortions. No new diversity, only a tighter fit to the training families.

So we fixed it and ran the epoch again from the pre-boundary checkpoint, under a gate written in advance: adopt it only if it beats epoch 1 on held-out families with non-overlapping confidence intervals. Three-way result, held-out families, n = 4,000 per condition:

condition	epoch 1	epoch 2 (buggy)	epoch 2 (fixed)
clean	0.8703	0.8045	0.8295
JPEG q30	0.8313	0.7205	0.7415
resize 0.25×	0.9158	0.7995	0.8335
noise σ0.10	0.8075	0.7628	0.8028
worst case	0.8075	0.7205	0.7415

The gate failed, and the experiment paid for itself anyway by splitting the damage in two. Fresh augmentation draws recovered 2–4 points on every condition, about a quarter of the gap — that is what the bug cost. The remaining 6.6 points is the second pass itself. Temperature settles it: the buggy epoch needed T=1.519 and the fixed one T=1.496, nearly identical and both far from epoch 1's 1.369 — so the confidence-scale drift is a property of training a second epoch on the same five generator families, not of the bug. Epoch 1 stands.

Accomplishments that we're proud of

We wrote down what would count as a failure, then went looking for it. Before scoring anything we built two 100-image evaluation sets designed to attack our own model from both sides, and committed three pass/fail criteria to the repository before the images were scored. The commit that fixed the criteria and the commit that recorded the scores are 38 hours apart in the git history, so the ordering is verifiable by anyone, not a claim we make about ourselves.

Two of the three resolved against our expectations. That is the point of fixing them in advance.

The expensive error in AIGC detection is not the missed fake, it is the false positive — a real photograph flagged as synthetic. It falls hardest on the photographers whose work is most processed: studio portraits with smooth lighting, landscapes with heavy HDR grading, minimal compositions with little structure to read. Set A is 100 real photographs curated along exactly those three axes.

Criterion: if false-positive rate on set A exceeds 5% at our frozen threshold, that is the headline finding. It did not fire — 2/100 at native resolution, 1/100 downscaled. And the two false positives turned out not to be about photography at all: both collapse from 0.80 and 0.70 to below 0.005 when brought to the generated set's scale. They were reporting resolution. The studio-portrait category produced no false positive in either variant.

And then the finding: our benchmark score does not transfer to 2026 consumer models.

Set B includes 25 images from a model with no watermark and no provenance manifest at all (Flux Dev) alongside 75 that carry vendor watermarks — because without a no-watermark control group a strong result would be unattributable. We could not tell whether the detector found the generator's fingerprint or a watermark those 75 images all share.

set B slice	detected at frozen threshold	median score
Flux Dev — no watermark, seen family	25/25	0.9993
gpt-image-2 (Azure)	0/30	0.0038
Gemini 3 Pro Image	3/25	0.0076
Gemini 3.1 Flash Image	2/20	0.0065

Criterion: if the watermarked 75 and the Flux 25 differ by more than ~5 points, the detector may be reading watermarks rather than generators. It fired at 93.3 points — in the opposite direction from the mechanism we were guarding against. A watermark-reading detector would find the watermarked images easy. Ours finds them nearly invisible. That refutes watermark-reading and localises the failure precisely:

it is not the transforms — the 15-condition matrix holds
it is not unseen-generator fragility as such — DALL·E 3 is also unseen and holds 0.9891 clean
it is the 2026 generation of consumer endpoints, producing mundane 1–1.6 MP PNGs

These are not threshold-marginal misses. The missed images score around 0.005 where Flux scores 0.999, so ranking fails along with the operating point — no threshold adjustment recovers them. Not one of the 30 gpt-image-2 images reaches even a tenth of our threshold.

We do not claim to know the mechanism. Candidate explanations, none proven here: decoder architectures our 2023–24 training families do not span; vendor-side post-processing overwriting the high-frequency traces the detector reads; the mundane-curation brief removing semantic tells.

What we would ask a reviewer to take from this: the 0.958–0.989 on the organiser's benchmark is real and reproducible, and it is not a claim about 2026 frontier models. We found that boundary ourselves, on our own adversarial set, and we are reporting it rather than leaving it for someone else to find.

What we learned

An in-distribution validation split cannot detect overfitting to the generators it shares with training. It will actively reward it. We measured this twice independently — the buggy epoch and the fixed one both beat epoch 1 on dev while collapsing on held-out families. This is why we built a held-out-families split, and why the decision went against our own written metric. It is recorded as a dated deviation with its rationale, not quietly overridden.

Writing the acceptance criteria down first changes what you are allowed to conclude. Criterion 2 fired backwards. Because it was fixed in advance, that reversal is reportable as a finding instead of a story we reverse-engineered from the data.

A measurement whose expected answer is zero needs a positive control. "Nothing detected" and "the detector is broken" produce identical output. This bit us once during provenance analysis and is now a rule we apply to every null result.

Calibration does not transfer to unseen generators. Expected calibration error rises from 0.013 in-distribution to 0.10–0.19 on held-out families. This is the price of fitting temperature once on a deployment mixture and never refitting per transform — refitting per condition would be tuning on the evaluation conditions, which we consider indefensible.

What's next for Robust Detection of AI-Generated Images Under Transforms

Age the training corpus. The 2026-endpoint finding is the first thing more time would go to, and it is a data problem rather than a method problem: our fake classes are 2023–24 generators, and the consumer endpoints people actually use in 2026 are a generation past them. The recipe stays; the corpus ages. This is also the honest general lesson — a detector's training corpus has a shelf life, and ours is the shortest-lived part of this submission.

Determine the direction of miscalibration. We now know the sign on held-out data — the model systematically under-predicts P(AIGC), with every reliability point above the diagonal — but we have not yet established whether that holds across deployment mixtures, which is what decides whether the drift costs false positives or false negatives in practice.

Test whether the crop policy costs us anything. Training on 448 px native crops rather than resized images bought +6–15 AUC, but it excludes four low-resolution generator families from training entirely. A second model at 224 px would tell us whether that trade was right.

All of it is written up in track5/KNOWN_LIMITATIONS.md — eleven entries, ordered by how much each should move a reader's confidence, with the 2026-endpoint finding noted as belonging near the top despite being numbered last.

Demo video: https://youtu.be/HoKdBR0hjNc Repository: https://github.com/ning-bao/TechJam


Bao Ning	Repository, training pipeline, model, evaluation harness, calibration, CI. All GPU work.
Cai Haitong (lead)	Problem framing, plan and interface contracts, protected-set discipline, independent verification of deliverables, evaluation-protocol review. Authored the submission README, KNOWN_LIMITATIONS, and deliverables 4 and 5.
Xiong Yuxuan	Hard-case datasets: 100 curated real photographs and 100 generated images across four models, with provenance manifest and validation scripts.
Zhang Xinghan	R analysis pipeline for calibration curves, ECE, decision-curve analysis and threshold drift.
Yang Zihao	Degradation estimator (DCT-based blind JPEG quality-factor estimation); provenance-survival measurement of C2PA declarations under the 15 transforms.