-
-
AUC of different models
-
architecture and approach
Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
Inspiration
The brief asked for a detector that survives compression, blur, and cropping — so before writing code I went looking for prior art, and found this exact task had run as a CVPR 2026 challenge (NTIRE, arXiv:2604.11487) with the winning methods published. Not one of them invented a new architecture. All nine top teams did the same thing: a large pretrained vision model, frozen, with a small classifier on top, trained on deliberately degraded images.
That reframed the problem. Robustness looked like a data property, not a model property. My question became: that worked with 42 generators and a cluster — does it hold with one dataset and one GPU?
What it does
Point it at a folder of images, get a JSON with an AI-generated probability per image. It holds accuracy under JPEG re-encoding, blur, downscaling, sensor noise, cropping, and colour jitter.
How I built it
Frozen SigLIP + linear head. ViT-SO400M-14-SigLIP-384 with every backbone
weight frozen — only the classifier trains. It already knows what photographs
look like, and freezing it means it cannot memorise a 100k-image dataset.
Degradation as augmentation. Every training image is damaged before the model sees it, using the same transform families the evaluation uses.
Three models, not one. Baseline, +augmentation, +pairwise consistency loss — each trained from scratch so the ablation isolates one variable at a time. Roughly 50 min per run on an H100.
Evaluated on two axes. A 15-cell transform × severity grid on 20,000 CIFAKE test images, and an out-of-distribution probe of 25 personal photographs against 25 images from Gemini and ChatGPT.
Results
| Clean AUC | Robust AUC | Gap | OOD AUC | |
|---|---|---|---|---|
| Baseline | 0.9263 | 0.8972 | 0.0291 | 0.637 |
| + augmentation | 0.9304 | 0.9276 | 0.0028 | 0.589 |
| + pairwise loss | 0.9310 | 0.9277 | 0.0033 | 0.549 |
Augmentation delivered the entire robustness gain, $+0.0303$ AUC, and collapsed the clean-to-degraded gap by a factor of ten. It also cost nothing on clean images. The pairwise loss added $+0.0002$ — nothing — because augmentation had already solved the problem it targets.
What I learned
Augmentation helps most exactly where the model is weakest. Its largest gain was $+0.1031$ on Gaussian noise at $\sigma=0.1$, the transform the baseline had never encountered, against $+0.0075$ on JPEG q=90. Spread across the fourteen degraded conditions fell from $0.104$ to $0.021$.
AUC and accuracy fail differently. On the demo set, degradation shifted every score down by $0.249$ on average while leaving the ordering intact — AUC held, but accuracy at a fixed 0.5 threshold fell from 16/20 to 13/20. A detector can keep detecting and still make the wrong call, if nobody moves the threshold.
Measuring the second axis changed what I ship. A paired bootstrap over 10,000 resamples showed my best-scoring model is significantly worse out of distribution than the baseline it improved on: $-0.088$, 95% CI $[-0.163, -0.018]$. Every technique that bought robustness cost generalisation. So I ship the middle checkpoint, not the top-scoring one.
The honest number matters more than the headline. At the default threshold the detector catches 93.8% of AI images — and wrongly flags 21.5% of real photos. Hold false positives to 1% and recall drops to 30.4%.
Challenges
The evaluation was broken before the model was. My first results looked like the model had never learned — clean AUC 0.51, accuracy pinned at chance. I misdiagnosed it twice (device placement, then inverted labels) and nearly retrained on a GPU I was paying for. The real bug: the dataset returned already-normalised tensors and the evaluation normalised them again, then for degraded cells converted a normalised tensor back to PIL — producing garbage. Training had been fine the whole time.
Compute. Three epochs on a MacBook CPU projected to 44 hours. Moving to an H100 brought it to ~50 minutes per run.
Deciding to report the failure. The generalisation result is bad: AUC 0.549, indistinguishable from chance, 100% false positives on real photographs. I could have shown only the robustness number. But it is the more useful finding, and it has a clear cause — CIFAKE is 32×32 upscaled, so the model learned what an upscaled thumbnail looks like, not what a photograph looks like.
What's next
Generator-diverse and resolution-diverse training data. That is what the challenge winners had and I did not, and it addresses the failure directly.
Log in or sign up for Devpost to join the conversation.