Robust AIGC Detection Under Real-World Transformations

Inspiration

Every AI-image detector I read about reported something like 95% accuracy. Then I looked at how those numbers were produced: pristine images, straight out of the generator, never touched.

That is not how images reach anyone. By the time something lands on a platform it has been re-encoded by a messaging app, downscaled into a thumbnail, cropped to fit a profile frame, run through an auto-enhance filter. Every one of those operations destroys fine detail, and fine detail is exactly where generation artifacts live.

So the question I actually wanted answered was not "can a model detect AI images" but "how much of that detection survives contact with the internet."

What it does

detect.py takes a directory of images and writes a JSON file with a confidence score for each one, from 0.0 (real) to 1.0 (AI-generated), plus a label applied at a threshold selected on validation data.

python detect.py --input ./images --output results.json

[
  {"image_path": "images/photo1.jpg", "pred": 0.8734, "label": "ai"},
  {"image_path": "images/photo2.jpg", "pred": 0.0121, "label": "real"}
]

Alongside the detector I built a degradation harness that applies any chain of transformations at fixed parameters, so the same evaluation runs across 13 conditions including realistic multi-step chains like "downscale to 50%, then JPEG-70," which is roughly what a social platform does to an upload.

How we built it

Data. ArtiFact (Awsaf et al., ICIP 2023): 2.5M images across 25 generators and 8 real-image sources. I chose it over the suggested datasets deliberately. CIFAKE contains one generator (Stable Diffusion 1.4), so a detector trained on it cannot be tested for cross-architecture generalization. WildFake has the diversity but ships as a 1.29TB archive with a single 47GB GAN blob, which is not workable at hackathon scale.

From it I sampled 120,000 images: 60k real, 30k GAN, 30k diffusion. The sampling is the part I would defend hardest. stylegan2 alone holds 1,000,000 images, 80% of all GAN data in ArtiFact, so uniform random sampling would have produced a StyleGAN2 detector wearing a general-purpose label. Every architecture is quota-capped, with automatic redistribution when a source runs short, so 20 generators are represented roughly evenly.

Model. ConvNeXt-Tiny, ImageNet-pretrained, fine-tuned end to end. 28M parameters, well under the 2B limit.

I fine-tuned the whole network rather than probing a frozen backbone, and the reason matters: ImageNet features encode object semantics, and a real photo of a dog and a StyleGAN dog both encode as "dog." The distinguishing signal is texture and upsampling statistics, which ImageNet training explicitly learns to discard as nuisance variation. A linear probe cannot recover information the early layers have already thrown away.

Augmentation. Instead of the usual flip-and-crop, training applies the deployment degradations themselves: JPEG 30-95, Gaussian blur, downscaling, noise, colour jitter. Sampled from continuous ranges wider than the evaluation points, so the model learns the degradation rather than four memorized settings.

A baseline I built to argue against myself. Sobel gradient statistics (14 features) and 2D FFT radial energy (9 features) fed to random forests. Built specifically to test whether explicit, interpretable frequency features are competitive with a learned model.

Challenges we ran into

Two days lost to the wrong dataset. I started on WildFake and discovered only after several hours of streaming that its ModelScope entry is a metadata index, not the images. Image_path points into a 1.29TB file tree, the manifest is 33 stitched CSVs with duplicated rows, and the GAN images are one undivided 47GB zip. Switching to ArtiFact cost an afternoon and saved the project.

Diagnosing underfitting versus overfitting. The first run plateaued at 76.7% training accuracy with the learning rate annealed to zero while still improving. I assumed the learning rate was too low and went 5x higher. It made almost no difference, which is what actually identified the bottleneck: not optimization, but data volume. Tripling the dataset from 40k to 120k gained 3.6 points.

Two preprocessing paths that fail silently. The CNN needs ImageNet-normalized input; the gradient and FFT feature extractors need raw [0,1] pixels. Feeding normalized tensors to the feature extractors produces garbage scores with no error message at all. This cost me a debugging session and is now the loudest comment in the codebase.

Accomplishments that we're proud of

A robustness finding with a mechanism behind it, not just a number.

Branch	Clean AUC	Worst-case AUC	Drop
ConvNeXt (learned)	0.880	0.790	-0.090
FFT (9 features)	0.735	0.491	-0.244

The frequency baseline falls below chance. And there is a reason, not a coincidence: those features measure energy in high-frequency radial bins, and JPEG compression works by transforming blocks into frequency space and quantizing away high-frequency coefficients. JPEG is mechanically a device for deleting the quantity those features measure. Any detector built on explicit frequency statistics inherits that vulnerability.

The baseline was trained with the full degradation pipeline. It saw compression throughout training and collapses anyway, because augmentation can only teach a model to cope with degradation if the feature it depends on survives.

Testing something and rejecting it. I ensembled the hand-crafted branch with the CNN across all 13 conditions. It added 0.004 AUC on clean images and made things slightly worse under heavy degradation. So it is not in the shipped system. The code stays in the repo behind a --rf-weight flag so the experiment reproduces.

Finding that the intuition about generator families is wrong. Error rates within the GAN family span more than a hundredfold: stylegan3 evades detection 47.8% of the time while star_gan fails 0.4% of the time. Difficulty tracks how good the generator is, not whether it is a GAN or a diffusion model.

What we learned

JPEG is not the main threat. Counter to the emphasis in most of this literature, blur (-0.048 AUC) and 4x downscaling (-0.043) hurt the learned detector more than JPEG-30 (-0.025). A pipeline that resizes before storing is riskier than one that compresses hard.

The failures are confident, not borderline. This was the uncomfortable finding. A StyleGAN2 image with a visibly corrupted rider, obvious to any human in a second, scored 0.006. An ordinary sunset photograph scored 1.000 for AI-generated. The errors sit at the extremes of the score distribution, not near the decision boundary, so no threshold adjustment fixes them. The model reads texture statistics; it does not reason about whether a horse has the right number of legs.

Art gets flagged. Real paintings from metfaces are misclassified 31.9% of the time. The model has partly learned "not photographic, therefore generated," which for a moderation deployment is a specific and serious failure mode.

Confidence is not calibration. The scores rank images usefully, which is what AUC measures, but a 1.000 does not mean the model is right 99.9% of the time. That distinction should be stated wherever a score is shown to a user.

What's next for Robust AIGC Detection Under Real-World Transformations

Leave-one-generator-out evaluation. Every generator currently appears in both train and test, so the reported numbers measure detecting architectures the model has seen. The harder and more honest question is an unseen generator. The source column is preserved in the manifest specifically to make this possible, and it is the first experiment I would run.

Multi-class training. 21 outputs (real plus each generator), summing the fake probabilities at inference. This gives each architecture its own output to specialize on rather than forcing 20 unrelated artifact types into one class. The ArtiFact authors report this beating binary training, and it directly targets my worst failure mode.

Calibration. Temperature scaling on a held-out set, so the scores can be read as probabilities rather than only as a ranking.

Newer generators. ArtiFact predates SDXL, Midjourney v6 and Flux. Performance on 2026-era models is unmeasured and probably worse.

A semantic second opinion. The model structurally cannot catch the broken StyleGAN horse, because anatomy is not a texture statistic. A vision-language model consulted on high-uncertainty cases would cover a failure class this architecture cannot reach.

Built With

• albumentations
• convnext
• huggingface
• kaggle
• matplotlib
• numpy
• pandas
• python
• pytorch
• scikit-learn
• timm