Perry - Embedding Consistency for Robust Fake Detection

Robust Detection of AI-Generated Images Under Real-World Transformations

Objective

This project explores a practical question: can we tell whether an image is authentic or AI-generated after it has been reposted or lightly edited? A detector may work well on an original file and still struggle once compression, blur, resizing, or cropping removes some of the clues it relies on. Our aim is to make that drop as small as possible.

Transformation setup

For the robustness check, we use six transformation families:

transformation	settings used
JPEG compression	quality 90, 70, 50, and 30
Gaussian blur	σ 0.5, 1.0, and 2.0
Resize down and back up	scale 0.5× and 0.25×
Gaussian noise	σ 0.02, 0.05, and 0.10
Colour adjustment	brightness, contrast, and saturation raised together by 20%
Centre crop	keep the central 80%

We test these both one at a time and in seeded stacks of distinct families. A depth-six example has passed through all six families once, which is closer to the kind of accumulated damage an image might pick up through several reposts. The colour-adjustment cell is our current deterministic implementation; it does not cover every positive and negative 20% combination from the brief.

Strong result

The shipped canon6_AlowLR model held up encouragingly in these tests:

Clean AUROC: 0.9999
Mean AUROC over 14 implemented single-transform conditions: 0.9995
Worst single-transform AUROC: 0.9982
Six distinct transform families stacked: 0.9968 AUROC

At one cutoff calibrated on the pooled clean-plus-stacked evaluation distribution, it caught 98.8% of the AI images after all six transform families were composed, while flagging 2.6% of the real photographs in that depth-six condition. This is a small reference-set experiment rather than a claim about every image found in the wild, but it is a useful sign that the model is not relying only on pristine-image artifacts.

Limitations and future work

The model is weak on tampered images — authentic photographs where one region has been partially generated or replaced. On a 2,364-image test set of such edits it caught only 27.1%, against 90–100% for fully generated images. The cause is mechanical rather than mysterious: scoring averages 27 crops, and an edit confined to one region is outvoted by the authentic remainder of the picture.

We plan two changes. First, we will add tampered images to the training corpus: in our measurement this raises recall on them from 27.1% to 72.1% at a cost of 0.3 points on the main benchmark. Second, we will remove the crop mechanic and train and infer on the full image, so no averaging step exists to wash out a local signal. Another potential direction is to find a better way to merge the signal across crops. An early retraining run in this direction was ahead of the shipped model on validation but unfinished at the deadline, so we make no claim for it here.