Cereal Crunch Binary AI-Generated Image Classifier

Inspiration

Have you ever been fooled into thinking an AI-generated image was real? Fret not, for you are not alone! In recent years the internet has been bombarded with AI-generated content (AIGC) of all kinds, in particular countless synthetic images created with malicious intent to fuel misinformation, impersonation, and fraud at scale. Improvements in advanced AI models have also rendered them capable of generating increasingly convincing synthetic images, and as such detection of these images has become exponentially more difficult, especially when images have undergone transformations like cropping, blurring, compression, colour adjustment, and rescaling.

How I built it

Cereal Crunch is a binary AI-generated image classifier (<2B parameters) that provides detection on both clean images as well as on images under six post-processing image transforms (JPEG compression q=90/70/50/30, Gaussian blur σ=0.5/1.0/2.0, resize 0.5×/0.25, Gaussian noise σ=0.02/0.05/0.10, colour jitter ±20%, center crop 80%). The model uses a dual-branch architecture that combines a frozen CLIP ViT-B/32 vision encoder (high-level semantic features) with a small trainable noise-residual branch (Sobel/Laplacian high-frequency filters), feeding a lightweight trainable head. The choice of using a frozen CLIP over fine-tuning a backbone was informed by literature (Ojha et al., 2023) showing that frozen features generalise better across different generators than fine-tuned ones, which is particularly important for the generalisability of the model’s detection capabilities. The classifier can be pointed at a directory of images to be returned a JSON file of {image_path, pred} confidence scores.

Development Tools used

Development tools that I used include VS Code, Git and Python. The model was developed and trained locally on Apple Silicon (M4 chip).

Models or APIs used

I loaded the CLIP ViT-B/32 model (openai/clip-vit-base-patch32) via the Hugging Face transformers and used it frozen instead of fine-tuned as the semantic feature-extraction branch.

Libraries and Frameworks used

Libraries and frameworks used include PyTorch, NumPy, Pillow, OpenCV, pytest, scikit-learn, matplotlib, Hugging Face transformers, huggingface_hub, datasets.

Datasets used

I used data from three datasets.

• CIFAKE (Kaggle, birdy654/cifake-real-and-ai-generated-synthetic-images), used early on to validate the full pipeline end-to-end on 120,000 small 32 x 32 images, of which 60K were real and 60K were fake.
• SID_Set (Hugging Face, saberzl/SID_Set), the main training and evaluation set initially, consisting of real photos from OpenImages V7 and FLUX-generated synthetic images, both of about 1024 x 1024 resolution. I used a split of 50K/50K training, 10K/10K validation, 10K/10K testing.
• ArtiFact (Hugging Face mirror of awsaf49/artifact, originally published ICIP 2023), a monolithic set of almost 2.5M images that consists of roughly 1M real images taken from 8 sources (LSUN, COCO, ImageNet, FFHQ, CelebA-HQ, CycleGAN, Landscape, MetFaces) and roughly 1.5M fake images taken from 25 generators (13 GANs, 7 diffusion models, 5 miscellaneous methods). I pulled stratified samples from this dataset to ensure equal representation across the 8 sources of real images and 25 generators of synthetic images. I used a split of 5K/5K training, 1K/1K validation, 1K/1K testing.

Accomplishments

• I developed a robust image transformation pipeline to allow the model to be trained, validated, and tested on images that have undergone randomly-selected transformations. This was randomly applied at training time to expose the model to realistic compound degradation, then once again at evaluation time across all 14 conditions to robustly assess each transform’s individual effect on accuracy performance.
• The final classifier trained on a stratified subset of the ArtiFact dataset achieved a blended AUC score of ~0.94 on a held-out test set.

Challenges

• My first round of training on the SID_Set returned near perfect classification accuracies (0.9990) even across all 6 image transforms. Instead of taking these figures at face value, further inspection of the data revealed that every fake image was exactly 1024x1024 with one identical JPEG compression signature, while real images varied in their size, aspect ratio and quality. The model was thus likely learning these signals that were consistent across the dataset. To address this problem, the size, resolution and quality of the images were subtly randomised moving forward.
• Even after controlling for confounders in the images, the model still returned extremely high accuracies (0.9890) even on the held-out test set. I hypothesised that this was a trade-off between performance and generalisability; the SID_Set dataset consisted of 60,000 synthetic images generated only from FLUX, and the model had become extremely good at detecting FLUX-generated synthetic images. To test this hypothesis, I sourced a third dataset (ArtiFact) which included synthetic images across 25 different generators and evaluated the model on a subset of that dataset. As I expected, the SID_Set-trained model failed almost completely, returning AUC scores of roughly 0.48 for some families of generators, which was below chance.
• After the cross-generator realisation, I imported a stratified subset of 12,000 images from the ArtiFact dataset and retrained the model from scratch. This addressed generator diversity within ArtiFact’s own diversity, as evidenced by the improved AUC scores across all generators. However, I tested the ArtiFact-trained model back against SID_Set’s FLUX-generated images, and it once again returned AUC scores of roughly 0.48. I acknowledge this as an inevitable tradeoff between classification performance and generalisability across image generators.

Built With

• python
• pytorch