Vettage

Inspiration

AI-generated images are becoming increasingly realistic, making it harder to distinguish between what is real and what is synthetic. Modern generators like Stable Diffusion and Midjourney v6 produce images that are remarkably close to photographs. But the problem only gets harder when these images enter the real world: social media platforms re-encode uploads as JPEG, messaging apps resize them, users crop, filter, compress and repost them. A detector that only works on pristine lab images is useless in practice, particularly when such images are encountered online.

We wanted to build a detector around this reality. Vettage is designed to answer a simple question: can we still detect an AI-generated image after it has been altered and redistributed?

What It Does

Vettage takes a folder of images and returns a confidence score for each one. Scores closer to 0 indicate stronger evidence of an authentic photograph, while scores closer to 1 indicate stronger evidence of synthetic generation. Our current threshold of 0.5 separates likely authentic and likely AI-generated images.

Under the hood, Vettage uses a fine-tuned CLIP vision model that has learned to spot visual representations that distinguish real and synthetic imagery. Rather than relying exclusively on pristine pixel-level details, we trained the model under realistic image transformations so that it can learn to spot the subtle statistical fingerprints that generators leave behind in texture, lighting, and frequency patterns which artifacts that persist even after heavy JPEG compression, blurring, resizing, noise, colour adjustment, and cropping.

How Our Solution Addresses the Problem Statement

Vettage directly addresses the competition’s core focus on robust AI-generated image detection under realistic post-processing and redistribution by maintaining detection accuracy.

During training, images were stochastically subjected to the transformations specified in the problem statement: JPEG compression, Gaussian blur, downscale-then-upscale resizing, Gaussian noise, colour jitter, and centre cropping. This encourages the model to rely on features that remain informative even when the original image has been degraded.

We also built a dedicated robustness evaluation pipeline that compares performance on clean images against images subjected to each transformation. This allows us to measure not only whether the model can detect synthetic images, but how its performance degrades as the image is altered.

In our experiments, Vettage maintains an AUC above 0.97 across the the harshest single-transform scenarios we tested.

The model produces a score between 0 and 1, with scores above 0.5 classified as AI-generated, which is converted into the competition’s required output format of {image_path, pred} for each image.

How We Built It

Architecture. We started with OpenCLIP ViT-L/14 (303M parameters), a vision transformer pretrained on over 12 billion image-text pairs. CLIP has seen so many real photographs during pretraining that it carries an implicit understanding of what "real" looks like. Rather than training a detector from scratch, we leveraged this foundation and focused on teaching the model the difference between real and generated.

Fine-tuning with LoRA. We used Low-Rank Adaptation (LoRA) to inject small trainable matrices into the vision encoder's MLP layers. This lets us fine-tune only 7.9M parameters (~2.5% of the backbone) while keeping the rest of CLIP frozen. The result is we avoid catastrophic overfitting on our training data while staying well under the 2B parameter cap at 311M total.

Robustness by design. During training, each image has a 70% chance of being randomly degraded with one of the competition's specified transforms with JPEG compression (quality 30–90), Gaussian blur (σ 0.5–2.0), downscale-then-upscale (0.25×–0.5×), Gaussian noise (σ 0.02–0.10), colour jitter (±20% brightness/contrast/saturation), and center cropping to 80%. This forces the model to rely on features that survive real-world redistribution rather than fragile pixel-level artifacts.

Dual-head design. We discovered that tiny images (≤64px, like CIFAR-scale thumbnails) behave completely differently from native-resolution photos when upsampled to CLIP's 224×224 input. To handle this, we trained a separate lightweight classification head specifically for low-resolution inputs, routing images to the appropriate head based on their original size.

Diverse training data. To maximise cross-generator generalisation, we trained on four complementary datasets covering 8+ distinct AI generators from research benchmarks to real in-the-wild AI images scraped from social media (which have already been through platform compression).

Challenges We Ran Into

Our biggest challenge was the gap between in-distribution and real-world performance. Our first model achieved 99% accuracy on its validation set, but its performance dropped significantly when we tested it on out-of-distribution AI images. The validation results were misleading because the model had only seen two relatively narrow data sources during training. This taught us that a high validation score does not necessarily mean that a detector has learned general characteristics of AI-generated imagery. It may simply have learned the fingerprints of the datasets or generators it has already seen.

We also struggled with false positives on stylised content. Because much of our initial real-image data consisted of conventional photographs, the model's concept of "real" was anchored to COCO-style photographs. This meant that the model could sometimes interpret unusual textures, such as those found in painterly or highly edited visual styles, as evidence of synthetic generation. One particularly memorable failure was a genuine photograph of a cat that Vettage assigned a synthetic score of 0.987, effectively showing almost maximum confidence that this real image was AI-generated.

These failures pushed us to expand our training data across multiple generators, introduce stronger robustness augmentation, tune learning rate and regularisation to prevent overfitting, and pay much more attention to false positives rather than optimising only for aggregate accuracy.

What's Next for Vettage

Given more time, we would focus on improving both generalisation and interpretability.

First, we would expand the "real" class beyond conventional photographs to include paintings, illustrations, digital art, heavily edited photographs, and other visually unusual but authentic content. This would give the model a broader understanding of what real-world imagery can look like and help reduce false positives.

Second, we would add more generator sources and perform more systematic unseen-generator evaluation to determine whether Vettage is learning general characteristics of synthetic imagery rather than memorising generator-specific fingerprints.

Third, we would investigate hard-negative mining: identifying genuine images that Vettage confidently misclassifies and feeding these difficult examples back into training.

We would also explore Grad-CAM or similar visualisation techniques to show which regions of an image influenced the model's prediction. This could make the system more interpretable and help us diagnose failures such as our false-positive cat.

Finally, we would build a simple web interface where non-technical users could drag and drop an image and receive a synthetic score together with an explanation of the result.

Development Tools

Language/Environment: Python 3.11
IDE: VS Code
Version Control: Git + Git LFS (for model weight versioning)
Hardware Acceleration: macOS with Apple Silicon (MPS) for local training and inference; CUDA supported via --device cuda

Models and APIs

Backbone: OpenCLIP ViT-L/14 pretrained on CommonPool XL (12.8B samples) — laion/CLIP-ViT-L-14-CommonPool.XL-s13B-b90K
Detector Head: GRIP-UNINA clipdet_latent10k_plus linear head (Cozzolino et al., "Raising the Bar of AI-generated Image Detection with CLIP", CVPRW 2024)
Fine-Tuned Checkpoint: checkpoints/lora/lora_best.pt — LoRA adapters (rank 32, alpha 64) on vision MLP layers + classification head + tiny-image head

Libraries and Frameworks

Core: PyTorch, torchvision, OpenCLIP (open_clip_torch), timm
Fine-Tuning: PEFT (peft) for LoRA
Data Handling: Hugging Face datasets, pandas, Pillow
Evaluation: scikit-learn, matplotlib, tqdm
Utilities: PyYAML, huggingface-hub

Datasets and Assets

All training data is publicly available and properly licensed. The competition's WildFake / COCO val2017 demonstration subset was never used during training.

SID_Set (saberzl/SID_Set on HF): 2,000 Real | 2,000 Fake | Multiple research generators
CIFAKE (dragonintelligence/CIFAKE-image-dataset on HF): 4,000 Real | 4,000 Fake | Stable Diffusion v1.4
Defactify (Rajarshi-Roy-research/Defactify_Image_Dataset on HF): 3,000 Real | 3,000 Fake | SD 2.1, SDXL, SD3, DALL·E 3, Midjourney v6
SynthWildX (bundled in repo): — Real | ~1,500 Fake | DALL·E 3, Midjourney, Firefly
Pretrained Weights: CLIP ViT-L/14 from LAION/OpenCLIP; detector linear head from grip-unina/ClipBased-SyntheticImageDetection