model pipeline

(un4gettables) [track#5]

Amber Yeo En, Ashleigh Lim Tze Yen, Chew Rui Qi Chloe, Tay Jo-lynn

Project Description

Overview

AI-generated and manipulated images are becoming increasingly difficult to distinguish from authentic images. This challenge becomes even more significant when images undergo common real-world transformations such as compression, blur, resizing, or noise that alter the visual patterns which AI-generated content(AIGC) detectors commonly rely on.

Our project develops a robust image-level AIGC detection pipeline designed to identify whether an image is real or AI-generated/manipulated, while evaluating how well the detector performs under common image transformations.

Our approach combines additional image preprocessing, patch-based analysis, data augmentation, and transfer learning. Particularly, we divide images into overlapping patches before fine-tuning a pretrained DINOv2-base image classification model. This allows the model to analyse localised regions of an image instead of relying solely on the appearance of the image as a whole. During training, we additionally apply randomly selected image transformations such as JPEG compression, blur, resizing, Gaussian noise, and colour adjustments to expose the model to variations that may occur in real-world images.

Our Approach

Dataset and Label Processing

We use the SID_Set dataset, which contains images, labels, and manipulation masks where available.

The original labels are converted into a binary classification task:

0 — REAL
1 — AI/MANIPULATED

Images are converted to RGB format before being passed through the model.

Patch-based Image Processing

A key component of our approach is dividing each image into smaller overlapping regions, or patches, rather than processing the entire image as a single input.

We extract 224 × 224 pixel patches using a stride of 180 pixels, resulting in overlapping patches across the image.

This approach is motivated by the fact that an AI-generated or manipulated region may occupy only a small portion of an otherwise authentic image. Analysing localized regions gives the model an opportunity to identify suspicious visual patterns that could be diluted when considering the entire image at once.

For images with manipulation masks, we use the masks to determine which patches contain manipulated content. A patch is labelled as AI/manipulated when at least 5% of its pixels overlap with the manipulated region. Real images are labelled as real at the patch level, while AI/manipulated images without an available mask fall back to their original image-level label.

Data Augmentation

To expose the model to variations that may occur in real-world images, we apply random image transformations during training.

With a probability of 0.5, one transformation is randomly selected and applied to a training image before it is passed through the model.

Our transformations include:

JPEG compression at different quality levels
Gaussian blur at different strengths
Downscaling followed by upscaling
Gaussian noise at different noise levels
Brightness, contrast, and saturation adjustments

Note that the validation data is kept unaugmented.

These augmentations simulate common changes that images may experience when they are stored, uploaded, shared, or processed.

Importantly, the Gaussian noise component refers to adding noise to the image as data augmentation, rather than extracting existing noise from the image.

Model Design

The final model is DINOv2 Base(facebook/dinov2-base), a pretrained model obtained from HuggingFace which has been fine-tuned to our domain for binary classification.

Model Training

After preprocessing and patch extraction, the training patches are passed to Dinov2.

Training configurations:

Optimizer: AdamW
Learning rate: 2e-5
Batch size: 8
Epochs: 3

Weighted random sampling is also applied to the training patches to reduce class imbalance at the patch level.

During training, the model makes a prediction for each patch. Its prediction is compared with the known patch label, producing a loss value. We then use backpropagation and the AdamW optimizer to update the model's parameters.

The model is trained for multiple epochs over the training data. One epoch represents one complete pass through the training dataset.

The goal of training is for DINOv2 to learn visual patterns that help distinguish real image regions from AI-generated or manipulated regions.

Image-Level AIGC Detection

Our model operates on image patches, but the final output is an image-level prediction.

During inference, an image is divided into overlapping patches and DINOv2 produces a score for each patch. We take the maximum patch score as the overall image score.

The image is classified as:

AI/MANIPULATED if the score ≥ threshold
REAL if the score < threshold

We tested thresholds of 0.60, 0.50, and 0.42 to examine the precision-recall trade-off.

The final threshold we established was 0.60.

Robustness Evaluation

We evaluate how well the detector performs when images undergo common real-world transformations.

We compare performance on clean images against images affected by:

JPEG compression
Gaussian blur
Resizing
Gaussian noise
Colour adjustments

We report accuracy, precision, recall, and F1-score for each condition.

Development Tools

The project was primarily developed in Google Colab using:

Python
Hugging Face Datasets
Hugging Face Transformers
PyTorch
scikit-learn
Pandas
NumPy
Pillow(PIL)

Datasets and Assets

We use the SID_Set dataset from Hugging Face, containing real and AI-generated/manipulated images and, where available, manipulation masks. The masks are used to help assign labels to image patches.

We use the pretrained DINOv2 model (facebook/dinov2-base) from Hugging Face Transformers and fine-tune it for binary REAL vs AI/MANIPULATED classification.