InvariantGuard

AI-image detectors often fail after images are compressed, resized, or reposted. InvariantGuard keeps working anyway.

Comment

Robust Detection of AI-Generated Images Under Real-World Transformations

1. Written Project Description — Devpost

Inspiration

AI-generated images are becoming increasingly realistic, but detecting them in clean laboratory conditions is only part of the problem.

Images distributed through real platforms are often compressed, resized, blurred, cropped, filtered, or otherwise transformed before they reach a detector. These transformations preserve whether an image is authentic or AI-generated, but they may alter the visual evidence used by a classifier.

Our project therefore focuses on a stricter question:

Can an AI-generated image detector preserve its decision after realistic image transformations?

What Our Project Does

We built an image-level detector that outputs the probability that an input image is AI-generated.

Our system uses a pretrained DINOv2 ViT-B/14 visual backbone with a lightweight binary classification head.

We compare two training approaches.

Clean Baseline

Our baseline is trained only to classify clean images as:

  • 0 — authentic / real
  • 1 — AI-generated

The frozen DINOv2 backbone extracts visual features, which are passed through a lightweight MLP classifier.

The clean baseline reached a best validation ROC-AUC of 0.9945, showing that the underlying visual representation is already highly effective for clean-image discrimination.

Pairwise Robust Detector

Our proposed method extends the same detector with robustness-oriented training.

For every training image, we construct two views:

  1. the original clean image; and
  2. a transformed version of the same image.

Because transformations such as JPEG compression, blur, resizing, noise, color adjustment, and cropping do not change the underlying provenance of the image, both views share the same real/AI label.

The two views pass through the same DINOv2-based detector.

We optimize three objectives:

Classification loss Both the clean and transformed images should be classified correctly.

Prediction consistency loss The AI-generation probabilities for the clean and transformed versions should remain similar.

Representation consistency loss The internal feature representations of the clean and transformed versions should remain close.

Our complete training objective is:

L = L_classification + 0.5 L_prediction + 0.25 L_representation

This encourages the model to learn authenticity evidence that remains useful even when the pixels have been modified by realistic post-processing.

The robust detector achieved a best clean validation ROC-AUC of 0.9949, indicating that robustness-oriented training preserved the detector's strong clean-image discrimination.

Robustness Transformations

Our fixed evaluation benchmark includes transformations corresponding to realistic image redistribution:

  • JPEG compression at qualities 90, 70, 50, and 30
  • Gaussian blur with σ = 0.5, 1.0, and 2.0
  • downsampling to 0.5× and 0.25× followed by upscaling
  • Gaussian noise with σ = 0.02, 0.05, and 0.10
  • brightness, contrast, and saturation adjustment
  • 80% center crop
  • compound transformation pipelines

The benchmark transformations are deterministic and identical for both models, allowing a controlled comparison.

Dataset

Our core experiment uses SID_Set.

SID_Set contains authentic, fully synthetic, and tampered images. For this project we formulate the task specifically as authentic-versus-fully-synthetic detection:

  • authentic images → 0
  • fully synthetic images → 1
  • tampered images → excluded

We use deterministic balanced manifests containing:

  • 2,500 training images
  • 500 validation images
  • 500 held-out internal test images

The exact same data splits are used for both the clean baseline and robust detector.

The hackathon-provided demonstration benchmark is kept separate from training.

Evaluation

Because clean accuracy alone does not measure robustness, we report:

Clean ROC-AUC Discrimination on unmodified images.

Robust Pooled ROC-AUC ROC-AUC after pooling predictions across transformed conditions.

Mean Condition ROC-AUC Average ROC-AUC across individual transformation conditions.

Worst-Case ROC-AUC Performance under the detector's weakest tested condition.

Robustness Drop The difference between clean ROC-AUC and pooled robust ROC-AUC.

Final Results

Model Clean AUC Robust Pooled AUC Mean Condition AUC Worst-Case AUC Robustness Drop
Clean Baseline [BASELINE_CLEAN] [BASELINE_ROBUST] [BASELINE_MEAN] [BASELINE_WORST] [BASELINE_DROP]
Pairwise Robust Detector [ROBUST_CLEAN] [ROBUST_POOLED] [ROBUST_MEAN] [ROBUST_WORST] [ROBUST_DROP]

Replace these placeholders only with the final held-out test results.

Result Interpretation

[USE AFTER RESULTS]

Example if the robust model improves:

"While both models maintain similarly strong clean-image discrimination, pairwise robustness training improves performance after realistic post-processing. The largest gains are observed under [TRANSFORMATIONS], while the worst-condition ROC-AUC improves from [X] to [Y]. This suggests that jointly aligning clean and transformed predictions and representations can reduce sensitivity to provenance-preserving transformations."

Do not attribute the improvement individually to prediction consistency or representation consistency because the experiment evaluates the complete robustness strategy as one method.

Tools and Technologies

Development

  • Python
  • Visual Studio Code
  • Git and GitHub
  • school GPU compute environment
  • local development and testing environments

Models

  • DINOv2 ViT-B/14 visual backbone
  • lightweight binary MLP classification head

Libraries

  • PyTorch
  • torchvision
  • timm
  • Hugging Face Datasets
  • scikit-learn
  • pandas
  • NumPy
  • Pillow
  • PyYAML
  • matplotlib
  • pytest

No commercial inference API is required by the final detector.

Engineering Design

The repository separates the system into several components:

Data pipeline Builds and validates deterministic train, validation, and test manifests.

Transformation engine Implements deterministic benchmark corruptions and stochastic training transformations.

Model Implements the DINOv2-based binary detector.

Consistency losses Implements prediction-level and representation-level alignment between clean and transformed images.

Training system Supports clean baseline training and pairwise robust training.

Evaluation system Runs clean and transformed evaluation, logs raw predictions, and computes robustness metrics.

Inference interface Takes an arbitrary directory of images and produces a JSON file containing the AI-generation probability for every image.

Final Inference Interface

The final system can be executed using:

python inference.py --input <IMAGE_DIRECTORY> --model-id M3 --checkpoint <CHECKPOINT> --output predictions.json

The output follows the required structure:

[
  {
    "image_path": "images/example_1.jpg",
    "pred": 0.9472
  },
  {
    "image_path": "images/example_2.png",
    "pred": 0.0835
  }
]

pred represents the estimated probability that the image is AI-generated.

Impact

Transformation-robust AI-image detection is relevant to environments where images undergo repeated processing before analysis.

Potential applications include:

  • trust and safety systems
  • misinformation investigation
  • content provenance analysis
  • moderation triage
  • forensic screening tools

Our detector is intended to provide a machine-learning signal rather than definitive proof of image provenance.

Limitations

Our experiment uses a limited hackathon-scale training subset and therefore cannot represent every visual domain, camera pipeline, or generative model.

The current experiment also evaluates the complete pairwise robustness strategy against a clean-only baseline. Due to compute and time constraints, we do not independently isolate how much improvement comes from transformation augmentation, prediction consistency, or representation consistency.

Severe transformations can also permanently destroy forensic information. In such cases, invariance cannot fully recover evidence that is no longer present in the image.

Finally, strong performance on the current benchmark does not guarantee equivalent generalisation to future image generators or substantially different datasets.

What We Would Improve With More Time

Given additional time and compute, we would:

  • train on a larger and more diverse mixture of AI-generation datasets
  • evaluate additional generator families and cross-dataset generalisation
  • perform controlled ablations of augmentation, prediction consistency, and representation consistency
  • investigate adaptive consistency weighting
  • test partial backbone fine-tuning
  • compare additional visual foundation models
  • evaluate more realistic multi-stage social-media processing pipelines
  • calibrate model probabilities for deployment-oriented threshold selection

Team Contributions

[NAME] — Data Pipeline and Baseline Built the controlled dataset pipeline, manifest generation and validation, preprocessing, and clean baseline training.

[NAME] — Transformation System Implemented benchmark transformations and stochastic corruption sampling used for robustness training and evaluation.

[NAME] — Robust Learning Implemented the DINOv2 robust detector, paired clean/transformed training, prediction consistency, representation consistency, and robust-training infrastructure.

[NAME] — Evaluation and Integration Implemented clean and robustness evaluation, metric aggregation, raw prediction logging, inference integration, and submission analysis.

2. Public GitHub Repository — Final Checklist

The public repository should contain:

  • complete source code
  • requirements.txt
  • Python 3.9 compatibility support
  • dataset manifest-generation instructions
  • clean baseline training command
  • pairwise robust training command
  • clean evaluation command
  • robustness evaluation command
  • final inference.py
  • result aggregation scripts
  • robustness plotting scripts
  • error-analysis scripts
  • test suite

The README should clearly show the end-to-end reproduction sequence:

Install

python -m pip install -r requirements.txt

Generate Data

python -m data.build_sid_manifests --train-per-class 1250 --validation-per-class 250 --test-per-class 250 --sampling-pool-per-class 1500 --seed 42

Validate Data

python -m data.validate_manifests --skip-image-check

Train Clean Baseline

python -m training.train_corrected_baseline --config configs/corrected_baseline.yaml

Train Proposed Robust Model

python -m training.train_pairwise --config configs/M3_pairwise.yaml

Evaluate

Run both models through the clean and robustness evaluators.

Inference

python inference.py --input <IMAGE_DIRECTORY> --model-id M3 --checkpoint checkpoints/M3_pairwise.pth --output predictions.json

The README should also contain the final M1-versus-M3 results, limitations, team contributions, and a link to the final checkpoint.

3. Robustness Evaluation Summary

Headline Table

Model Clean AUC ↑ Robust Pooled AUC ↑ Mean Condition AUC ↑ Worst AUC ↑ Robustness Drop ↓
Clean Baseline [ ] [ ] [ ] [ ] [ ]
Pairwise Robust Detector [ ] [ ] [ ] [ ] [ ]

Transformation-Level Comparison

Condition Baseline AUC Robust Detector AUC Difference
Clean [ ] [ ] [ ]
JPEG 90 [ ] [ ] [ ]
JPEG 70 [ ] [ ] [ ]
JPEG 50 [ ] [ ] [ ]
JPEG 30 [ ] [ ] [ ]
Blur 0.5 [ ] [ ] [ ]
Blur 1.0 [ ] [ ] [ ]
Blur 2.0 [ ] [ ] [ ]
Resize 0.5× [ ] [ ] [ ]
Resize 0.25× [ ] [ ] [ ]
Noise 0.02 [ ] [ ] [ ]
Noise 0.05 [ ] [ ] [ ]
Noise 0.10 [ ] [ ] [ ]
Color Jitter [ ] [ ] [ ]
Center Crop 80% [ ] [ ] [ ]
Compound Mild [ ] [ ] [ ]
Compound Medium [ ] [ ] [ ]
Compound Severe [ ] [ ] [ ]

Use the generated clean_vs_robust.png as the compact main visual and condition_auc_comparison.png as the detailed supporting visual.

4. Error Analysis Note

Aggregate performance does not capture every failure mode, so we additionally inspect representative false positives, false negatives, and transformation-induced prediction shifts.

False Positive

Ground truth: Authentic Predicted AI probability: [SCORE] Condition: [CONDITION]

The detector incorrectly assigned a high AI-generation probability to an authentic image.

Visual inspection suggests [OBSERVATION BASED ON ACTUAL IMAGE].

This illustrates that naturally occurring visual statistics, aggressive photography or editing, or compression artifacts may sometimes resemble features associated with generated images.

False Negative

Ground truth: AI-generated Predicted AI probability: [SCORE] Condition: [CONDITION]

The detector incorrectly assigned a low AI-generation probability to a synthetic image.

Possible factors visible in this example include [OBSERVATION].

This demonstrates that highly photorealistic generation or transformation-induced information loss can reduce the strength of detectable synthetic-image evidence.

Transformation-Induced Failure

Ground truth: [REAL / AI] Clean AI probability: [SCORE] Transformed AI probability: [SCORE] Transformation: [TRANSFORMATION]

The image retains the same provenance, but the transformation causes a large shift in the detector's prediction.

This represents the central failure mode targeted by our pairwise robustness strategy.

Trade-Offs

Clean discrimination vs robustness Robustness regularisation may change clean-image performance slightly. We therefore report both rather than optimizing one metric in isolation.

Robustness vs information destruction Very severe transformations can remove genuinely useful forensic evidence.

Sensitivity vs false positives A threshold that catches more AI-generated images may also incorrectly flag more authentic images.

Benchmark robustness vs generalisation Strong robustness to the tested transformations does not guarantee robustness to every generator, visual domain, or future processing pipeline.

5. Final Submission Narrative

Our main finding should be summarized as:

"Our clean DINOv2 baseline already provides extremely strong clean-image discrimination. We therefore focus on a harder problem: preserving that decision after realistic image transformations. Our proposed detector jointly trains on clean and transformed views while explicitly aligning both its output predictions and internal representations. We evaluate whether this complete pairwise robustness strategy improves transformed-image performance without sacrificing clean discrimination."

Once final results are available, append:

"Compared with the clean baseline, our proposed detector changes pooled robust ROC-AUC from [X] to [Y] and worst-condition ROC-AUC from [A] to [B], while clean ROC-AUC remains [C]."

Built With

Share this project:

Updates

Submission history