ForgeGate

Written Project Description

Problem Statement

The rapid advancement of AI image generation has created an urgent need for reliable detection systems. However, existing forensic detectors fail in real-world scenarios because they rely on subtle manipulation artifacts that are easily destroyed by common image transformations. When a user saves an AI-generated image as JPEG, posts it on social media (which applies compression), or takes a screenshot, the forensic traces these detectors depend on vanish.

The core challenge: AI-generated images in the wild undergo the same transformations as real images—JPEG compression, resizing, blur, noise addition, color adjustments, and cropping. A detection system must maintain accuracy despite these real-world degradations.

How Our Solution Addresses the Problem

ForgeGate addresses this fundamental limitation through a novel architecture that adaptively balances semantic and forensic features based on image degradation level:

1. Semantic Stream (Frozen CLIP ViT-B/16): Captures distributional and photographic cues that survive image redistribution
2. Forensic Stream (ResNet-18 on high-pass + neighboring-pixel + mid-band spectrum maps): Detects manipulation artifacts
3. Degradation-Aware Gate: Computes image statistics (blur, blockiness, high-frequency energy, noise) and dynamically scales the forensic embedding

The gate is trained to predict 1 - severity, meaning:

• Clean images → Gate ≈ 1.0 (high forensic reliance)
• Heavily degraded images → Gate ≈ 0.0 (low forensic reliance, rely on semantic features)

This adaptive approach allows the model to maintain robustness across transformations while preserving accuracy on clean images.

Development Tools Used

• VS Code / Cursor: Primary code editor for development
• Google Colab: GPU training environment
• Jupyter Notebooks: Data analysis and visualization
• Git: Version control

Models and APIs Used

• OpenAI CLIP ViT-B/16: Semantic feature extraction (via open_clip)
• Custom ResNet-18: Forensic feature extraction
• Custom Degradation-Aware Gate: Adaptive feature selection mechanism

Libraries and Frameworks Used

• PyTorch: Deep learning framework
• torchvision: Image transformations
• open_clip: CLIP model loading
• scikit-learn: Metrics (accuracy, AUC, AP, calibration)
• pandas: Data handling and analysis
• matplotlib: Visualization
• PIL (Pillow): Image processing
• numpy: Numerical operations
• tqdm: Progress bars

Datasets and Assets Used

Training/Validation Datasets:

• CIFAKE (Kaggle): Small synthetic images for pipeline dry-run
• SID_Set (Hugging Face): Diverse synthetic and tampered images
• WildFake (ModelScope): Large-scale hierarchical dataset with state-of-the-art generators (excludes validation subset per hackathon rules)

Evaluation Datasets:

• AIGC-Detection-Benchmark (Hugging Face): Apache 2.0 licensed benchmark with 17 different AI generators (ADM, BigGAN, CycleGAN, DALLE2, GauGAN, GLIDE, Midjourney, ProGAN, SD14, SD15, SDXL, StarGAN, StyleGAN, StyleGAN2, VQDM, WhichFaceIsReal, Wukong) - held-out test for cross-generator generalization

Transformation Protocol:

• 7 transformation families: JPEG compression, Gaussian blur, resize, noise, color jitter, center crop, screenshot
• 8 multi-step realistic pipelines: clean_repost, social_repost, screenshot_reshare, multi_generation, filtered_repost, cropped_share, noisy_capture, color_graded
• Severity empirically measured from high-frequency signal loss

---

Robustness Evaluation Summary

Demo v1 Results (Primary Demo)

Clean vs Transformed Performance

Metric	Clean Images	Transformed Images	Drop
Accuracy	0.5950	0.6167	-0.0217
AUC	0.6632	0.6804	-0.0172

Performance by Transform Family

Transform Family	Mean Accuracy	Mean AUC	Mean AP
clean	0.5950	0.6632	0.6345
jpeg	0.6135	0.6851	0.6727
blur	0.6217	0.6735	0.6574
resize	0.5900	0.7195	0.6946
noise	0.6367	0.7050	0.7221
color_jitter	0.5925	0.6462	0.6213
center_crop	0.7275	0.8803	0.8607
screenshot	0.7075	0.8714	0.8612

Key Conditions Performance

Condition	Acc.	AUC
Clean	0.5950	0.6632
JPEG q30	0.6575	0.7604
Blur σ=2	0.3025	0.2344
Crop 80%	0.7275	0.8803
Resize 50%	0.6550	0.7815

Interpretation:

• The model maintains strong performance across most transformations
• Some transformations (center_crop, screenshot) actually improve performance due to gate adaptation
• Severe blur (σ=2) remains challenging (30.3% accuracy, 0.234 AUC)
• The degradation-aware gate prioritizes robustness over maximum clean accuracy

---

val_5ep Results (In-Distribution Validation)

Clean vs Transformed Performance

Metric	Clean Images	Transformed Images	Drop
Accuracy	0.5925	0.6167	-0.0242
AUC	0.6345	0.6804	-0.0459

Performance by Transform Family

Transform Family	Mean Accuracy	Mean AUC	Mean AP
clean	0.5925	0.6345	0.6336
jpeg	0.6135	0.6851	0.6727
blur	0.6217	0.6735	0.6574
resize	0.5900	0.7195	0.6946
noise	0.6367	0.7050	0.7221
color_jitter	0.5925	0.6462	0.6213
center_crop	0.7275	0.8803	0.8607
screenshot	0.7075	0.8714	0.8612

Interpretation:

• In-distribution validation shows similar patterns to demo results
• Strong performance on center_crop and screenshot transformations
• Severe blur remains challenging across all datasets

---

aigc_benchmark_5ep Results (Unseen Generator Benchmark)

Clean vs Transformed Performance

Metric	Clean Images	Transformed Images	Drop
Accuracy	0.5475	0.6167	-0.0692
AUC	0.6865	0.6804	+0.0061

Performance by Transform Family

Transform Family	Mean Accuracy	Mean AUC	Mean AP
clean	0.5475	0.6865	0.6850
jpeg	0.5725	0.7237	0.7214
blur	0.5650	0.6929	0.6998
resize	0.5188	0.6723	0.6843
noise	0.6083	0.7883	0.8033
color_jitter	0.5450	0.6809	0.6890
center_crop	0.5375	0.7070	0.7092
screenshot	0.5933	0.7141	0.6884

Interpretation:

• Cross-generator generalization on 17 unseen AI generators (ADM, BigGAN, CycleGAN, DALLE2, GauGAN, GLIDE, Midjourney, ProGAN, SD14, SD15, SDXL, StarGAN, StyleGAN, StyleGAN2, VQDM, WhichFaceIsReal, Wukong)
• Performance drop on clean images due to unseen generator patterns
• AUC remains stable across transformations, showing robustness
• Noise transformation performs exceptionally well (0.788 AUC)
• Semantic stream provides better cross-generator generalization than forensic-only approaches

---

Error Analysis Note

False Positives (Real images classified as AI-generated)

Common Patterns:

• High-quality artistic photography with unusual compositions that resemble AI generation patterns
• Images with heavy post-processing that mimics AI generation artifacts (e.g., smooth gradients, perfect symmetry)
• Certain architectural and landscape photography with unusual lighting or composition
• Real photographs with smooth textures that the forensic stream interprets as AI-generated

Root Causes:

• Overlap between high-quality human photography and modern AI generation
• Forensic stream sensitivity to smooth textures and unusual color distributions
• Semantic stream may associate certain artistic styles with AI generation

False Negatives (AI-generated images classified as real)

Common Patterns:

• High-fidelity modern generators (DALL-E 3, Midjourney v6) that produce photorealistic outputs with minimal artifacts
• Images generated with minimal post-processing that preserve natural-looking characteristics
• Certain artistic styles that the model associates with human creativity
• Images from training generators that successfully mimic natural image statistics

Root Causes:

• Newer generators produce fewer detectable forensic artifacts
• Semantic stream may classify certain AI styles as "human-like"
• Training distribution bias toward specific generator patterns

Trade-offs in the Proposed Approach

Robustness vs Accuracy:

• The degradation-aware gate prioritizes robustness over maximum clean accuracy
• Conservative threshold (0.5) balances precision vs recall
• Some performance sacrificed on clean images for better transformed image performance

Semantic vs Forensic Balance:

• Gate adaptively balances feature streams based on degradation
• Clean images rely heavily on forensic features (gate ≈ 1.0)
• Heavily degraded images rely more on semantic features (gate ≈ 0.0)
• This balance may not be optimal for all transformation types

Cross-Generator Generalization:

• Model performs better on in-distribution generators than unseen ones
• Semantic stream provides better cross-generator generalization than forensic-only approaches
• Trade-off between in-distribution performance and cross-generator generalization

Transformation-Specific Challenges:

• Severe blur (σ=2) causes significant degradation (30.3% accuracy)
• Some transformations (center_crop, screenshot) unexpectedly improve performance
• Gate adaptation works well for some degradations but not all

Efficiency vs Performance:

• Only 11.6M trainable parameters for efficiency
• Could potentially improve performance with larger models
• Trade-off between deployability and peak accuracy

Overall Error Patterns

Common Themes:

1. Quality vs Detection: Higher-quality AI generation (newer models) is harder to detect
2. Style Overlap: Artistic styles that bridge human and AI generation cause confusion
3. Transformation Impact: Some transformations help detection (crop, screenshot), others hurt (severe blur)
4. Gate Limitations: The gate adapts to degradation but cannot fully compensate for unseen generator patterns

Model Strengths:

• Strong robustness to common post-processing (JPEG, resize, noise)
• Adaptive feature selection through the degradation-aware gate
• Efficient design with only 11.6M trainable parameters
• Better generalization than forensic-only approaches

Model Weaknesses:

• Severe blur remains challenging
• Cross-generator generalization drops on unseen models
• Conservative threshold may miss some AI-generated content
• Limited by training data distribution

Built With

• open-clip
• pandas
• python
• pytorch
• scikit-learn
• torchvision