SMIFF: Synthetic Media Image Forensics

SMIFF Landing Page
SMIFF Real Detection
SMIFF Robustness Evaluation Summary

Inspiration 💡

AI-generated images are everywhere now, and they don't stay clean. A fake image gets screenshotted, sent through WhatsApp, reposted on TikTok, compressed, cropped, resized. By the time it reaches a moderator or a platform's detection system, it's been through several rounds of degradation. We tested existing AI-image detectors against this reality and found something alarming: accuracy drops 30–50% on real-world compressed images, sometimes falling to near-random guessing.

Scammers don't need to fool a detector on a clean image, they just need to let the internet's own compression pipeline do the work for them. That's the gap we set out to close: not another detector that scores well on a lab benchmark, but one engineered from the ground up to survive redistribution.

Key Features ✨

🔍 Dual-Stream Detection Architecture

Frozen CLIP ViT-B/32 vision backbone for semantic image features
Independent frequency-domain CNN reading the 2D FFT log-magnitude spectrum for generator fingerprints
Fusion MLP head (1024 → 256 → 128 → 1) combining both streams into a single fake-probability logit

🧩 Patch-Level Robustness

Overlapping patch extraction + one global downsampled view per image
Learned attention-pooling aggregator, weighting patch-level predictions instead of naive averaging
Improves resilience when only part of an image carries generator artifacts

🌐 Local, Deployable Forensic Tool

Flask backend + browser-based frontend for single-image and ZIP batch scanning
Batch ZIP uploads generate an Excel/CSV report with filename, prediction, probabilities, and confidence
Fully local inference, no external API dependency, CPU or CUDA auto-detected

Overall Flow 🔀

Input image → patch extraction (overlapping crops, 224×224, CLIP-normalized) → parallel CLIP ViT branch (frozen, transformer blocks, pooled CLS token) and Frequency CNN branch (grayscale FFT → 3 conv blocks) → feature fusion via concatenation + MLP head → patch aggregation via attention pooling → sigmoid output → REAL / AI-GENERATED classification (or batch Excel report for ZIP input).

How We Built It 🛠️

Dataset:

CIFAKE benchmark (real photos vs. Stable Diffusion 1.4-generated images), trained on 10k images/class for 10 epochs given the hackathon time constraint

Robustness Engineering:

A dedicated RandomRobustnessAugment pipeline simulating real-world redistribution: JPEG re-compression at multiple qualities, social-media-style downscale + re-encode, Gaussian blur, resize/upscale distortion, sensor and low-light noise, color jitter, and center-crop framing shifts
Random subsets of these transforms applied per training sample, so the model never sees only clean images

Model Selection:

Compared CLIP ViT-B/32, CLIP ViT-B/16, a timm backbone, and an augmented timm variant across six transform categories
Chose CLIP ViT-B/32 over the higher-scoring augmented timm model for greater stability and generalization, prioritizing consistent real-world performance over peak clean-set accuracy

Evaluation ⚡

We stress-tested SMIFF across 19 real-world transformations, clean, JPEG at q30/q50/q70/q90, social media re-encoding, Gaussian blur (σ 0.5–2.0), resize down to 0.25×, noise, low-light noise, color jitter, center crop, and composed multi-attack scenarios stacking resize + JPEG + jitter together.

AUC stayed above 0.94 across nearly every transformation, and above 0.98 on the attacks that matter most in practice: compression and re-encoding
Highest false-positive rate appeared under aggressive resize (0.25×) and heavy blur (σ 2.0), where over-smoothing was misread as a generation artifact
Highest false-negative rate appeared under noise injection and composed resize+JPEG attacks, where added noise was misread as authentic sensor noise

Error Analysis 🔬

We didn't stop at aggregate metrics, we dug into individual failures:

False Positive (99.44% AI on a real photo): A heavily retouched celebrity photo was flagged as AI-generated. Root cause: CIFAKE's "fake" class is built from older SD1.4 outputs, a completely different distribution from modern retouched photography. The frequency branch's "smooth = fake" shortcut compounded the error.
False Negative (97.32% real on an AI-generated image): A stylized AI illustration was flagged as real. Root cause: CIFAKE's fake class is photorealistic SD1.4, not illustrated content — fine detail (snowflakes, sparkles) was misread as camera noise, and CLIP had no exposure to this style during training.

Both failures trace back to the same root cause: narrow training distribution, which gives us a clear, actionable fix rather than a mystery to chase.

Development Tools & Tech Stack 🧰

Development Environment:

Google Colab and Jupyter Notebook: model training and experimentation
VSCode: backend/frontend development and code editing
Local development with Python virtual environments (venv)
CPU/CUDA auto-detection for training and inference

Models / APIs Used:

CLIP ViT-B/32 vision encoder (openai/clip-vit-base-patch32) via Hugging Face transformers, used as a frozen semantic feature extractor
Custom frequency-domain CNN (3 conv blocks) operating on the 2D FFT log-magnitude spectrum of grayscale images
Custom fusion MLP head combining both feature streams into a single classification logit

Libraries & Frameworks:

PyTorch (torch, torchvision) - model architecture, training loop, FFT computation
Hugging Face transformers - CLIP backbone loading and inference
Flask + flask-cors - backend server and REST API
Pillow, numpy - image preprocessing and array operations
openpyxl - Excel report generation for batch ZIP scans
python-multipart, uvicorn, fastapi - listed dependencies for API tooling

Datasets & Assets:

CIFAKE benchmark dataset (REAL vs. Stable Diffusion 1.4-generated images), used for training and validation
Custom robustness evaluation suite (19 transformations: JPEG compression, blur, resize, noise, color jitter, crop, composed attacks) built in-house for stress-testing

Accomplishments We're Proud Of 🏆

Built a detector that holds AUC ≥ 0.94 under nearly every real-world degradation tested, instead of collapsing like standard detectors do
Diagnosed our own failure modes with root-cause precision instead of shipping a black box
Delivered a fully working local tool: browser UI, single-image and batch ZIP scanning, Excel/CSV reporting, not just a notebook

Limitations 🔐

Trained on a single benchmark (CIFAKE), so performance can drop on unseen generator families (Midjourney, SDXL, Flux) or stylized/illustrated content
Frozen CLIP backbone caps the ceiling, features were never fine-tuned specifically for the real-vs-fake task
Augmentation covers redistribution artifacts (compression, resize, crop) but not content or style shift
No defense yet against subtle adversarial manipulation or watermark-based attacks
Research/demo scale, no confidence calibration or production monitoring in place

What's Next 🚀

Diversify training data, modern generators (Midjourney, SDXL, Flux) and real illustrated/retouched content, directly addressing our error analysis findings
Cross-generator & cross-dataset evaluation using held-out generator testing
Unfreeze / fine-tune CLIP to close the domain gap instead of relying on frozen features
Sharpen the frequency and patch modules for more localized artifact sensitivity
Add confidence calibration and a production-ready batch UI for real deployment