PixelProof (by Simpsons)

Inspiration

We picked challenge #5 partly because AI-generated image detection felt like one of those problems where the naive solution — bolt a classifier onto a pretrained vision model — looks fine in a demo and then falls apart the second someone reposts, resizes, or JPEG-compresses the image. Which is exactly what happens to basically every image on the internet within seconds of being posted. The brief said as much itself: don't just fine-tune a classifier, think about what it's actually learning, and test it against the kind of damage images take in the real world, not just clean benchmark pairs. That's what pulled us in — less "can you train a good classifier," more "can you build something that doesn't quietly memorize a shortcut."

What it does

PixelProof takes an image, or a whole folder of them, and outputs a calibrated probability that it's AI-generated.

bash run.sh predict --input_dir path/to/images --out predictions.json
# → [{"image_path": "...", "pred": 0.87}, ...]

That's the actual required deliverable — a script, not a demo notebook. Under the hood it's a hybrid model:

image → CLIP ViT-B/32 (frozen)  ─┐
                                   ├─→ fusion MLP → learned temperature → calibrated probability
image → FFT log-magnitude → CNN ─┘

• Semantic branch — frozen CLIP, "does this look real"
• Frequency branch — small CNN over the FFT spectrum, "does this have a generator's statistical fingerprint"

We went hybrid because those two signals tend to survive different kinds of damage — CLIP's semantic read holds up reasonably well under blur and noise, while frequency-domain artifacts are exactly what compression and downsampling are most likely to disturb.

How we built it

Four of us, split by pipeline stage:

• Ngiam — data pipeline: fetching WildFake and SID_Set (including reading directly from remote zip/parquet without downloading the whole dataset locally), building the manifest that drives everything downstream
• Letao — semantic branch + training loop
• Aaron — frequency branch + fusion head
• Aarav — robustness evaluation + calibration

Stack:

• PyTorch + torchvision — model and training loop
• open_clip_torch — CLIP backbone (openai ViT-B/32 weights, the open-source checkpoint, no paid API involved)
• scikit-learn — AUC / accuracy
• pandas — manifest-driven dataset loader
• Pillow + OpenCV — image I/O and the transform pipeline (JPEG recompression, blur, resize, noise, color jitter, crop)
• Hugging Face datasets + pyarrow — pulling SID_Set
• VS Code + terminal — a Python venv and run.sh as the single entrypoint the brief asked for (train, build_testset, evaluate, calibrate, predict, find_errors)

Training data is WildFake — multi-generator (ADM, StyleGAN3, VQVAE, plus real images), which mattered a lot given how central generalization is to this brief. SID_Set is held out completely and never touched during training, so our "unseen generator" number is an honest one. CIFAKE was used exactly once, day one, purely as an end-to-end smoke test before we committed hours to the real WildFake run — it's not part of the final model.

We deliberately kept the architecture to two branches instead of something fancier (a DCT or PRNU branch came up in planning and got cut). A model that actually finishes training on hackathon-weekend compute with real numbers behind it beats a more elaborate idea that doesn't run in time.

Robustness Evaluation Summary

The robustness half of this actually works, across the full transform grid the brief describes — not a cherry-picked subset. Clean vs. every post-processing condition we tested:

Condition	Acc.	AUC
clean	0.87	0.94
jpeg_q90	0.84	0.93
jpeg_q70	0.85	0.93
jpeg_q50	0.82	0.91
jpeg_q30	0.82	0.89
blur_sigma0.5	0.85	0.93
blur_sigma1.0	0.81	0.91
blur_sigma2	0.71	0.82
resize_0.5x	0.81	0.90
resize_0.25x	0.75	0.82
noise_sigma0.02	0.83	0.91
noise_sigma0.05	0.77	0.87
noise_sigma0.10	0.73	0.82
color_jitter	0.82	0.92
crop_80pct	0.81	0.90
unseen_generator	0.48	0.51

Final Score ≈ 0.915 (0.5 × AUC_clean + 0.5 × mean of the 14 post-processing rows), and every one of those 14 conditions stays above 0.82 AUC even at the harshest settings we tested.

(This table is the brief's required Robustness Evaluation Summary — deliverable 5.5.4. Same numbers, plus how we got them, are in docs/error_analysis.md in the repo.)

Challenges we ran into

Cross-generator generalization was the big one — the 0.51 AUC row above. A coin flip, on SID_Set, a generator family the model's never seen, despite scoring 0.94 on WildFake's own generators. That's a known hard problem; the brief calls it out directly as one of the two things that make this task genuinely difficult, and says there's no silver bullet for it.

We built a diagnostic to check whether the frequency branch was actually contributing anything, since the entire premise of going hybrid was that it'd catch what CLIP misses. Honest answer: not really — zeroing it out barely moved predictions anywhere we tested. We tried two fixes:

1. A higher learning rate for the frequency branch, so it wouldn't get drowned out by the much larger, pretrained CLIP branch. It did start mattering more — but what it learned made cross-generator performance worse, not better. It was locking onto WildFake-specific artifacts, not anything general.
2. Wider training-time noise/resize augmentation, to better match our harder eval conditions. Also came out slightly worse across the board, most likely because adding another randomly-chosen augmentation option diluted how often the existing ones fired.

Both got reverted, and both got written up as negative results rather than quietly deleted. We'd rather show "we tried the obvious fix, it made things worse, here's our best guess why" than pretend the first thing we tried just worked.

We also caught this one, which if we'd missed it would have wrecked everything else:

WildFake's real images are almost entirely JPEG and its fakes are almost entirely PNG — a lazy model could get a suspiciously good score by learning "is this a PNG" instead of "is this AI-generated." We now re-encode everything to a common format/resolution before it ever reaches the model.

Error Analysis Note

When we scored our balanced clean/no-transform test set (420 images), the model's most confident false positives and false negatives weren't random.

22 of 210 real images (10.5%) got scored as AI-generated; 34 of 210 AI images (16.2%) got scored as real — and the false negatives cluster hard by generator:

Generator	False negatives	Rate
adm (diffusion)	4 / 70	5.7%
stylegan3	16 / 70	22.9%
vqvae	14 / 70	20.0%

adm — a diffusion model, the same family WildFake trains on most — gets caught almost every time. stylegan3 and vqvae slip past roughly 4x as often, which lines up with the unseen-generator result above: the model's strength tracks specific generator fingerprints it saw a lot of in training, not a clean "diffusion vs. GAN vs. VQ" boundary.

What the actual misses look like:

False positive (real, scored as AI)	False negative (AI, scored as real)
A real, professionally-lit studio photo — pred 0.95 (AI)	A StyleGAN3-generated headshot — pred 0.01 (real)

That pattern held across all 8 of our top examples: false positives tend to be unusually polished or texture-heavy real photos (macro shots, glossy studio portraits); false negatives tend to be mundane, photorealistic AI portraits and candids with nothing visually "off." Our read: the model is keying off surface polish more than a true generative artifact — a real limitation, not just a number.

(This section covers the brief's required Error Analysis Note — deliverable 5.5.5. All 8 representative examples, the full breakdown, and the trade-offs discussion are in the same docs/error_analysis.md.)

Accomplishments that we're proud of

The numbers above are half of it. We're just as proud of how we handled the parts that didn't work: instead of quietly reverting the two failed fixes and moving on, we built the tooling to understand why they failed (a gradient-norm check and an ablation check on the frequency branch), and came away with a diagnosis we actually trust — this is a representation/data problem, not an architecture or optimization problem. That's a more useful thing to know than a number.

What we learned

Optimization tricks — a higher learning rate for an undertrained branch, wider augmentation ranges — can't manufacture generalization out of a training set that doesn't have the diversity you actually need. If anything, giving an underfit branch more capacity just let it overfit harder to what was already there. We only really believed "the fix is more representative data, not a fancier schedule" after watching two well-motivated attempts at the latter both make things worse.

We also learned that a small pipeline bug can silently sabotage several scripts at once if they copy the same pattern. The same dual-tensor wiring mistake — feeding the wrong tensor into the frequency branch — showed up independently in our calibration script and one of our diagnostic scripts, because both were adapted from the same evaluation logic. Same bug, three places, three separate fixes — worth checking every copy explicitly rather than assuming one fix covers the rest.

What's next

• Add a second, generator-diverse training dataset (GenImage, covering several more diffusion and GAN families) — without touching SID_Set, so the unseen-generator number stays an honest, untrained-on measurement.
• Revisit the augmentation-widening idea with a better-targeted approach: reweight how often each op fires instead of adding a new one to an unweighted random choice.
• Add a false-positive-rate discussion at a chosen threshold, now that our calibration script produces real calibrated probabilities to work with.

Built With

• bash
• clip
• cnn
• computer-vision
• deep-learning
• fft
• git
• huggingface-datasets
• image-processing
• machine-learning
• matplotlib
• numpy
• openclip
• opencv
• pandas
• pillow
• pyarrow
• python
• pytorch
• pyyaml
• scikit-learn
• torchvision
• vscode