TechJammers

Inspiration

Most AI-image detectors are benchmarked on the same distribution they were trained on: clean, unmodified images straight out of a generator or a dataset folder. In today's day and age, an AI generated image goes through a ton of editing before being posted. By the time an image reaches a moderation pipeline, it's usually been re-compressed, resized into a thumbnail, cropped for a profile picture, or run through a filter. A detector that only performs well on pristine inputs is answering the wrong question. We wanted to know something more specific and more useful: does robustness to that kind of processing come from genuine augmentation, or is it something a model can fake by memorizing exact degradation patterns it was trained on? We strive to create a robust model that can accurately determine if an image is AI generated, across a wide range of augmentations.

What it does

The system fine-tunes an EfficientNet-B0 binary classifier (real vs. AI-generated) on a merged dataset built from CIFAKE, SID_Set, and WildFake, then evaluates it against a locked grid of 17 degradation conditions across six transform families mainly JPEG re-compression, Gaussian blur, resize down-and-up, Gaussian noise, colour jitter, and center crop (according to the listed criterions) plus two chained "repost pipeline" transforms. Rather than training one model, we trained four checkpoints under four different augmentation policies (none, continuous, heldout, spec) and ran all four through the identical eval grid, so any difference in robustness could be attributed to the augmentation strategy rather than noise between runs. The outcome is in the form of predict.py, which takes in an image folder, a calibrated threshold via temperature scaling along with a decision threshold selected using Youden's J statistic during our own model evaluation, and returns a confidence probability and binary classifier in a JSON file.

How we built it

We split the work into 4 sections: data curation and sanity-checking, the augmentation/transform library, the training scripts per policy, and the evaluation and prediction tooling, all synced through Google Colab mounted on Google Drive to save progress.

Our models were trained concurrently, namely: none -> Baseline with no training-time augmentation. Images are canonicalized only (compression/resolution fingerprint stripped), with no distortions applied. This isolates how well the model performs using only "raw" content-level cues, without any exposure to the transformation types it will later be tested against.

spec -> Trained using the exact deterministic parameter values specified by TechJam's fixed evaluation grid, EVAL_GIRD (JPEG quality 90/70/50/30; Gaussian blur σ 0.5/1.0/2.0; resize 0.5×/0.25× then upscale; Gaussian noise σ 0.02/0.05/0.10; ±20% color jitter; 80% center crop). This represents the best-case scenario if the model only needs to handle exactly the conditions it will be graded on.

continuous -> Trained using randomized, continuous-range augmentation parameters that deliberately do not match the eval grid's exact values, to prevent the model from simply memorizing the specific test conditions rather than learning generalizable robustness.

heldout -> Trained using continuous-range augmentation, but with parameter neighborhoods immediately surrounding the eval grid's exact values excluded from training. This tests whether the model generalizes to distortion severities near what its evaluated on without ever training on those exact points.

Each notebook loads the consolidated dataset, builds EfficientNet-B0 with an ImageNet-pretrained backbone and a replaced single-logit classifier head, and trains for 6 epochs (AdamW, cosine LR schedule), saving the best-AUC checkpoint.

Accomplishments that we're proud of

We highlighted that real and AI generated images had their own specific formatting and signatures. As such, we developed canonicalize() which aims to strip dataset-level "container" artifacts from an image before it reaches the model, like the specific JPEG compression signature, resolution, and re-encoding history left behind by whichever pipeline originally produced the file. These artifacts have nothing to do with whether an image is AI-generated, but they correlate strongly with which source dataset an image came from (SID vs. CIFAKE vs. WildFake, each with their own typical resolution/compression characteristics). Without canonicalization, a model can achieve high accuracy by learning to recognize which dataset an image belongs to rather than learning genuine AI-vs-real content cues. This shortcut inflates in-distribution metrics while quietly failing to transfer to any image processed through a different pipeline, hence when evaluating against an unseen dataset from a completely different source, the model might achieve even lower accuracy than expected.

heldout is trained with canonicalize() applied to force the model to rely on content-level signal rather than dataset fingerprints, hence making it a more robust model despite its lower accuracy.

Challenges we ran into

continuous was trained without canonicalize(). This matters specifically for the WildFake reference benchmark, because continuous's own training data includes wildfake_clean which is a different subset of the same WildFake dataset that the reference benchmark's DALL·E-Advanced and CocoVal images are also drawn from. Without canonicalization stripping that pipeline-level signature out, continuous may have learned to recognize WildFake's characteristic compression/processing fingerprint during training, and could then be partially "recognizing" that same fingerprint in the reference benchmark's AI-labeled images.

Thus its higher OOD AUC may not be because it generalizes better to unseen AI generators in general, but because it has effectively seen this dataset's signature before, just not these specific images.

To overcome this challenge, we decided that continuous's higher accuracy may not be enough grounds to reasonably justify and be the determining factor of which is the better policy due to the reason mentioned above. This supports heldout as the more defensible choice for a proof-of-concept: canonicalization is introduced in this policy to prevent such misleading result, though at the expense of raw accuracy.

What we learned

Accuracy and even AUC on a single, in-domain test set can be deceiving. Checkpoints can perform decently well on cleaned datasets, but their accuracy will drop significantly upon processing images from outside that distribution, which means the real test of "robustness" has to include an explicit out-of-distribution check. We also learnt that how augmentation breadth improves generalization greatly. For example, our continuous policy model consistently outperforms our spec model, since it is trained over a wide and diverse range of augmentations, compared to spec's specific augmentation according to the fixed evaluation grid.

Ultimately, we selected heldout as our flagship model. Its augmentation design, training on continuous-range distortions while deliberately excluding the exact neighbourhoods around the locked eval grid, is the most methodologically principled of the four for the scored evaluation itself, and it pairs with canonicalize() to strip source-specific compression/resolution fingerprints before training, directly targeting the kind of shortcut learning this reference benchmark is sensitive to.

We prioritised a proof-of-concept demonstrating robustness against the locked evaluation grid, rather than a production-grade service, and heldout's in-distribution performance (98.4% clean accuracy, 96.4% mean accuracy across all 16 distortion conditions, only a 2.0% drop from clean) proves this policy is noteworthy.

Limitations & What We'd Improve With More Time

• Dataset diversity across sources (not class balance). Real-vs-fake balance is close to even in every split (train 42,544 real / 41,843 fake; val 7,509 / 7,401; test 10,010 / 10,095). However, because the SID_Set and WildFake files are large, we mainly opted for CIFAKE images. CIFAKE accounts for the large majority of images by volume, while WildFake and SID_Set have significantly less volume (WildFake: 350/75/75 across train/val/test, and fake-only; SID_Set: 200 images total). Near-ceiling clean-validation numbers across all four policies likely reflect CIFAKE's scale more than true cross-generator robustness. Given more time, we would oversample WildFake and SID_Set relative to CIFAKE (or cap CIFAKE's contribution) so no single generator dominates what the model learns and it wont be trained to report based on a specific dataset signature.

• Single fixed decision threshold. All reported accuracy, precision, and recall use a 0.5 cutoff on the sigmoid output. We haven't calibrated or swept the threshold per degradation condition, even though confidence naturally shifts under heavy degradation. A proper threshold analysis or a calibration step would make the precision and recall numbers more representative of a real deployment setting.

• We only trained and evaluated EfficientNet-B0 exclusively. We didn't get to compare against a ResNet or a small ViT backbone to see whether the robustness gap between policies is architecture-dependent.

• The 17-condition grid covers common real-world processing (compression, blur, resize, noise, jitter, crop, and two chained pipelines) but there are no optimized perturbations or multi-generation repost chains (re-uploading a repost of a repost). Extending the eval grid to compounding degradations is something we could have done if we had more time.

• The COCO/DALL·E reference-only set is intentionally small (demo-video scale) rather than a large independent test set. It is useful as a sanity check but not as a statistically powered generalization claim.

Built With

• deep-learning
• efficientnet-b0
• image-augmentation
• image-classification
• numpy
• pillow
• python
• pytorch
• scikit-learn
• torchvision