Inspiration
AI-generated images are becoming increasingly realistic, but real-world detection is not just a clean-image classification problem. Images are compressed by social platforms, resized by messaging apps, cropped into screenshots, blurred, filtered, or corrupted by noise before they reach a detector.
We therefore wanted to answer a more practical question:
Can a compact AIGC detector remain reliable after the transformations that happen during real-world image sharing?
Instead of pursuing a larger model, we focused on robustness, reproducibility, and deployment. Our goal was to build a detector that performs strongly on clean images, survives realistic post-processing, exposes its remaining failure modes, and can run locally without depending on a cloud API.
What it does
LingShu Robust AIGC Detector classifies an input image as authentic or AI-generated and outputs a continuous AIGC confidence score.
Our final model, B2-NJR, is a 21.8M-parameter ViT-based detector trained with targeted Gaussian Noise + JPEG Compression + Resize augmentation.
The system supports:
- image-level Real vs AIGC detection;
- continuous AIGC confidence scores;
- robust inference under JPEG compression, blur, resize, noise, color jitter, and cropping;
- directory-level batch inference;
- JSON output containing
image_pathandpred; - automatic CPU/GPU execution;
- fully offline inference after model files are available;
- a WebDemo using FP32 ONNX with WebGPU/WASM browser-side inference.
On our frozen held-out test set, B2-NJR achieves:
- 0.9999 clean AUC
- 99.58% clean accuracy
- 97.40% mean robust accuracy
- 92.71% worst-case accuracy
- 0.9968 mean robust AUC
Compared with our clean-only baseline, robustness training improves mean robust accuracy from 88.95% to 97.40% and worst-case accuracy from 64.08% to 92.71%, while clean accuracy decreases by only 0.07 percentage points.
We also evaluated the frozen model on the isolated COCO + DALL·E Advanced demonstration benchmark, where it achieves 0.9931 AUC and 93.57% balanced accuracy.
How we built it
We started from the open Community Forensics ViT-S/16 detector with the public OwensLab/commfor-model-384 checkpoint.
We then built a controlled three-stage development pipeline:
B0 — Open pretrained baseline
We first evaluated the original pretrained detector without adaptation.B1 — Clean-domain fine-tuning
We fine-tuned the same backbone on clean SID_Set + WildFake images to measure how much domain adaptation alone could improve performance.B2 — Robustness-aware fine-tuning
We independently tested Gaussian Noise, JPEG Compression, Resize, and Gaussian Blur augmentation. Rather than assuming that every augmentation would help, we performed one-factor-at-a-time ablations and retained only the transformations that produced complementary robustness gains.
The ablation study showed that:
- Gaussian Noise produced the largest overall robustness improvement;
- JPEG strongly improved compression resistance;
- Resize significantly improved severe downsampling robustness;
- Gaussian Blur improved blur-specific performance but added limited global value.
This led to our final B2-NJR = Noise + JPEG + Resize training recipe.
Our main benchmark contains 29,900 SID_Set + WildFake images, split into:
- 20,930 training images;
- 4,485 validation images;
- 4,485 held-out test images.
We audited exact SHA-256 overlap between train, validation, and test and found zero exact duplicates across splits. Augmentations were generated only from the training split.
The final operating threshold, 0.55657113, was selected using clean validation data only. After model selection, we froze the checkpoint, preprocessing pipeline, augmentation recipe, and threshold before evaluating the held-out test and official demonstration benchmark.
The implementation uses Python, PyTorch, torchvision, timm, Hugging Face Hub, Pillow, NumPy, pandas, scikit-learn, ONNX, WebGPU, and WASM.
Challenges we ran into
The biggest challenge was that clean accuracy alone was misleading.
Our clean-fine-tuned B1 model already achieved extremely strong clean performance, but its robustness degraded sharply under severe real-world transformations. For example, Gaussian noise at σ = 0.10 reduced B1 accuracy to only 64.08%.
A second challenge was that augmentation was not uniformly beneficial. Some transformations improved the exact condition they targeted but caused negative transfer elsewhere. Gaussian Blur, for example, greatly improved strong-blur accuracy but provided only limited overall robustness improvement, which is why we excluded it from the final recipe.
We also found that different transformations produced very different error directions:
- strong blur was mainly false-positive dominant;
- severe resizing was mainly false-negative dominant;
- strong Gaussian noise produced substantial errors in both directions.
This showed us that robustness is not simply a matter of choosing a better global threshold.
Another challenge was evaluation discipline. We needed to ensure that held-out test data and the official COCO + DALL·E demonstration benchmark never influenced training, checkpoint selection, augmentation selection, or threshold calibration.
Accomplishments that we're proud of
We are especially proud that the final robustness improvement did not require a larger model.
B2-NJR remains a compact 21.8M-parameter detector while increasing:
- mean robust accuracy by +8.45 percentage points;
- worst-case robust accuracy by +28.63 percentage points;
relative to clean-only fine-tuning.
At the same time, clean accuracy decreases by only 0.07 percentage points.
Our sample-level analysis also confirmed that the gain is not just an aggregate metric effect. Under Gaussian noise σ = 0.10:
- B1 makes 1,611 errors;
- B2 makes 327 errors;
- B2 fixes 1,455 B1 failures;
- B2 introduces only 171 new errors;
- net error reduction: 1,284 errors (79.7%).
We are also proud that the project goes beyond model training. We delivered a complete end-to-end prototype with:
- reproducible training and evaluation;
- controlled robustness ablations;
- explicit FP/FN analysis;
- directory-to-JSON inference;
- CPU and CUDA support;
- browser-side ONNX inference;
- a local WebDemo;
- frozen evaluation and deployment checks.
What we learned
The most important lesson was that robustness must be measured directly rather than inferred from clean performance.
A detector can look nearly perfect on clean data while still failing badly after transformations that are common in real-world sharing pipelines.
We also learned that augmentation should be selected experimentally rather than added indiscriminately. Different corruptions contribute differently, and more augmentation does not automatically mean better robustness.
Our error analysis showed that different transformations shift prediction scores in different directions. This means a single threshold may work well overall but still create condition-specific failure modes.
Finally, we learned that reproducibility and evaluation discipline matter as much as headline accuracy. Keeping validation, held-out testing, and external demonstration strictly separated made it possible to understand which improvements were real and which decisions were actually supported by evidence.
What's next for LingShu Robust AIGC Detector
The current system is a hackathon-scale proof of concept, not a universal AIGC detector.
Our next priorities are:
- Evaluate more unseen generator families and public benchmarks to better measure cross-generator generalisation.
- Test composite and reordered transformations, such as resize → JPEG → screenshot-like degradation, to better simulate real sharing pipelines.
- Explore transformation-aware confidence calibration to reduce blur-driven false positives and resize-driven false negatives.
- Evaluate unique-content metrics on the official demonstration benchmark to reduce the influence of repeated content.
- Add lightweight explainability and confidence monitoring to make uncertain decisions easier to inspect.
- Benchmark CPU latency, quantisation, and batch inference for more resource-constrained deployment.
Our longer-term goal is to turn the detector from a strong robustness prototype into a lightweight, transparent, and deployment-friendly AIGC screening component.
Log in or sign up for Devpost to join the conversation.