gtitr AIGC Image Detector

Spots AI-generated images even after the internet has chewed them up. Point it at a folder and it scores every file, offline, with the false-positive rate calibrated so real photos aren't accused.

Comment

AIGC Image Detector

TikTok TechJam 2026, Track 5: robust detection of AI-generated images github.com/BruhClient/Tik-Tok-Jam-2026

Inspiration

Almost every AI-image detector we looked at is demoed on clean, full-resolution files straight out of the generator. That isn't what an image looks like by the time you actually see it. By then it's been uploaded, re-encoded, resized for a feed, screenshotted by someone who wanted to repost it, and re-encoded again. Detection is close to solved on the pristine version and much less solved on the one that reaches a user.

That gap is what we built around. The question we cared about wasn't whether you can tell a diffusion output from a photograph. It was whether the answer survives the internet.

The second thing that shaped the project came out of reading the datasets before we trained on anything. A lot of public real-vs-AI datasets don't separate their classes on generation at all. They separate on image size, or file format, or subject matter. If you don't check, your validation accuracy goes up and you ship something that has learned nothing. We decided early that we'd rather report an honest 0.83 than a dishonest 0.97, and that shows up all over the rest of this.

What it does

You give it a folder of images. It gives you back a probability that each one was generated.

python detect.py <dir>        # -> predictions.json

Three other programs sit around that one and share its code, so a number you read in the GUI is the same number the CLI printed.

command what you get
python detect.py <dir> predictions.json, plus metrics if the folder is labelled
python robustness.py <dir> a transform sweep, written to robustness_report.json
python calibrate.py <dir> threshold recommendations for a given false-positive budget
python gui.py [<dir>] the same pipeline behind a desktop window

The output is exactly what the problem statement asks for, plus a verdict so the file is readable on its own:

[
  {"image_path": "C:/data/img_0001.jpg", "pred": 0.8731, "prediction": "fake"},
  {"image_path": "C:/data/img_0002.png", "pred": 0.0412, "prediction": "real"}
]

Two things worth knowing about it. The scores are always raw. Moving the threshold changes prediction and nothing else, so an export from the GUI matches whatever is on screen. And an image that fails to decode still gets a record, scored 0.5 with a null verdict, so the record count always matches the file count instead of silently dropping files.

robustness.py is the part we'd point a judge at first. It re-scores your folder under ten transforms at five severities each (JPEG, WebP, blur, rescale, crop, brightness, saturation, noise, a social-repost chain and a screenshot resample) and compares every cell against a clean baseline measured through the same pipeline. --official runs the challenge's exact parameter table instead.

calibrate.py exists because there is no single threshold that works everywhere, and we'd rather hand you the tool than pretend otherwise.

How we built it

A frozen CLIP tower with a small trained head

We use a stock openai/clip-vit-large-patch14 vision tower, roughly 303M parameters, to turn each image into a 768-dimensional embedding. We never fine-tune it. The only thing we train is a small MLP head on top: 768 to 512 to 256 to 1, with BatchNorm and dropout 0.3, about 0.7M parameters.

The reason is generalisation across generators. A network trained end to end on raw pixels learns the low-level fingerprint of whatever generators you showed it: upsampling artefacts, spectral signatures, the noise floor of one particular sampler. Those are the worst things to depend on. They differ between generators, and they're the first thing recompression destroys. Running everything through the frozen tower first collapses the image into a semantic representation, so most of that generator-specific detail is gone before the head ever sees it. The head is forced to work with something that transfers.

Training on damaged images, not clean ones

We don't show the model clean images on their own. Every training image gets cached as six views: view 0 is the original, and the other five are random degradation chains from a continuous bank: JPEG q30-95, blur sigma 0 to 2.0, resize 0.25-1.0x, noise sigma 0 to 0.10, colour 0.80-1.20x, crop 0.75-1.0. They're applied in the order those things happen to a real image: crop, resize, colour, noise, blur, then JPEG.

Training is a min-max game over that space:

min_head  max_{t in T}  L(head(CLIP(t(x))), y)

We approximate the inner maximisation by sampling 3 of the cached views per image on each step and backpropagating through the worst ones (--adv-mode cvar, which averages the worst half). Two smaller decisions matter as much as the objective:

  • We pick the checkpoint with the best worst-view AUROC, not the best clean AUROC. The model that looks best on clean data is not the one we ship.
  • We fit the threshold and the Platt calibration on pooled augmented validation scores rather than clean ones. Calibrate on clean data and the threshold drifts as soon as compression shifts the distribution, and you get a detector whose AUROC still looks fine while its accuracy quietly collapses.

The banks used for training and the banks used for evaluation never overlap, which is the only reason the robustness numbers mean anything:

bank what's in it what it measures
TRAIN_RANGES continuous ranges, sampled per view training only
EVAL_GRID the challenge's discrete settings, never trained on interpolation
HELDOUT_GRID deliberately outside the training ranges: jpeg q20, blur sigma 3.0, 0.125x resize, and WebP q50, a codec we never trained on extrapolation
CHAIN_GRID realistic chains: social repost, filtered share, thumbnail crop, screenshot the case that actually happens

Everything ships in one file

Scoring reproduces training exactly, step for step:

JPEG re-encode at q92, 4:2:0 chroma subsampling
shortest side -> 224 (bicubic), centre crop 224
CLIP normalisation (CLIP's constants, not ImageNet's)
frozen ViT-L/14 -> pooler_output -> visual_projection    768-d
L2-normalise -> (x - mu) / sd
MLP head -> logit -> sigmoid(platt_a * logit + platt_b)

Everything the model needs lives in models/bundle.pt: tower config and weights, head config and weights, feature standardisation, Platt coefficients, and the operating point. export_bundle.py writes only plain data, so it loads with weights_only=True, doesn't need a HuggingFace cache and doesn't need network access. It also means there's no separate config file that can drift out of sync with the weights, which is a class of bug we didn't want to spend a hackathon debugging.

Inside app/ the dependency direction only goes one way: widgets/ can import from app/, app/ never imports from widgets/. That's what lets the three CLI tools run on a machine with no PyQt6 installed.

Built with

Tools. Visual Studio Code for everything. Git and GitHub for version control across the team. Python 3.9+ with venv. CUDA optionally, for feature extraction. gpu_testing.py and diagnose_weights.py are in the repo because we lost time to driver and meta-device problems and wanted them diagnosable next time.

Models and APIs. openai/clip-vit-large-patch14 as the frozen tower, shipped inside the bundle so inference never touches the network. Hugging Face Hub and datasets for pulling SID_Set and Defactify, and transformers for building the tower. COCO (cocodataset.org) for reals and the benchmark. Unsplash Lite, TSV metadata plus our own image fetch. Nothing is called over the network at inference time.

Libraries.

library what for
PyTorch >= 2.0 the model, training loop, AdamW with cosine schedule, AMP
torchvision image transforms in the data pipeline
transformers >= 4.40 builds the CLIP tower; the weights come from the bundle
datasets streaming and splitting the HF-hosted corpora
NumPy >= 1.24 the feature caches (feats.npy, labels.npy) and all the numeric work
Pillow >= 10.0 decoding, JPEG and WebP re-encoding, and the entire degradation bank
scikit-learn >= 1.3 AUROC, average precision, ROC curves, Platt scaling
Matplotlib >= 3.7 score histograms, ROC and PR curves, confusion matrices, degradation curves
PyQt6 >= 6.5 the desktop app: screens, worker threads, embedded charts
Python standard library argparse, pathlib, json, csv, dataclasses, urllib, random, io

Datasets.

dataset source role how much we used
SID_Set saberzl/SID_Set on HF; OpenImages reals, fully synthetic fakes main train / val / test 25,000 per class train, 3,000 val, 4,000 test (offset so it's disjoint from train)
SID_Set tampered same dataset, label 2; real photos with an edited region and a mask evaluation probe only, never trained on 2,000 per class
Kaggle AI-vs-Human alessandrasala79/ai-vs-human-generated-dataset reals and fakes 3,000 per class, 15% held back for val
Defactify / MS COCOAI Rajarshi-Roy-research/Defactify_Image_Dataset semantically aligned pairs, labelled per generator 3,000 per generator train, 500 val, across SD 3, SD 2.1, SDXL, DALL·E 3, MidJourney v6
MS COCO cocodataset.org training reals and benchmark reals, kept apart by setup_coco.py train2017 capped around 6,000 for training; val2017 (5k) held out entirely
Unsplash Lite github.com/unsplash/datasets variety in the real class ~3,000 downloaded, keyworded portrait / person / food / city / animal
Generator pool a flat local pool, split by split_pool.py fakes, separated by generator real 70,000, SDXL 53,087, FLUX_DEV 7,273, FLUX_PRO 3,209, capped per generator so the fake set isn't 83% SDXL
Official validation benchmark COCO val2017 plus DALL·E Advanced held out, our preview of the hidden test set ~4,998 real / ~8,843 fake, about 1:1.77, which is why we use balanced accuracy

The model we're submitting was trained on a nine-group mix of the above: 21,400 images, 7,150 real and 14,250 generated. Validation was kept at exactly 50/50, since the operating point comes from it and would inherit any skew directly. Every dataset stays under its own licence and we don't redistribute any of them.

Assets. models/bundle.pt, about 1.2 GB, shipped as a GitHub Release asset rather than committed to the tree. sample_data/, a small labelled folder the GUI offers by default, gitignored and rebuilt rather than cloned. No third-party fonts, icons or stock media are bundled. The GUI's palette and type scale live in app/theme.py.

Challenges we ran into

The datasets were lying to us

The first thing we ran was an audit, not a training job, and it's the reason the project didn't go sideways. SID_Set has every synthetic image at 1024x1024 and every real image as a non-square OpenImages photo. The rule "width equals height means fake" scores in the 90s on the raw data. We would have trained on that, reported a number in the high nineties, and shipped something that had learned image dimensions.

We found four separate shortcuts and each prepare_*.py script now has an --audit mode that runs before the data is built:

shortcut where what we did
Geometry SID_Set's 1024x1024 synthetics against non-square reals --normalise-geometry: random square crop plus random resize, applied to both classes
Format PNG generations against JPEG photographs re-encode everything to JPEG on the way in, random q85-98
Resolution history Defactify pairs ~640x480 COCO reals against 1024x1024 synthetics couldn't fix it at ingest. Resizing the outputs to match doesn't undo the resampling history. Only an external test set catches this one
Content distribution prompted, aesthetic subjects against candid snapshots only addressable by mixing corpora

The threshold wouldn't sit still

This is the one that changed how we report everything. The cut-off separating real from AI moves a lot depending on which set you evaluate on. The ranking survives, and generated images still score higher than real ones, but the whole distribution slides, so a threshold that was right for one folder is wrong for the next.

That makes accuracy at a fixed threshold nearly useless as a development metric, because a bad number could mean the model can't separate the classes or that the threshold landed in the wrong place, and those need completely different fixes. So we lead with AUC, which doesn't use a threshold at all and measures separation only.

We also print the best-F1 threshold, but never as a result. It's a ceiling: what the model would score if the threshold were placed correctly for that data. Read next to AUC it splits the two failures apart: low AUROC means retrain, high AUROC with poor accuracy means recalibrate. calibrate.py handles the second case and never touches the weights.

Accuracy becomes the right metric once the threshold is genuinely fixed, and for the shipped model it is. models/bundle.pt carries its operating point at P = 0.545, calibrated for 1% FPR on real images rather than for peak accuracy, because the expensive mistake here is accusing a real photograph of being fake. So we report accuracy at that fixed point too.

A 0.05 shift hiding in the calibration

The bundle stores its operating point as a raw logit, and we assumed for a while that converting it to a probability was just sigmoid(logit). It isn't. It has to go through the bundle's own Platt coefficients: P = sigmoid(platt_a * logit + platt_b). For our bundle that's sigmoid(1.3104 * -0.0296 + 0.2205) = 0.545. The naive version gives 0.493.

A 0.05 shift, in exactly the direction that manufactures false positives, out of a system that otherwise looked completely fine. Nothing crashes, nothing looks wrong, you just quietly accuse more real photographs. The operating point is now always read from the file and never typed in anywhere.

Near-chance embeddings on GPU

At one point CLIP was producing near-chance embeddings on the GPU while the same code worked on CPU. We ruled out fp16 autocast and TF32, because three runs at different precisions gave identical AUROC trajectories and real precision differences always wobble the numbers. The remaining suspect was low_cpu_mem_usage=True leaving parameters on the meta device. diagnose_weights.py exists because of that afternoon; it checks for exactly that, and also whether two clearly different images produce meaningfully different embeddings.

Things we didn't solve

Listing these out, because they're the honest state of the project.

The false-positive rate is the real problem, not recall. Recall sits between 76% and 98% across content categories. FPR ranges from 2% on food to 24.5% on people at a single global threshold. Any serious deployment needs per-domain calibration; calibrate.py is the cheap version of that, not a solution.

Cross-family generator transfer doesn't work. Holding out FLUX_DEV costs 0.007 AUROC, which is nothing. GPT-4o images score 0.7333, and adding DALL·E 3 as the closest proxy we had access to didn't fix it. Treat any generator that isn't architecturally similar to the training set as undetected until you've measured it. That ChatGPT figure is only 19 images, so it establishes "this doesn't work" rather than giving you a number.

We don't know why people and animals are weak. We tested the obvious hypothesis, that CLIP's 224px resize was destroying the fine texture those categories rely on, and it was wrong. So it isn't a resolution problem. What it actually is, we ran out of time to find out.

Everything here is a snapshot. Category sets are ~200 images, so anything under about 0.03 AUROC is noise. The robustness grid was only run on the in-distribution test set, so we genuinely don't know whether the weak content categories get worse under compression. And it's all measured against the generators that exist right now.

We claim nothing against an adaptive adversary. Our threat model is incidental degradation, whatever happens to a picture between upload and screen. A perturbation optimised specifically against this detector is a much harder problem and we haven't touched it.

Tampered images mostly don't work. The head is trained binary, real versus fully synthetic. Locally edited photos were never in training, and since CLIP resizes to 224x224, a 3% edited patch ends up around 7x7 pixels. probe_tampered.py measures where that boundary sits instead of pretending the model handles it.

Accomplishments that we're proud of

It holds up on content it never saw

Content-category sets the head never trained on, roughly 200 images each at 50 generated to 150 real. Balanced accuracy is the headline because at 1:3, labelling everything "real" already scores 75%.

benchmark AUROC balanced acc FPR at deployed threshold
in-distribution test set 0.9639 n/a n/a
city 0.9885 0.9600 4.7%
food 0.9844 0.9500 2.0%
animals 0.9634 0.8786 18.9%
people 0.8355 0.7510 24.5%
ChatGPT / GPT-4o 0.7333 0.6278 44.4%

Holding out an entire generator costs almost nothing

We trained with FLUX_DEV removed entirely (--holdout-group FLUX_DEV) and scored 0.8894 AUROC on it, against 0.8963 and 0.8929 for generators that were in training. A gap of 0.007. Within the diffusion family, not having seen a generator barely matters. That is what the frozen tower was supposed to buy us, and it did.

The two halves of the repo agree

Same model, scored through training_process/evaluate.py and then independently through app/:

category training pipeline inference repo delta
food 0.9844 0.983 0.001
animals 0.9634 0.960 0.003
city 0.9885 0.990 0.002
people 0.8355 0.836 0.000

TPR and FPR matched exactly on food, 0.920 and 0.020. Getting two separately written pipelines to agree to three decimal places took real care about details like chroma subsampling and normalisation constants, and it means a number in the training logs is a number you can trust in the app.

We measured where it breaks instead of hiding it

All three of these are false-positive failures on inputs we deliberately put outside the training ranges:

condition accuracy FPR
heavy downscaling, 0.125x 0.790 n/a
thumbnail then crop 0.800 31.9%
WebP q50, a codec we never trained on 0.802 36.2%

The WebP number is the one we like most, because it's a real measurement of an unseen codec rather than a training artefact. That only means something because HELDOUT_GRID was kept strictly apart from TRAIN_RANGES from day one.

Caching the features let us run twenty experiments instead of three

Encoding is the only slow step, about an hour for 50k images at six views each. Once the features are cached, a head trains in seconds. That's what made the geometry ablation, three objective variants, augmentation and consistency ablations, head capacity, feature tap, preprocessing and leave-one-generator-out all affordable inside three days. Nearly everything in the "what we learned" section below exists because the experiment was cheap enough to bother running.

The whole thing runs offline

One 1.2 GB file, loaded with weights_only=True, no HuggingFace cache, no network. On a judging machine with no internet that's the difference between a demo and a screenshot of a demo.

What we learned

The real images matter more than the fake ones. We tried four external datasets and all four failed to transfer: the generator pool at 0.555 AUROC, COCOAI at 0.762 and 0.787, SID_Set at 0.637. What they had in common is that they all differed on the real side. The one merge that worked kept our original reals and borrowed only the fakes. We spent most of our early effort thinking about which generators to include and it turned out to be the wrong axis.

Variety in the real class helps; volume doesn't. Adding more ordinary web photography (COCO, OpenImages) as reals made every content category worse. Adding about 3,000 professionally shot Unsplash photos made every one better: city +0.157, food +0.192, people +0.199, animals +0.076 AUROC. Roughly the same number of images, opposite result.

The obvious explanation is often wrong, and testing it is worth the time anyway. We assumed CLIP's downsampling to 224px was destroying the fine texture that animals and nature depend on, so we trained on native-resolution 224px crops instead. It didn't help animals at all and it cost city and food about 0.065 AUROC each. The resize is now an evidenced choice rather than a default we never questioned, which is worth something even though the hypothesis failed.

One held-out set is not a neutral judge. We rejected a dataset that scored 0.6246 standalone against its own in-distribution test set, and it went on to produce the best model in the project (animals +0.076 AUROC) once we judged it on the content categories instead. That test set was narrow in generator; the category sets are narrow in content. Neither one alone should decide whether a dataset is any good.

Optimise for the worst view, not the average one. Selecting checkpoints on worst-view AUROC and calibrating on pooled augmented scores is a small change to write and the single biggest reason the robustness numbers hold. Most of the failure modes we were worried about come down to the threshold being set somewhere the deployed data isn't.

Audit first, train second. Building --audit into the prep scripts was probably the best hour we spent. Had we trained on raw SID_Set and reported the number, it would have looked excellent and meant nothing.

What's next for AIGC Image Detector

  1. Per-domain calibration as a real feature. A lightweight content classifier in front of the head, routing to a per-domain threshold, goes straight at the 2%-to-24.5% FPR spread. That's a bigger practical win than any AUROC point we could chase elsewhere.
  2. Actually diagnose the people and animals gap. Run error_analysis.py at scale on those two categories and look at the misses, then check whether the CLIP embedding itself separates them (a linear probe on the frozen features) before assuming the head is the problem.
  3. Broaden the real class deliberately rather than by volume. The Unsplash result says variety pays and count doesn't, so we'd sample reals across acquisition axes: phone versus DSLR, flash versus ambient, heavily filtered versus untouched, screenshotted versus original.
  4. Either close the cross-family gap or scope the claim honestly. That means adding genuinely non-diffusion generators (autoregressive, GAN, GPT-4o-class) and re-running leave-one-family-out, or shipping an out-of-distribution detector that abstains instead of returning a confident 0.73-AUROC guess.
  5. Run the robustness grid on the weak categories. It's cheap and it would tell us whether people and animals degrade further under compression, which is probably the biggest thing we currently haven't measured.
  6. Ensemble properly. ensemble.py already runs several heads over one shared tower for a few extra matmuls, but our two members come out of the same pipeline, so they're correlated. Training deliberately decorrelated heads, with different seeds, different corpus slices and different feature taps, is nearly free once the features are cached, and we never got to it.
  7. Build an evaluation set we trust. Every conclusion above is limited by ~200-image category sets and a 19-image ChatGPT set. Before tuning anything else we'd build something larger across generators and content, and check which of these findings actually survive it.

Built With

Share this project:

Updates

Submission history