ByteMe

Performance across the full transformation grid.
Two lowest and two highest scores from one run.
Tested on the demonstration set, 5,000 COCO photos against 8,843 DALL-E 3 images.
Demo interface

What it does

ByteMe takes a directory of images and writes a JSON file scoring each one from 0 to 1 on how likely a generative model made it. It is built to keep working after those images have been compressed, resized, cropped or reposted, which is the state most images arrive in on a real platform.

Inspiration

Detection accuracy on clean images is close to solved. The moment an image passes through a social feed it gets re-encoded, downscaled and cropped, and most of the signal a detector relies on goes with it. We wanted to measure that gap rather than report a clean-data number and hope.

How we built it

A frozen CLIP ViT-B/16 encoder turns a 224-pixel patch into 768 numbers, and a small MLP head reads those and outputs one score. Only 395,777 parameters train, so a full run takes 70 minutes on a single GPU. During training every image is degraded first, using one transformation drawn at random from the challenge grid, so the head never learns to expect clean input.

We trained on SID_Set, 140,000 images after dropping the tampered class. The parquet shards are decoded straight into memory and never written back to disk, because re-encoding them as JPEG would add compression artifacts and obscure exactly the signal we are trying to read.

What we learned

The most useful thing we found was a trap in the training data. Every synthetic image in SID_Set is 1024x1024, and only 3.8% of the real ones are. Feed the model whole resized images and it can separate the classes on aspect ratio alone, scoring near-perfectly on this dataset and generalising to nothing.

Cropping a fixed 224 patch at native resolution removes image dimensions from the input entirely, and it leaves the pixel statistics intact. Generator fingerprints live in those statistics rather than in the composition.

The second thing we learned came from testing honestly. On held-out validation we reach 0.9993 AUROC and stay above 0.98 through JPEG 30, blur and downscaling. On the demonstration set, 5,000 COCO photos against 8,843 DALL-E 3 images from generators we never trained on, AUROC is 0.9574 but detection at our threshold falls from 98.9% to 47.7%.

Those two numbers say different things. The model still ranks generated images above real ones almost every time. Our threshold simply sits above most DALL-E 3 scores, whose median is 0.5917. Ranking transfers across generators. Calibration does not.

Challenges

The dataset is 126GB of parquet, which ruled out any approach that decodes images to disk first. We stream shards instead and shuffle within them.

Training also ran four times slower than it should have, until we found that autograd was building a graph across all twelve frozen CLIP layers to reach a head that did not need it.

What's next

Broader generator coverage is the binding constraint, not model size. GenImage supplies eight generators and Chameleon supplies 26,000 images that already fooled human annotators, though all of those date from 2022 and 2023. Generating our own evaluation set from current models would close that gap faster. Beyond data, CLIP was trained to match images against captions, so its features describe content, and generator fingerprints are not content. A second branch reading high-frequency residuals should age better as generators improve.