Telltale

How we built it

Three models, built to disagree with each other.

The feature-based detectors run on hand-designed forensic descriptors — azimuthal spectrum shape, half- and [...] residual moments, channel correlation — fed to either a ~1.2M-parameter MLP or an XGBoost [...]. A generator has to fool a texture model, a semantic model and [...] simultaneously. Extraction caches to disk, so every subsequent experiment costs minutes, and XGBoost's gain table names which physical artifact the detector is keying on.

model_03 — train nothing, localize everything. A multi-scale mapper scores overlapping patches with an existing detector, blends them [into a] dense map, proposes regions by connected components, routes them to forensic specialists, and fuses with shrinkage and noisy-OR. It ships no [...], a prototype of the architecture, and the README tabulates exactly which [numbers] are deterministic.

The benchmark is a first-class [concern]. [The standard benchmark carries a single] generator family at 32×32, so cross-generator transfer is unmeasurable on it. We built three paired datasets at native resolution (ProGAN/LSUN, MidJourney/Open Images, DALL·E 3/Op[en Images]), [and spent most of] that effort removing shortcuts rather than adding data.

Challenges we ran into

Our dataset was separable at AUC > [0.9x] by [size] or artifact. Two independent leaks.

Resolution. Generated sources are fixed size while real ones are variable, so "how big is it" stands in for the label — residual standard [deviation alone separates] MidJourney from ImageNet at AUC 0.976.
Compression history. MidJourney ships [losslessly] as PNG while every real-photo corpus ships as JPEG, so "was this [JPEG-compressed]" is a near-perfect proxy — and it's exactly what 64 of our 130 FFT columns measure.

The fix was center crops at native resolution with off[sets] snapped to the JPEG 8-pixel grid, [re-encoding b]oth classes, and a per-family single-scalar shortcut probe re-run [on every build].

Our localization backend was worse [than chance]. IoU 0.460 and patch AUC 0.335 against ground-truth masks — asking a whole-[image detector to score cr]ops doesn't weakly localize, it anti-correlates, tracking local smoothness instead. Two downstream fixes (per-scale isotonic calibration, a coarse-to-fine casca[de]) [...] what a missing signal looks like as opposed to a miscalibrated one.

A metric that improved while the sy[stem got worse]. [Ranking] AUC went 0.460 → 0.955 while real photographs still averaged 0.537 against partially-AI images at 0.311. Within-image ranking is invariant to absolute level, so a m[odel can max] that metric and still flag every authentic photo.

A bug that only surfaced because [...]. Our patch evaluation read logit index 1 while that model's AI class sits at index 0, reporting a well-trained scorer as AUC 0.02. A working model looking broken is the [failure mode almost no] one catches.

The robustness prediction was wrong. We predicted that heavy degradation would drop confidence and collapse verdicts to uncertain — the safe failure. Instead confidence sat at exactly 0.60 in all 14 conditions ([making] the mechanism inert), and under noise σ=0.05 only 9% of images kept the verdict they had when clean while AUC barely moved. Ranking survives; labels churn. A user re-u[ploading a slightly different c]opy gets a different answer.

Accomplishments that we're proud of

The leave-one-generator-out table. [In-distribution, ~0.9x across] all three families. Held out: 0.686 / 0.850 / 0.822 with accuracy near chance — and the failure mode is specific. FPR stays at 1.5–3.4% while F1 collapses, meaning unseen [generators get cl]assified as real. That is the costly direction of error for moderation, [and it is invisible] in a single-generator evaluation. Producing that table, and publish[ing the out-of-di]stribution number, is the contribution we'd defend hardest.

Evidence that localization is worth its complexity. On fully-AI images a whole-image detector wins (0.935 vs 0.889). On partially-AI i[mages it drops to 0.xxx] — while the region-aware pipeline reaches 0.589, a significant gain on the subset where the thesis is actually contested.

We killed our own optimizations whe[n they didn't pay]. The coars[e]-to-fine cascade removed 35% of windows for a 1.19× speedup, and moved scores by up to 0.66 while cutting regions on genuinely tampered images from 4 to 1. Cheap [wins on easy images, wrong answers on] hard ones. It ships off, documented rather than deleted. Separately, mo[ving to batched GPU inference] gave a 3× win and took utilization from 21% to 96%, which is the kind of optimization worth keeping — and larger batches, TF32 and channels_last were all measured and [dropped as no-op]s.

What we learned

Robustness and generalization are d[ifferent axes, and we conflated] both. Degradation was not our binding constraint: AUC held at 0.674–0.802 [aga]inst 0.782 clean. That's less reassuring than it sounds — the sys[tem is blunt rather] than fragile. The real gap is cross-generator, and it is enormous.

Ranking transfers; calibration does not. The in-distribution 5%-FPR threshold calibrates to 0.05–0.12, and on a held-out genera[tor it is simply] wrong. It is why AUC (0.69–0.85) reads so much better than accuracy, [and why any syst]em shipping a fixed 0.5 threshold is making a claim it can't support.

Almost every strong number we prod[uced was worth a second look]. The discipline that actually moved the project was adve[rsarial: assume the number is] lying, find the mechanism, then decide.

What's next for Telltale

A compound-degradation benchmark. Real re-uploads chain operators — resize, then recompress, then resize again. We're building RealDeg-style chains of N ∈ {1..5} operators sampled with replacement, scored by balanced accuracy, with the sampled sequence recorded so every condition is reproducible.
Settle the local-artifact hypothesis. The train-crop/resize × test-crop/resize 2×2. Published results show a crop-trained model losing ~8.8 BAcc under resize while a resize-trained model barely moves — and our inference path resizes. A few GPU-hours either confirms the diagnosis or redirects the whole effort.
Two interventions from the literature, ablated rather than stacked. Texture-aware masking targets cross-generator transfer; deep-feature smoothing targets degradation. They intervene at opposite ends of the network and one paper's method is exactly the category the other argues against. We'll measure, not assume.
Train the scorer on the distribution the mapper actually queries — patches, with the ambiguous 5–70% mask-coverage band discarded rather than forced to a label, and hard-negative mining on the real-image patches that currently produce every false positive.
Make confidence mean something. Uncertainty is the safe failure mode for a moderation tool, and ours is currently a constant.