PatchProof

Model Result on Real TikTok Office Image
Model Result on AI Upscaled TikTok Office Image
Model Predictions on AI Upscaled Augmented TikTok Office Image
Model Results for Internal Dataset Evolution
Architecture Evolution
Data Processing & Training
Final Architecture & Inference Diagram

Try our model and see the evolution here: https://patchproof.gritty-wolf-2281.chatgpt.site/ Error Analysis, Write Ups and Comparisons can be found in GitHub: https://github.com/DangerousPotential/tiktok-tjam5

Inspiration

Images rarely stay in their original form. They are resized, compressed, blurred, screenshotted, cropped, and reposted—often before someone has to decide whether they may be AI-generated. We wanted to build a practical “second opinion” for people working in content review, journalism, fraud investigation, and online safety.

What it does

PatchProof lets a user upload an image and receive a continuous score indicating whether the model considers it more likely to be synthetic or camera-captured.

Our app also exemplifies the model’s evidence path: it samples deterministic image patches, processes them through a frozen DINOv2 vision backbone, and combines the patch-level results through a compact FiSeR classification head.

Our Augmentation Lab lets users compare the prediction across JPEG compression, resizing, blur, noise, colour shifts, and cropping. This helps answer a practical question: “Would this result remain stable after the image has travelled through the internet?”

How we built it

We built PatchProof as a Vite/React web application with a FastAPI inference endpoint running on Modal GPUs. The final detector uses the public facebook/dinov2-large backbone, which remains frozen, combined with our compact FiSeR classifier and projector.

The model takes eight deterministic native-resolution crops from each image. DINOv2 produces 1,024-dimensional CLS embeddings for those crops, and the our novel FiSeR head predicts a real-versus-synthetic signal while retaining coarse source and generator information. The crop logits are averaged and passed through a sigmoid to produce the final score.

We trained and evaluated the system using source-aware and generator-disjoint protocols. The repository includes the model code, checkpoint, evaluation scripts, augmentation generator, model card, licensing notices, and reproducibility metadata.

Challenges we ran into

The biggest challenge was building a detector that generalized beyond the data it was trained on. AI-generated image detection is especially difficult because images come from many cameras, datasets, and generator families—and are often compressed, resized, blurred, cropped, or reposted before being analyzed.

Architecture Evolution

Our first prototype used a frozen CLIP backbone with native-resolution patches and frequency-based features. It achieved a clean ROC-AUC of 0.9831, confirming that local image regions contained useful detection signals. However, its ROC-AUC dropped to 0.6533 on our initial 30-image, out-of-distribution WildFake pilot. This revealed that strong internal performance did not guarantee real-world generalization.

We then developed a more complex global-and-local CLIP hybrid. It combined a whole-image view with 16 local crops, FFT residual features, attention pooling, ranking loss, and augmentation-consistency training. Although its internal ROC-AUC rose to 0.9996, its WildFake ROC-AUC fell to 0.5289—close to random ranking. We learned that adding complexity could improve internal fit without solving domain shift.

Our next iteration focused on the evaluation process itself. We introduced exact and perceptual deduplication, source-aware splits, generator-disjoint testing, balanced sampling, and robustness evaluation across 15 deterministic image transformations. This helped us distinguish genuine generalization from memorizing patterns associated with familiar datasets or generators.

These lessons led us to FiSeR, a compact, source-aware head built on frozen DINO representations. Alongside real-versus-synthetic classification, its auxiliary projection objective retains information about source and generator families. This makes source variation part of the learning process and gives us a clearer way to identify domain shift.

We initially experimented with DINOv3 + FiSeR. After watching the competition webinar, we retrained our model under the facebook/dinov2-large backbone. Our full projects adhere to the competition standards of MIT and Apache licensed sources.

The resulting DINOv2 + FiSeR model achieved ROC-AUC scores of 0.997943 on validation, 0.995596 on the generator holdout, and 0.997405 on the organizer evaluation. On the separate 250-image WildFake pilot, it achieved a ROC-AUC of 0.8719.

Evaluating the Model Honestly

At the default threshold of 0.5, the final model achieved 0.8900 recall on the WildFake pilot, but it also incorrectly flagged 65 authentic photographs, producing a real-photo false-positive rate of 0.4333. This showed us that a detector can rank images well while still requiring careful threshold calibration before deployment.

Accomplishments that we're proud of

We are proud that PatchProof became a complete, usable workflow rather than just a model checkpoint:

A live upload experience with an interactive score and model metadata.
An architecture story that shows how the detector evolved through multiple experiments.
An Augmentation Lab for testing prediction stability under common transformations.
A source-aware FiSeR design that makes domain shift part of the investigation.
A reproducible local batch-scoring workflow and open-source repository.
Strong campaign results from the selected DINOv2 model: validation ROC-AUC of 0.997943, generator-holdout ROC-AUC of 0.995596, and organizer ROC-AUC of 0.997405.
Honest error analysis showing both what the model catches and where it fails.

Most importantly, we built the product around the principle that a high score should mean “look more closely,” not “this image is conclusively AI-made.”

What we learned

We learned that excellent in-distribution performance is not enough for an image detector. A model can perform nearly perfectly on a frozen campaign split and still struggle with unfamiliar cameras, datasets, generators, or image-processing pipelines. (Our first MVP - CLIP + FFT)

We also learned that evaluation design is part of the product. Generator-disjoint testing, deterministic transformations, split hashes, external pilots, and explicit false-positive analysis give users a much more realistic understanding of model behaviour.

The architecture experiments showed us that more complexity does not automatically improve generalization. The global/local CLIP hybrid reduced some thresholded false positives but produced near-random external ranking. The final DINOv2 + FiSeR system was chosen because it offered the strongest results albeit not having the most complex architectures.

What's next for PatchProof

Our next priority is improving performance on authentic photographs from a wider range of cameras, environments, and processing pipelines. The WildFake pilot suggests that real-photo diversity and deployment-specific calibration are more valuable next steps than adding another architecture branch blindly.

We would also like to add threshold-sensitivity controls so users can see how the trade-off between false positives and false negatives changes. Future versions could include larger evaluation sets, more generator families, per-source calibration, and monitoring for distribution shift after deployment.

On the product side, we want to make the evidence easier to interpret without overstating certainty—for example, by showing confidence ranges, representative patch evidence, and clearer guidance about when a result should be escalated to human review.