cya | Devpost

robustness evaluation summary
error analysis

Inspiration

As generative models improve, distinguishing authentic photographs from fully AI-generated images is becoming increasingly difficult. The problem becomes even harder after images are shared online, where platforms routinely resize, compress and re-encode them.

Many detectors perform well on untouched images but depend on fragile generation artifacts that disappear during redistribution. We built cya to explore a more practical question: can an AI-image detector remain reliable after the image has undergone the ordinary transformations applied by websites, messaging services and social platforms?

What it does

cya is a binary image detector for two clearly defined classes: fully authentic images and fully AI-generated images. Mixed, composited and AI-edited images are intentionally outside its scope.

Users can upload one or more images through the React interface or process an entire directory with the command-line tool. For every valid image, cya returns an authentic or ai_generated classification with a probability-based confidence score.

Before model inference, cya checks the received file for a valid C2PA manifest. Only a cryptographically verified claim that explicitly identifies the image as AI-generated can produce an early AI-generated result. Missing, invalid or authenticity-related metadata is never treated as evidence that an image is real; it simply falls through to the visual detector.

The same inference pipeline powers the CLI, FastAPI backend and frontend, preventing the visual demonstration from behaving differently from the submitted script.

How we built it

Model architecture

Our production detector uses controlled RINE with a frozen OpenAI CLIP ViT-L/14-336 vision backbone. RINE (Representations from INtermediatE layers) is built on a simple observation: a frozen CLIP backbone builds progressively richer representations at each transformer layer, and the final layer isn't necessarily the most discriminative one for spotting synthetic images — useful evidence is often spread across several intermediate layers instead. Rather than using only CLIP's final representation, RINE extracts intermediate CLS-token features from layers 6, 12, 18 and 24, then learns a small trainable importance estimator that weighs and combines those layers before a lightweight binary classifier produces P(ai_generated). The CLIP backbone itself stays completely frozen, so only the importance estimator and classifier head are trained.

Freezing CLIP kept the trainable model small and reduced the risk of overfitting our available dataset. It also allowed us to reuse representations learned from a broad collection of web images.

We began with a conventional frozen-CLIP linear-probe baseline similar to the approach popularized by UnivFD. We selected RINE because intermediate CLIP layers retain different levels of visual and semantic information. In our evaluation, the frozen-CLIP baseline achieved a 94.71% mean combined score, while robustness-controlled RINE reached 99.81%.

Why CLIP instead of frequency detection

Frequency-based detectors can identify spectral traces produced by generative decoders, but those traces are concentrated in the same high-frequency information removed by JPEG compression, blur and resizing.

We implemented and evaluated frequency magnitude, residual, phase and periodicity features. The strongest standalone frequency representation reached 83.03% clean accuracy, while RINE-plus-frequency fusion achieved only a 52.15% combined score and became unstable across seeds. We therefore disabled the proposed frequency fast-track and retained frequency analysis only as diagnostic evidence.

This was an important design decision: the absence of a synthetic frequency pattern cannot prove that an image is authentic, because routine redistribution may have erased that pattern.

Dataset preparation

Our primary dataset was SID_Set, which contains real photographs and synthetic images from generators including FLUX, Kandinsky, SDXL and AbsoluteReality.

We streamed 10,000 authentic and 10,000 synthetic examples through Hugging Face. SID_Set's tampered-image class was excluded completely because AI-edited images fall outside our binary scope. After integrity and duplicate auditing, 19,882 images were eligible for the project's controlled data contract.

During preparation, we found that compression history could reveal the label independently of image content. A detector could therefore appear accurate by learning dataset-specific JPEG differences instead of authentic-versus-generated visual evidence.

To reduce that shortcut, both classes were decoded and re-encoded with the same fixed JPEG quality-96 policy, RGB conversion, metadata removal and chroma-subsampling settings. This matched normalization was performed offline while constructing the canonical dataset. It is deliberately not applied during inference because cya must score the exact image it receives, without silently modifying it or adding another transformation.

For experimental physical-camera analysis, we evaluated licensed native images from PREMIER Dataset v3 N1/N2 and AI-generated images from Tiny-GenImage, a repackaging of AI-only rows from the official GenImage dataset. These images supported the PRNU research track and were not used to train the retained production classifier.

Robustness design

We created 14 independent transformation cells covering:

JPEG quality 90, 70, 50 and 30
Gaussian blur at three severities
Gaussian noise at three severities
Resize round trips at 0.5× and 0.25×
Colour jitter
Centre cropping

Every transformed image was generated directly from its clean parent and received exactly one transformation. This prevented chained transformations from making the benchmark ambiguous.

A resize round trip was treated as one compound transformation because downsampling and restoring the image together simulate how a platform reduces an image and later displays it at its original dimensions. Both steps used bilinear interpolation. Resize results were stored losslessly so the resize test did not accidentally become a resize-plus-JPEG test.

We also evaluated a local-texture branch that combined global context with detailed crops selected before CLIP resized the image. It matched the main model on clean data but fell to 93.13% mean robustness, compared with 99.80% for controlled RINE on the same tested cells. Its AI-generated accuracy dropped close to chance under aggressive downsampling, so the local branch was rejected rather than shipped.

Development tools

We used:

Google Colab and Jupyter notebooks for GPU training and evaluation
VS Code for development and debugging
Google Drive for durable Colab checkpoints and experiment artifacts
Git and GitHub for version control and branch integration

Models, libraries and frameworks

The project uses:

OpenAI CLIP ViT-L/14-336 and the RINE architecture
PyTorch for model training and inference
Hugging Face Transformers for loading the frozen CLIP backbone
NumPy, pandas and scikit-learn for data processing, metrics and evaluation
Pillow, OpenCV and scikit-image for image processing and forensic experiments
c2pa-python for provenance verification
FastAPI and Uvicorn for the backend
React, TypeScript and Vite for the frontend
pytest and Ruff for automated testing and code quality

No commercial detection API is used in the submitted inference path.

Challenges we ran into

The largest challenge was preventing the model from learning shortcuts. File encoding, compression history, image dimensions and generator families can correlate with labels without representing genuine evidence of authenticity.

Another challenge was resisting the temptation to retain every promising feature. Frequency, Lab-colour, PRNU and local-texture models each produced useful diagnostic results, but none improved the locked clean-and-robust score without unacceptable regressions. The final system became simpler as our evaluation became more rigorous.

We also had to manage 19,460 transformed images efficiently in Colab. Copying every derivative through Google Drive was extremely slow, so we changed the workflow to regenerate deterministic transformations on Colab-local storage and synchronize only durable checkpoints, manifests and reports.

Accomplishments that we're proud of

Controlled RINE achieved:

100.00% mean clean accuracy
99.62% mean robustness accuracy
99.81% combined clean-and-robust score across seeds 42, 43 and 44

After freezing the model, checkpoint, threshold and feature set, we evaluated the sealed final test set exactly once. cya achieved 99.29% overall accuracy across 141 images:

69/69 AI-generated images correctly detected
71/72 authentic images correctly detected
0.00% AI-generated false-negative rate
1.39% authentic false-positive rate

We are also proud that the submitted package is not just a notebook demonstration. It includes a tested directory-based CLI, committed classifier checkpoint, FastAPI service and working frontend, all using the same prediction implementation.

What we learned

We learned that clean accuracy alone is not a meaningful measure of real-world reliability. A feature can look excellent on untouched images and fail completely after compression or resizing.

We also learned that additional signals do not necessarily make a stronger detector. Frequency, colour, sensor and texture features must demonstrate incremental value through controlled ablations rather than being retained because their theory sounds convincing.

Finally, physical-camera signals such as PRNU require suitable native camera data, multiple images per device and careful device-held-out evaluation. They should not be interpreted as proof of authenticity when extracted from recompressed or platform-processed images.

What's next for cya

Our next priorities are to:

Calibrate confidence scores using a validation set containing enough errors to support meaningful calibration
Validate C2PA parsing against real, signed AI-generation manifests
Measure latency, memory consumption and cache requirements on target deployment hardware
Expand evaluation to additional authentic sources and unseen generator families
Revisit texture and physical-camera signals using better native data and more robust fusion methods
Extend the product cautiously to AI-edited or mixed-content detection as a separately defined task, without weakening the current binary contract