Artifact

Inspiration

AI images are getting harder to recognise, especially after they have been compressed, resized, blurred, or edited online. We wanted to build something practical that could test images as people actually encounter them, not only perfect files taken directly from an image generator.

What it does

Artifact checks an image and returns the probability that it was AI generated. It accepts JPEG, PNG, and WebP files and can process an entire folder at once.

The model was tested against compression, blur, resizing, noise, cropping, and colour adjustments. Its result is an estimate rather than proof, so we present the confidence clearly and encourage people to use it as one signal among many.

How we built it

We built Artifact with Python, PyTorch, and Hugging Face Transformers.

A frozen CLIP ViT L 14 model converts each image into 768 numerical features. We then pass those features into a small neural network trained to separate authentic and fully synthetic images.

We trained using SID Set images and created transformed versions during preparation so the classifier could learn from both clean and altered images. Embeddings were stored on disk, which allowed us to train several classifier designs without running every image through CLIP again.

We also added probability calibration, selected a decision threshold using validation data, and built a command line tool that can process folders of images while skipping damaged files safely.

Challenges we ran into

The biggest challenge was handling the dataset within 32 GB of memory. Our first embedding extraction process eventually consumed almost all available RAM because too much image data was being cached.

We redesigned the loader to read small groups of records, process one data file at a time, and release memory before continuing. This reduced peak memory use to about 4.2 GB and allowed extraction to finish reliably.

We also found duplicate and visually similar images across dataset splits. These could make evaluation results look better than they really are, so we added hashing and source tracking to detect leakage before training.

Another important challenge was generalisation. Artifact performed strongly on SID Set but much worse on CIFAKE, an external dataset containing small 32 by 32 images. This reminded us that good results on one dataset do not guarantee that a detector will work equally well everywhere.

Accomplishments that we are proud of

Artifact reached 99.25 percent balanced accuracy on clean internal SID evaluation images.

Its worst balanced accuracy across the graded transformations was 98.55 percent, while its worst result across unseen transformation conditions was 97.90 percent.

We are also proud that the project is reproducible. The model revision, training configuration, calibration values, decision threshold, and checkpoint fingerprint are recorded. The command line tool uses the same inference code as the rest of the project, and all 47 automated tests pass.

Most importantly, we reported the weak CIFAKE result instead of hiding it. We believe an honest limitation is more useful than an impressive number without context.

What we learned

We learned that robustness and generalisation are different problems. Training with blur and compression can help a model handle those changes, but it does not automatically prepare the model for images from a new dataset or generator.

We also learned how easily machine learning experiments can leak information between training and evaluation. Careful source tracking, fixed validation rules, and untouched final evaluation data were just as important as the model itself.

What comes next for Artifact

The next step is to train on a wider range of authentic images and AI generators while keeping entire datasets and generator families unseen during evaluation.

We also want to build a simple web experience where users can upload an image, view the result, and compare how the prediction changes after compression or resizing.

Artifact is not intended to be the final judge of whether an image is real. Our goal is to make it a useful and transparent tool that helps people pause, investigate, and think more carefully about what they see online.

Built With

• clip
• machine-learning
• python
• pytoch