spoTTing The difference

naruto or cappuccino assassino? AI image detection that works in the real world — 98.14% accuracy across 15 transforms (JPEG, blur, noise, resize, crop, jitter) with a frozen threshold.

Comment

Try out our LIVE Telegram Bot (link below!! 👇)

spoTTing The Difference

TikTok TechJam 2026 Track 5 Submission

Team: 三T

Inspiration

AI-generated images are everywhere now! On social media, in ads, even in news articles. But how do you tell what's real and what's AI?

We wanted to build a detector that works in the real world and not just on clean, pristine images in a lab, but on the compressed, blurry, noisy images people actually share online. The TikTok TechJam challenge forced us to solve this properly: maintain high accuracy across 15 real-world transforms with a frozen decision threshold.

What it does

Our system detects whether an image is AI-generated or real, even after it's been through real-world degradations like:

  • JPEG compression (quality 30-90)
  • Gaussian blur (σ 0.5-2.0)
  • Resize and upscale (0.25x-0.5x)
  • Gaussian noise (σ 0.02-0.10)
  • Color jitter (±20%)
  • Center crop (80%)

Results on the official benchmark:

  • Clean accuracy: 98.60%
  • Mean transform accuracy: 98.14%
  • Worst-case transform: 97.10% (noise)
  • AUROC: 0.996+

How to use it:

# Scan a single image
python scan_image.py path/to/image.jpg

# Or use the Telegram bot
python telegram_bot.py
# Send an image to the bot → get "AI" or "REAL" response

How we built it

Architecture

Input Image → CLIP ViT-L/14 (frozen) → 768-dim features → 8 Logistic Regression Probes → Weighted Average → P(AI)
  • Backbone: OpenAI CLIP ViT-L/14 (304M params, frozen)
  • Probes: 8 lightweight logistic regression classifiers (769 params each)
  • Ensemble: Weighted average of P(AI) across all 8 members
  • Threshold: Frozen at 0.5 (no tuning on eval data)

The Math

1. Feature Extraction

Given an input image \(x\), we extract features using frozen CLIP:

$$z = \text{CLIP}(x) \in \mathbb{R}^{768}$$

We then L2-normalize to ensure consistent scale:

$$\hat{z} = \frac{z}{|z|_2}$$

2. Logistic Regression Probe

Each probe learns a linear decision boundary:

$$P(\text{AI} \mid \hat{z}) = \sigma(w^\top \hat{z} + b)$$

where \(\sigma(t) = \frac{1}{1 + e^{-t}}\) is the sigmoid function, \(w \in \mathbb{R}^{768}\) is the weight vector, and \(b \in \mathbb{R}\) is the bias.

3. Class Weighting

To fix calibration (push AI probabilities higher), we use weighted loss:

$$\mathcal{L} = -\sum_{i=1}^{N} \left[ w_{\text{AI}} \cdot y_i \log(\hat{y}_i) + w_{\text{real}} \cdot (1-y_i) \log(1-\hat{y}_i) \right]$$

where \(w_{\text{AI}} = 4.5\) and \(w_{\text{real}} = 1.0\) for most probes.

4. Ensemble Averaging

The final prediction is the weighted average across all 8 probes:

$$P_{\text{final}}(\text{AI} \mid \hat{z}) = \frac{1}{8} \sum_{k=1}^{8} P_k(\text{AI} \mid \hat{z})$$

5. Transform Augmentation

For each AI training image \(x_i\), we generate augmented copies:

$$\mathcal{D}_{\text{aug}} = {x_i, T_1(x_i), T_2(x_i), T_3(x_i)}$$

where \(T_j\) are random transforms (JPEG, blur, noise) sampled from the competition's 15-transform spec.

Key Innovation 1: Transform Augmentation

Problem: Our baseline dropped to 73-75% accuracy under noise and blur.

Solution: For each AI training image, we created 3 random transformed copies (JPEG, blur, noise) during feature extraction. This exposed the model to degraded AI images during training.

Result: Mean transform accuracy improved from 89.2% → 93.29%; worst-case from 78.1% → 91.9%.

Key Innovation 2: Pure Single-Generator Probes

Problem: Our 5-member ensemble (all trained on mixed generators) plateaued at 97.39%. Adding more "superset" probes made it worse.

Insight: Superset probes are highly correlated: they all learn similar patterns. We needed diversity.

Solution: Train individual probes on ONE generator's AI images + real negatives only. Each pure probe learns generator-specific signatures.

Result: Adding 3 pure probes (SD3-only, Midjourney-only, Sana-only) improved mean accuracy from 97.39% → 98.14%; worst-case from 96.80% → 97.10%.

Training Data

We combined 6 independent AI image sources:

  • SID_Set (Hugging Face): 8,000 images (4,000 real + 4,000 AI)
  • ComfyUI (own generator): 8,000 AI images
  • Flux_AIGC_Dataset: 3,000 AI images
  • Sana: 2,000 AI images
  • Midjourney: 3,000 AI images
  • SD3-medium: 6,600 AI images

Total: ~130,000 features (with augmentation)

Challenges we ran into

1. CIFAKE Dilution

Issue: Including CIFAKE (32×32 images) in training hurt performance on natural-resolution images.

Finding: CIFAKE's low resolution created conflicting signals. Dropping it improved clean accuracy from 81.1% → 88.0%.

2. Calibration vs. Discrimination

Issue: Training on SID_Set alone gave excellent AUROC (0.983) but poor accuracy (0.891) at the frozen t=0.5 threshold.

Diagnosis: The model ranked images correctly (high AUROC) but pushed AI probabilities too low. This was a calibration problem, not a discrimination problem.

Solution: Class weighting — upweighting the AI class during training shifted the decision boundary. At ai_weight=5.0, clean accuracy jumped to 92.0%.

3. Superset Probes Hurt Ensembles

Issue: Adding a 6th probe trained on (all generators + SD3) made the ensemble worse (97.18% vs 97.39%).

Root Cause: The superset probe was highly correlated with existing members — it didn't add diversity, just redundant information.

Lesson: Ensembles need diverse members, not just more members.

4. Feature Drift Doesn't Transfer

Issue: We hypothesized that AI images show larger embedding drift under perturbations. We added drift features [drift_mean, drift_std] to CLIP features.

Result: The drift probe (0.8920 clean, 0.6760 worst) was worse than our baseline and got worse under noise (AI accuracy dropped from 0.79 → 0.35 as noise increased).

Lesson: Drift is a real signal, but CLIP's drift isn't stable enough to use as a feature.

5. Noise Augmentation Skews Class Balance

Issue: We tried retraining all 8 probes with noise-augmented features (each image + 4 noise levels).

Result: The "Noisy8" ensemble was worse across all transforms (96.90% clean vs 98.60% baseline).

Root Cause: Adding 68k AI noise features skewed the class balance heavily toward AI, shifting the decision boundary. CLIP features don't capture added noise in a generalizable way.

Lesson: More data isn't always better — you need balanced, diverse data.

6. Ensemble Saturation

Issue: Adding a 9th probe (pure-Flux) made the ensemble worse (98.04% vs 98.60%).

Root Cause: One more AI-focused probe tipped the balance toward over-flagging noise as AI.

Lesson: There's an optimal ensemble size. For our setup, 8 members is optimal.

7. scikit-learn Version Compatibility

Issue: Our probes were trained with scikit-learn 1.9.0 (requires Python 3.11+), but the original venv was Python 3.10.

Error: 'LogisticRegression' object has no attribute 'multi_class'

Solution: Created a new venv with Python 3.14 + scikit-learn 1.9.0.

Accomplishments that we're proud of

  1. 98.14% mean transform accuracy : maintaining high accuracy across 15 real-world degradations
  2. Discovered the pure single-generator probe technique : a novel ensemble diversity strategy
  3. Solved the calibration problem : using class weighting to fix probability shifts without retraining the backbone
  4. Built a working Telegram bot : real-time AI image detection via messaging app
  5. Stayed well under the 2B parameter limit : only 304M params (15.2% of budget)

What we learned

  1. Ensembles need diversity, not just quantity : correlated members hurt performance
  2. Augmentation is critical for robustness : training on transformed images dramatically improves real-world performance
  3. Calibration ≠ Discrimination : a model can rank correctly (high AUROC) but still fail at a fixed threshold
  4. More data isn't always better : imbalanced or redundant data can hurt more than help
  5. Version compatibility matters : always pin your dependencies and document your environment

What's next for spoTTing The difference

Short-term improvements

  1. Higher-resolution training: Extract features at native resolution for better performance on high-res images
  2. Frequency-domain features: Add FFT-based features to capture subtle AI artifacts
  3. More generators: Add pure probes for Flux, Ideogram, Krea, and other emerging generators

Long-term vision

  1. Video detection: Extend to video by analyzing frame consistency and temporal patterns
  2. Online learning: Adapt to new generators without full retraining
  3. Explainability: Add attention maps or feature importance to show why an image is flagged
  4. Mobile deployment: Quantize the model for on-device inference

Known limitations

  • Resolution: Optimized for lower-resolution images due to time and storage constraints
  • Noise robustness: Noise transforms (σ0.05/0.10) remain our weakest point at 97.10%
  • Generator coverage: Newer generators may not be as well-detected
  • Single-frame only: Doesn't leverage temporal consistency for video

Repository

GitHub Repository

Quick Start:

pip install -r requirements.txt
python scan_image.py path/to/image.jpg

How to use Telegram bot

Click on the telegram link below 👇 /Start and just send in your image to detect if it's AI!

Built With

+ 8 more
Share this project:

Updates