Inspiration
Recently, in time there have been a lot of the same account archetypes going around on Instagram and tiktok as well as the same content being circulated on these platforms. I've seen social media accounts later to be surprised that they aren't real people at all, and digital personas are inhabiting social media posing to be real people, taking away from real-world connections, and spreading fear about what we can really trust. Additioanlly there are tutorials on TikTok and Instagram, and probably other social media, where people use new image and video generation tools as well as general LLMS to create automated accounts tha posts fake digital accounts that pose as real to engage and take advantage of people. This is a huge abuse of computing technology and an unethical abuse of human trust. Deepfake detection technology must be iterated on if we want to live with new and emerging technology without it consuming our trust and perception.
What it does
The model detects AI-generated images and videos by building on DINOv2, a vision transformer pretrained on large amounts of natural images. The backbone stays frozen, and we train only lightweight LoRA adapters on the last three transformer blocks, plus a small variational information bottleneck head that outputs a fake-vs-real score. Features are pooled from several depths (layers 4, 8, 10, and 12), combining the CLS token with mean-pooled patch tokens, so the detector sees both global and local evidence. For video, a temporal adapter aggregates per-frame features, and top-k logit pooling produces a clip-level score. An auxiliary head predicts beauty-filter use, which keeps the detector from learning that smoothing artifacts equal "fake." Training uses a loss that combines classification, a KL regularizer, and the auxiliary term. At inference scores pass through split-conformal calibration, so each decision comes with a controlled false-positive rate, and the system abstains rather than guessing when calibration is missing. I evaluate with leave-one-generator-out splits, so every reported number reflects performance on a generator the model never saw during training.
How we built it
The detector was built in Python 3.12 on PyTorch, using DINOv2-base vision transformer weights from Hugging Face as a frozen feature extractor. Only lightweight LoRA adapters (rank 8, on the query and value projections of the last three blocks) and a small variational information bottleneck head are trained, which keeps training cheap enough to run on a single cloud GPU. Features are taken from layers 4, 8, 10, and 12, combining the CLS token with mean-pooled patch tokens. The head produces a fake-vs-real logit, and an auxiliary head predicts beauty-filter use so the model doesn't treat smoothing artifacts as evidence of generation. Video inputs pass through a temporal adapter over per-frame features, with top-k logit pooling for the clip-level score. Scores are passed through split-conformal calibration, which sets a threshold with a controlled false-positive rate and makes the system abstain when no calibration exists. The data pipeline includes a platform transcoding emulator that re-encodes training images the way social-media platforms do, so the model learns from the compression it will meet in the wild. Experiments run from YAML configs, with pytest for unit tests, a synthetic-data smoke run for end-to-end checks, and training jobs dispatched to Modal L4 GPUs. For the vision-language route, we use Tinker to LoRA-fine-tune Qwen-based VLMs, and their logits are fused with the core detector's through logistic regression. Evaluation uses nested leave-one-generator-out splits, with checkpoint selection on one set of generators and a final score on held-out ones, and bootstrap confidence intervals because the real-image sample is small.
Challenges we ran into
Acquiring data was one of the biggest obstacles. Many of the most useful deepfake datasets and backbones sit behind Hugging Face authentication gates: DINOv3 access was granted after a request, and AV-Deepfake1M was still pending when last checked, so I couldn't rely on it. The datasets we could use were large, and downloading and storing full collections would have exceeded the local disk and time budget. I therefore selected subsets. Tiny-GenImage shards are interleaved across real and fake images with rotating generators, so we (my agentic team and I) chose a subset of shards that stayed balanced across both classes (configs/tiny_genimage_local.yaml). Some generators, such as SD14, were absent from the splits I could access, which limited the coverage of the generator landscape. Licensing added another constraint: Tiny-GenImage is CC-BY-NC-SA, so the work is restricted to research use. Compute was similarly tight. We ran GPU training on Modal using L4 instances at roughly $1.43 per hour, capped each job at two hours, and limited concurrency to seven jobs, which meant every leave-one-generator-out rotation had to fit that window. Our VLM route also depended on Tinker credits and API access. Within a hackathon timeline, these constraints pushed me to favor smaller, cheaper experiments, a frozen backbone with LoRA rather than full fine-tuning, and smoke tests on synthetic data before spending GPU hours on real runs.
Accomplishments that we're proud of
I built a hosted deepfake detector that runs on Modal and exposes a developer API and a web app. The web app includes a playground, a social-style feed with per-post AI flags, and a vertical video feed. The detector is a router over three specialists: a general AI-image detector, an AI-generated-face detector, and a face-swap video model. Content is flagged as fake if any specialist says so, treated as real only if all of them agree, and otherwise sent to human review. Each specialist's verdicts come from split-conformal thresholds fit on held-out data, and every response includes per-specialist scores, reasons, model versions, and a research-demo disclaimer. On generators the model never saw in training, the image model averages AUC 0.915 across seven leave-one-generator-out rotations (with confidence intervals), with Midjourney the hardest at 0.741. The face model catches 86–93% of Flux faces, with a 0.58% false-alarm rate on the FairFace real faces (single run). The face-swap video model reaches AUC 0.867 on WildDeepfake's untouched official test split with 0% of real clips called fake, but it sends 76% of clips to review and does not detect talking-head animation (AUC 0.385, below chance; single run). Warm latency is about 70 ms server-side for images and about 0.6 s for video, and the endpoint scales to zero when idle. Along the way, I hit several failed runs: a video model that flagged 53.5% of real in-the-wild clips, a training run that collapsed to a constant output until I lowered the learning rate and warm-started it, and a train/test leakage bug that code review caught before any training ran.
Caveats: this is a research demo, not an enforcement tool, and several datasets are research-only. Still-image face swaps remain unsolved, most face and video results are single runs, and Video V3 is still training and will replace V2b only if it passes its gate.
What we learned
I learned that modern deepfake detection is as much a data and compute-management problem as a modeling one. I used Modal's serverless L4 GPUs for training and serving, which kept each run at roughly $0.04 to $0.55 and let me scale to zero when idle, and I used Tinker's hosted LoRA fine-tuning for the vision-language route, which cost about $3.2 of a $150 credit across all experiments. Hugging Face supplied the datasets and the DINOv2 and DINOv3 backbones, though many of the most useful datasets sat behind access gates. I used Graphify to build a knowledge graph of the codebase so I could trace how the detector, losses, calibration, and pipeline connected without re-reading every file, and I used Antigravity as a research agent to pull benchmark numbers and literature into written notes. I also used Firecrawl to scrape research papers in bulk as an MCP for AGY. The bigger lesson was that the first numbers I got were often wrong: a held-out generator leaked into checkpoint selection, a VLM learned image size instead of content, and a fusion gain disappeared on the untouched test set. Each of those was caught only because I kept the test split out of every decision and reran the comparison, so I now treat any single result as a hypothesis until a held-out protocol confirms it.
What's next for Ventura
I want to make the model better by training on larger and more varied data, such as full GenImage and modern text-to-video and talking-head sets, and by giving it more compute and fine-tuning time. Runs are cheap enough on Modal that the constraint is data access and licensing, not GPU cost. The next technical direction is to close the gaps that the results expose. Midjourney remains the weakest unseen generator, and the image and video models still send most clips to review, so the priorities are a pre-registered generator-gap experiment, a talking-head video class, and generator-diverse calibration data. On fairness, I want to add the group-blind threshold and a false-negative audit across demographic groups, since the current fairness numbers are single runs. I also want to try a PAC-style conformal variant for a stronger guarantee on false positives, and a distilled, smaller model for edge or low-cost serving. Beyond the model, the path to real use is social media integration, where the detector is a plug-in that platforms and independent developers can call through the API, route uncertain content into human review, and attach provenance signals such as C2PA credentials. I would only offer it as a tool that informs human moderators, not one that removes content on its own, because the evidence does not support automatic enforcement and any platform would need its own calibration on its own compression pipeline. Done well, the project becomes a shared, open foundation that developers can extend with their own specialists and datasets, helping make digital media spaces safer while leaving the final judgment with people.
Log in or sign up for Devpost to join the conversation.