Deepfake Forensics & XAI Engine

An enterprise-grade, multi-modal platform using 15 biological, physical, and acoustic pipelines to detect deepfakes with Explainable AI and generate court-ready PDF reports.

Comment

Inspiration

As generative AI models (GANs, Diffusion Models, and sophisticated pipelines like Wav2Lip and Roop) approach total photorealism, human visual inspection is no longer a mathematically reliable metric for media authenticity. The potential for deepfakes to cause massive political, financial, and legal harm is terrifying. We realized that combating AI requires an even stronger AI—one that doesn't just say "Real" or "Fake", but actually proves it through rigorous scientific and mathematical evidence that can hold up in a court of law.

What it does

The Deepfake Forensics & XAI Engine operates as a state-of-the-art digital forensics laboratory. Rather than relying on a single monolithic "black-box" classifier, the system implements a 15-Dimensional Multi-Modal Ensemble Architecture.

It dissects media across biological, physical, frequency, and spectral dimensions in real-time. By tracking mathematically impossible generative artifacts—such as unnaturally smooth high-frequency spectral rolloffs, missing facial heartbeats (rPPG), corrupted Corneal Specular Highlights, and Audio-Visual desync—it easily flags out-of-distribution adversarial deepfakes. It then utilizes Explainable AI (XAI) to generate comprehensive, court-ready PDF reports that justify its verdicts with interpretable visual heatmaps.

How we built it

The platform is built on a highly concurrent microservice architecture:

  • The Interface: A responsive, glassmorphic React/Vite frontend utilizing Recharts to visualize the live telemetry of the 15 forensic dimensions.
  • The Brain: A FastAPI backend that bypasses the Python GIL by utilizing a ThreadPoolExecutor to run heavy OpenCV frame extractions and multi-model inferences in parallel.
  • The Models: We trained a custom EfficientNet-B4 spatial backbone using Contrastive Learning. By employing a Triplet Loss function: $$ L(a, p, n) = \max \left( 0, | a - p |^2_2 - | a - n |^2_2 + \text{margin} \right) $$ The network was forced to map authentic faces and GAN-generated faces into widely separated clusters in the latent embedding space.
  • Audio & Sync: We implemented a 2D-CNN for Acoustic Anti-Spoofing (detecting TTS engines like ElevenLabs) and the official 3D-CNN SyncNet architecture to compute the absolute Lip Sync Error Distance (LSE-D).
  • The Aggregator: A 3-layer Meta-Classifier MLP was trained to act as the ultimate judge, weighing the 15 diverse outputs and producing the final probability score.

Challenges we ran into

  1. The "Squish" Problem: Initially, our neural network blindly predicted "Authentic" on obvious deepfakes. We realized that squishing a 1080p full frame into \( 380 \times 380 \) tensor dimensions destroyed the microscopic, high-frequency GAN artifacts the model relied on. We solved this by implementing a dynamic HAAR cascade face cropper that isolates and expands the bounding box by 20% before inference, preserving critical pixel data.
  2. Computational Bottlenecks: Running 15 heavy mathematical operations (Dense Optical Flow, 2D-DCT Transforms, Face Meshing) simultaneously caused severe VRAM/CPU memory overflows. We solved this by strictly capping video extraction to 16 even intervals over 60 seconds and executing them via a concurrent Thread Pool.
  3. Black Box Skepticism: We knew people wouldn't trust an AI accusing them of a deepfake without proof. Implementing PyTorch Grad-CAM to reverse-engineer the spatial attention gradients allowed us to paint heatmaps over the exact pixels (e.g., blending boundaries) that triggered the alarm.

Accomplishments that we're proud of

  • Achieving a peak Validation Accuracy of 99.37% (ROC-AUC 0.998) on our spatial image backbone against a heavily imbalanced dataset of 53,000+ extracted frames.
  • Successfully fusing vastly different data structures—biological face meshes, acoustic MFCC spectrograms, and JPEG Error Levels—into a single, cohesive Meta-Classifier.
  • Providing a "wow-factor" UI that makes incredibly dense forensic data readable for non-technical analysts.

What we learned

We learned that deepfake detection cannot rely on a single modality. Visual-only models are easily fooled by low-resolution video, and audio-only models are fooled by authentic video with cloned audio dubbed over it. True robustness requires an ensemble approach. We also gained immense experience in managing PyTorch memory footprints, handling cross-origin API data streams in React, and mapping multi-dimensional data via Recharts.

What's next for Deepfake Forensics & XAI Engine

  • Real-time Livestream Interception: Transitioning our batch-processing backend into a WebRTC pipeline to detect deepfakes live during Zoom or Google Meet calls.
  • Micro-Expression Analysis: Integrating Action Unit (AU) tracking to determine if facial muscle movements are psychologically genuine or artificially puppeteered by a 2D mask.
  • Cloud Scaling: Containerizing the inference nodes with Kubernetes to dynamically scale GPU resources based on video upload volume.

Built With

Share this project:

Updates