ClippedAI — Autonomous AI Content Engine

Autonomous AI Content Engine that transforms long-form horizontal videos into viral, platform-ready 9:16 vertical clips with active speaker tracking, word-synced kinetic subtitles, hook thumbnails, and multi-platform SEO distribution packages.

🌐 Live Interactive Showcase & Playable Clips: https://ets2k7.github.io/clipped-ai/
💻 GitHub Repository: https://github.com/ETS2K7/clipped-ai

💡 Inspiration

Short-form vertical video (YouTube Shorts, TikTok, Instagram Reels) is the single highest-leverage distribution channel for modern creators. However, repurposing long-form podcasts is a grueling 4-hour manual nightmare:

• Scrubbing transcripts for 30-second conversational hooks.
• Manual keyframing to keep subjects centered.
• Handling conversational banter where single-camera crops cut out listener reactions.
• Nauseating camera drift caused by naive face-detection bounding boxes that constantly jitter.

I built ClippedAI to automate the modern creator's entire distribution workflow—ingesting a single YouTube URL or local video and delivering broadcast-ready, zero-jitter vertical clips and multi-platform SEO packages in under 3 minutes.

⚙️ What It Does

1. Word-Level Acoustic Ingestion: Extracts lightweight audio (48kbps mono Opus/AAC) and runs AssemblyAI Universal-2 speech diarization with millisecond word timestamps.
2. Psychological Hook Mining: A 4-tier LLM cascade evaluates transcripts against high-retention psychological triggers (Curiosity Gap, Contrarian, Bold Claim, Relatable Struggle) scored out of 100 with breath-pause boundary snapping.
3. Vision Intelligence & Active Speaker Detection: Employs Fast-ASD (TalkNet + S3FD) lip-sync cross-attention to isolate speaking faces in real time, ignoring silent bystanders.
4. Stationary Cluster Locking (0.0px Jitter): Solves camera drift. When subject motion is below threshold ($\sigma < 0.06$), spatial coordinates freeze to cluster medians.
5. Adaptive Split-Screen Layouts: In multi-speaker banter, automatically stacks speakers vertically (1080×960 each) so viewer chemistry is never lost.
6. Layout-Aware Kinetic Typography: Hormozi neon subtitles snap dynamically to the center dividing seam (\an5\pos(540,960)) in split mode so they never cover a speaker's face.
7. Complete Creator Distribution Pack: Outputs 1080×1920 hook thumbnails and ready-to-publish SEO metadata (titles, descriptions, hashtags, sound recommendations) tailored for Shorts, TikTok, and Reels.

🔬 How I Built It & Technical Architecture

• Core Orchestration: Clean, deterministic Python 3.11 architecture with FastAPI REST backend and rich CLI.
• Computer Vision: PyTorch, Fast-ASD, TalkNet lip-sync cross-attention, and PySceneDetect for instant transition snap-cuts.
• Mathematical Smoothing: SciPy spatial clustering and exponential moving average (EMA) velocity dampening.
• Virality Cascade: Resilient 4-tier fallback: Gemini 2.5 Flash ➔ Groq LLaMA 3.3 70B ➔ OpenRouter ➔ Heuristic scoring with cache reuse.
• Rendering Engine: Accelerated FFmpeg and Libass with 2D AR-safe headroom normalization ($cy = 0.42$).

🏆 Solved Engineering Challenges

📊 Creator Impact

• Saves 4 Hours Per Episode: Converts hours of timeline editing into a 2.4-minute single-command run.
• Production-Ready Artifacts: Delivers clean root .mp4 video files alongside an isolated assets/ directory containing hook thumbnails and SEO distribution copy.
• Zero Cloud Lock-In: Runs entirely locally on consumer hardware without subscription lock-in.

🔮 What's Next

• Direct multi-platform scheduled publishing via TikTok and YouTube APIs.
• Generative context-aware B-roll insertion using open-source diffusion models.
• Localized multi-language voice dubbing with lip-sync translation.

Built With

• assemblyai
• fast-asd
• fastapi
• ffmpeg
• gemini-2.5-flash
• groq
• libass
• llama-3
• numpy
• opencv
• pillow
• python
• pytorch
• scikit-learn
• scipy
• talknet
• yt-dlp