ClipCup

Repurposing long landscape videos into vertical shorts for TikTok, Reels, and YouTube Shorts currently demands hours of manual trimming and keyframe tracking.

Comment

Inspiration

Short-form vertical video (TikTok, Instagram Reels, YouTube Shorts) dominates social media, but creators spend hours manually scanning long podcast interviews or keynote talks, finding timestamps, and manually keyframing horizontal video into a 9:16 aspect ratio. Most existing solutions either lock users into expensive subscription tiers or output static center crops that cut off moving speakers. We built Clip Cup to make intelligent, speaker-centered vertical video generation accessible, automated, and cinematic with zero manual keyframing.

What It Does

Clip Cup is an end-to-end AI and computer vision pipeline:

  1. Ingestion & Transcription: Ingests direct video URLs or uploads (up to 10 minutes), extracting high-fidelity audio transcribed with word-level timestamps using AssemblyAI.
  2. AI Highlight Selection: AWS Bedrock with Claude 3.5 Sonnet analyzes the transcript semantics to detect the most captivating, punchy highlight (15 to 90 seconds) snapped to sentence boundaries.
  3. Computer Vision Reframing: Utilizes OpenCV YuNet deep-learning face detection to track the primary speaker across frames.
  4. Cinematic Motion Smoothing: Employs Exponential Moving Average (EMA) smoothing and pan velocity clamping to eliminate camera jitter: $$\hat{x}t = \alpha \cdot x_t + (1 - \alpha) \cdot \hat{x}{t-1}, \quad \alpha = 0.15$$ Combined with HSV histogram correlation to detect scene cuts and prevent jarring pans across transitions: $$\text{Correlation} = \text{compareHist}(H_{t-1}, H_t, \text{HISTCMP_CORREL})$$
  5. Rendering & Delivery: Streams a crisp 1080×1920 H.264 video with synchronized AAC audio via an FFmpeg pipe to AWS S3, providing users with a 24-hour presigned playback and download link.

How We Built It

  • Backend & Worker: Python 3.12, Django 5, and Django REST Framework with a thread pool background worker managing the 5-stage pipeline (PENDING → TRANSCRIBING → SELECTING → REFRAMING → RENDERING → DONE).
  • Database & Auth: Supabase PostgreSQL with JWT authentication (djangorestframework-simplejwt), supporting user accounts and job history.
  • AI & Cloud Services: AWS Bedrock (Claude 3.5 Sonnet), AssemblyAI Speech-to-Text SDK, and AWS S3 object storage.
  • Computer Vision: OpenCV 4 (cv2.FaceDetectorYN with YuNet ONNX) running headless inside Docker.
  • Frontend: React (Vite), Tailwind CSS, Lucide icons, responsive 9:16 video player with live polling progress bars and metrics display.

Challenges We Faced

  • Camera Shake vs. Fast Motion: Raw face detections from frame to frame created dizzying jitter when zoomed into a 9:16 crop. We solved this by implementing an EMA filter with an 18 px/frame maximum displacement clamp and automatic center resets on scene cuts.
  • Subprocess Memory & Video Streaming: Encoding full 1080×1920 frames in memory risked OOM crashes. We engineered a direct rawvideo pipe from OpenCV straight into the FFmpeg CLI subprocess without saving uncompressed intermediary frames.
  • Fail-Safe Offline Mode: Ensuring the pipeline worked even without cloud keys required designing graceful local fallbacks (density-based audio windowing, local disk storage, and mock speech transcription).

What We Learned

  • How to apply classic signal smoothing (EMA) to modernize deep-learning spatial detections.
  • Designing state machines that cleanly recover and clean up disk artifacts in finally blocks to guarantee zero persistent source video storage.
  • The power of disciplined specification-driven development using the Devpost Learn skill workflow.

Built With

Share this project:

Updates

Submission history