Vidgen

An autonomous multi-agent content creation and posting engine built including video generator, researcher, video editor, script writer and Auto-Post Agent

Comment

Inspiration

VIDGEN: Autonomous AI Video Creation & Multi-Platform Publishing Ecosystem

Inspiration

In today's digital economy, video is the undisputed king of content. However, producing high-quality, engaging video content consistently demands immense effort: researching trending topics, drafting compelling scripts, generating visual assets, synthesizing natural narration, editing clips, and managing multi-platform distribution. Independent creators, small businesses, and educators often spend 10 to 20 hours per video—or thousands of dollars hiring production teams.

We were inspired by a vision: What if an AI system could act as your full-stack Video Production Studio and Social Media Manager?

We wanted to build an autonomous platform where a user can simply chat with an AI "Director" about an idea or topic, and watch the system handle research, scriptwriting, image generation, neural voiceover, video animation (via Google Veo), dynamic editing, and instant publishing across YouTube, LinkedIn, X (Twitter), and Facebook.

What It Does

VIDGEN (alongside its companion video-editing agent engine VidCut) is an end-to-end autonomous video creation and monetization ecosystem powered by Google's Gemini multimodal models and serverless infrastructure.

Key features include:

  1. Interactive AI Director Chatbot: A conversational agent powered by Gemini that acts as a creative director. It brainstorms topics, adapts tone (e.g., tech news, entertainment, documentary, educational), refines scripts interactively, and requests approval before production.
  2. Automated Topic & Trend Research: Automatically ingests breaking news and trending topics via Brave Search, Perplexity Sonar, and RSS feeds, validating content freshness and relevance.
  3. Multimodal Media Generation:
    • Scripting & Storyboarding: Generates timed scene-by-scene storyboards.
    • Visual Asset Creation: Generates high-fidelity visual scenes using gemini-2.5-flash-image (with HuggingFace/Flux fallback).
    • Google Veo Video Animation: Converts static generated images into fluid 1080p video animations using Google's veo-3.0-generate-preview.
    • Neural TTS Voiceover: Synthesizes studio-grade audio narration using gemini-2.5-flash-tts (Kore voice) and ElevenLabs.
  4. Automated Video Compositing & Assembly: Dynamically syncs video clips, applies subtitles, performs sidechain audio ducking (lowering background music during speech), and renders final MP4 files via FFmpeg and MoviePy.
  5. Multi-Platform Publishing & Monetization:
    • Direct OAuth upload to YouTube with automatically generated custom thumbnails and metadata.
    • Cross-platform publishing to LinkedIn, Facebook, and X (Twitter) via Make.com webhooks.
    • Integrated SaaS billing infrastructure powered by Polar (tiered subscriptions and lifetime licenses).

How We Built It

We engineered a modular, serverless, and multi-agent pipeline:

  • Backend & Serverless Execution: Developed with Python FastAPI and deployed on Modal serverless containers for infinite scalability and zero idle costs.
  • AI Core & Google Ecosystem:
    • Gemini 2.5 Flash: Orchestrates script writing, storyboarding, and prompt expansion.
    • Gemini Flash Image: Generates vivid, context-aware imagery for every video scene.
    • Gemini 2.5 Flash TTS: Synthesizes lifelike natural audio narration.
    • Google Veo: Animates scene images into cinematic video clips.
  • Database & Storage: Supabase (PostgreSQL) handles multi-tenant user authentication, settings, and artifact storage.
  • Video & Audio Processing: MoviePy, FFmpeg, and pyttsx3 handle audio mixing, sidechain compression, caption overlaying, and final video rendering.
  • Frontend Interface: Clean responsive Web UI built with modern HTML5/CSS3 (glassmorphism UI) hosted on cPanel, accompanied by a native Android WebView wrapper.

Mathematical & Algorithmic Foundations

To ensure studio-quality audio-visual alignment, VIDGEN uses mathematical formulations for scene timing and audio dynamics:

1. Dynamic Audio Sidechain Ducking

To prevent background music from drowning out the AI narration, the music volume $A_{\text{music}}(t)$ is dynamically attenuated based on the absolute amplitude of the synthesized voiceover envelope $|S_{\text{voice}}(t)|$:

$$A_{\text{music}}(t) = A_0 \cdot \left(1 - \alpha \cdot \frac{|S_{\text{voice}}(t)|}{\max(|S_{\text{voice}}|) + \epsilon}\right)$$

Where:

  • $A_0$ is the baseline music gain.
  • $\alpha \in [0.6, 0.85]$ is the sidechain attenuation factor.
  • $\epsilon = 10^{-6}$ is a smoothing factor to prevent division by zero.

2. Narrative Scene Alignment Optimization

To eliminate visual drift and ensure every generated image or Veo clip seamlessly matches the voice narration timestamp $d_{\text{voice}, i}$ for scene $i$, the total video duration $T_{\text{total}}$ is optimized by minimizing narrative boundary error:

$$\min \sum_{i=1}^{N} \left| t_{\text{visual}, i} - \sum_{j=1}^{i} d_{\text{voice}, j} \right|$$

Subject to:

$$t_{\text{visual}, i} \ge d_{\text{voice}, i}, \quad \forall i \in {1, 2, \dots, N}$$

Challenges We Ran Into

  1. Managing Serverless Container Lifecycles: Moving long-running video rendering pipelines to Modal serverless containers required solving container termination issues. Daemon threads in Python background tasks were being terminated early when HTTP responses completed; we solved this by implementing non-daemon persistent thread managers and webhook status polling.
  2. Multimodal Model Latency & Synchronization: Syncing AI-generated audio duration with Google Veo video clip generation times was challenging due to varying TTS audio lengths. We engineered a dynamic storyboard calculation pass that measures precise audio duration before firing Veo animation jobs.
  3. Cross-Domain Webhook Media Delivery: Distributing social media posts with local generated images failed when webhooks couldn't access relative URLs. We built a public Modal artifact router (/artifacts/{run_id}/...) to serve generated media securely to external APIs like Make.com and YouTube.

Accomplishments That We're Proud Of

  • Fully Autonomous Pipeline: Created a system where a single sentence input can turn into a fully produced, voice-overdubbed, animated, subbed, and published video in under 3 minutes.
  • Seamless Google AI Integration: Successfully combined Gemini 2.5 Flash (text/scripting), Gemini Flash Image (visuals), Gemini TTS (speech), and Google Veo (video animation) into a unified workflow.
  • Production-Ready SaaS Monetization: Built user-tiered auth, OAuth channel connections, and Polar payment processing, making it ready for real-world business deployment.
  • Local Fallback Systems: Engine automatically falls back to local TTS (pyttsx3 / whisper.cpp) and static image compositing if API quotas or keys are constrained.

What We Learned

  • The Power of Agentic Co-Creation: Users prefer an interactive "Director" AI agent that collaborates and lets them edit scripts rather than a black-box "one-click" generator.
  • Multimodal Synergy with Google AI: Combining Gemini's multimodal reasoning with Veo video generation unlocks cinematic output quality that was previously impossible without dedicated video editors.
  • Robust Error Handling in Media Pipelines: Heavy media processing requires defensive programming—validating audio stream parameters, checking video bounds, and verifying output files with automated QA checks before publishing.

What's Next for VIDGEN

  • Multilingual Auto-Dubbing & Localization: Leveraging Gemini to instantly translate scripts and synthesize voiceovers in 30+ languages for global content distribution.
  • Real-Time Analytics & Feedback Loop: Integrating YouTube Analytics API so the Director AI can analyze video performance (CTR, retention rate) and automatically optimize future video scripts and thumbnails.
  • Interactive Timeline Editing in UI: Bringing direct timeline drag-and-drop editing (powered by our VidCut engine) into the main web application for creator fine-tuning.

(For local file i named it AutoPostMonetized)

Built With

Share this project:

Updates