Creative Director AI

Creative Director AI turns raw audio into synced, finished music videos using Gemini to autonomously plan scenes, generate video cuts, and inject C2PA provenance.

Comment

Inspiration

Independent music artists and content creators face immense bottlenecks producing visual content. Turning an audio track into a finished music video traditionally requires days of manual stem analysis, storyboarding, shot styling, and multi-track editing. While emerging generative models like Google Veo and Imagen 3 can produce stunning video clips, chaining them together into a rhythmic, beat-synced narrative requires constant human micromanagement.

We were inspired by Track 1 (The Taskmaster) to solve the "blank canvas problem" for creators. We set out to create an autonomous, zero-touch director that listens directly to raw music waveforms, reasons through lyrical and dynamic shifts, and coordinates production end-to-end.

What it does

Creative Director AI is an event-driven AI video director built on the Google Agent Development Kit (ADK) that turns raw audio tracks into finished, beat-synced music videos.

Multimodal Audio Decomposition: Ingests raw audio directly via Gemini to detect BPM, dynamic drops, verse/chorus boundaries, and emotional tone without needing pre-transcribed text.

Cinematic Storyboarding: Maps the emotional arc to timestamped visual sequences with detailed shot directions, camera motions, and color palettes.

Autonomous Scene Generation: Automatically dispatches structured tool calls to generate video cuts using Google Veo and Imagen 3.

Beat-Matched Assembly & Provenance: Stitches rendered cuts to the musical timeline and seals the master file with verifiable C2PA provenance metadata to establish content authenticity.

How we built it

Agent Orchestration: Scaffolded with @google/adk in TypeScript (NodeNext/strict ESM).

Model Engine: Powered by gemini-3.5-flash for low-latency multimodal audio analysis and agentic tool routing.

Custom Tool Bindings: Implemented strict Zod schemas for generate_video_cut, assemble_and_sync_timeline, and sign_c2pa_manifest.

Cloud Deployment: Hosted on Vertex AI Reasoning Engine (projects/1037240986506/locations/us-central1/reasoningEngines/5922098819817799680) for managed, serverless execution and Server-Sent Event (SSE) streaming.

Challenges we ran into

  • Tool Execution Boundaries: Ensuring the model strictly delegated tasks via structured tool schemas rather than hallucinating render URLs directly in conversational text.

  • Audio-Visual Pacing: Calibrating prompt timestamps so scene transitions strictly matched musical bar lines and dynamic build-ups.

  • Schema Validation: Enforcing strict runtime parameter typing between the ADK TypeScript layer, Zod validation, and the Vertex AI Reasoning Engine endpoint.

Accomplishments that we're proud of

  • Zero-Touch Execution: Successfully taking a raw audio input and autonomously generating a fully storyboarded, multi-scene production log with verified tool invocations.

  • Clean TypeScript ADK Architecture: Achieving a typecheck-clean (tsc --noEmit exit code 0) modular agent structure utilizing native sub-agent delegation.

  • Live Vertex AI Cloud Deployment: Successfully deploying the agent into production on Vertex AI Reasoning Engines with active SSE query streaming.

What we learned

  • Structured Schemas Over Chat: Using strict function schemas is essential for getting predictable generation pipelines out of generative models.

  • Multimodal Direct Ingestion: Passing raw audio waveforms directly into Gemini preserves tempo, rhythm, and mood nuance that get lost in text transcriptions.

  • Separation of Concerns: Splitting creative directing (Google ADK) from on-chain identity and broadcast hosting yields a modular, scalable creator architecture.

What's next for Creative Director AI

  • Multi-Agent Production Crews: Introducing a hierarchical loop system with an automated Visual Critic / Judge Agent to evaluate cut continuity before final rendering.

  • Direct Integration with Creative Pixels: Embedding the Reasoning Engine SSE stream directly into the live studio workstation canvas for real-time video rendering.

  • Ecosystem Bridge: Connecting finished C2PA video masters to our on-chain Pinata Digital Twins (ERC-8004) for automated broadcast hosting and social token reward distribution.

Built With

Share this project:

Updates