CinemaDirector-AI: Agentic Cinema Director Studio

An agentic AI film director powered by Gemini, ClickHouse, and Parallel Web Systems that turns raw footage and script pitches into multi-perspective, production-ready cinematic master cuts.

Comment

Inspiration

In the film and anime production industry, the creative loop between a director's vision and the final rendered cut is fraught with friction. When an initial animation cut lacks emotional punch or cinematic depth, diagnosing why—whether it is a flat 50mm lens angle, stagnant camera motion, or dull lighting—requires multiple review meetings and laborious redrawing.

We asked: What if a film director had an omniscient, agentic AI co-director that could watch raw footage, diagnose its cinematographic bottlenecks in seconds, propose three contrasting directorial treatments, and conversationally re-render high-definition cuts on the fly? That is the genesis of CinemaDirector-AI.

What it does

inemaDirector-AI transforms filmmaking into a fast, intuitive, conversational collaboration:

  1. Agentic Video Understanding (Gemini 3.8 Flash): Scans raw video or script concepts, detects focal lengths, lighting setups, and camera motion, and generates an exhaustive Cinematographic Diagnostic Card with bottleneck diagnosis and optimal climax window markers (-ss and duration).
  2. 3-Perspective Proposal Deck: Simultaneously synthesizes three contrasting cinematic visions:
    • Pattern A (Dynamic Kinetic): Low-angle, handheld tracking.
    • Pattern B (Atmospheric / Backlit): Ultra-wide, golden hour backlight, cinematic mood.
    • Pattern C (Intimate Drama): Telephoto macro, shallow depth of field.
  3. Direct AI Video Generation (Gemini Omni 1.1 Flash): Calls Google's gemini-omni-1.1-flash Interactions Video API directly using keyframe references to generate 720p 24fps anime cuts in approximately 35 seconds—no fake overlays, true generative pixel animation.
  4. Dual-Cut Comparison Viewer & Filmstrip Timeline: Allows directors to review the original benchmark cut side-by-side with proposed treatments in synchronized playback.
  5. Conversational Studio (Voice of the Director): Directors refine the scene by speaking naturally (e.g., "Make the sunset a deeper crimson and add anamorphic lens flare"). Gemini updates the shot parameters and renders a versioned iteration (v1.0 -> v1.1 -> v1.2).
  6. Master Sign-off & Vector Memory: Once satisfied, the director approves the cut, committing it to ClickHouse Cloud for long-term similarity matching and master timeline delivery.
  7. 1-Click Bilingual Experience: Full instant toggling between English and Japanese for international juries and global creative teams.

How we built it

We architected a pure Google Cloud and official partner stack complying 100% with the hackathon rules:

  • Google Gemini 3.8 Flash: Employed for sub-second multimodal video scanning and structured cinematography reasoning.
  • Google Gemini Omni 1.1 Flash: Invoked directly via the Interactions API (client.interactions.create) for multimodal video generation.
  • ClickHouse Cloud (Official Partner): Serves as the real-time vector database storing high-dimensional embeddings of scenes, director choices, and query logs with an interactive UI SQL inspector.
  • Parallel Web Systems (Official Partner): Provides live web grounding to fetch real-world film lighting setups and camera movement case studies.
  • Google Cloud Storage (GCS): Handles durable storage of generated MP4 assets.
  • Streamlit: Delivers a dark-themed, theater-grade studio UI.

Challenges we ran into

Preserving Character Consistency in Video Generation: Ensuring the newly generated video cuts retained the identity and facial structure of the original characters. We solved this by extracting keyframes at the Gemini-diagnosed climax timestamp and feeding them as multimodal visual prompts into Gemini Omni 1.1 Flash.

  • Balancing Token Consumption with Latency: Rather than feeding dense video frames, Gemini 3.8 Flash's agentic scanner strategically samples key moments, achieving an 88% token efficiency gain with ~2-second analysis latency.

Accomplishments that we're proud of

True Direct Video Synthesis: Moving beyond static filters to actual gemini-omni-1.1-flash direct video generation that produces gorgeous 24fps anime cuts.

  • Real-time Dual Partner Integration: Active runtime querying to both ClickHouse Cloud (vector distance searches) and Parallel Web Systems in every analysis cycle.
  • Flawless Bilingual Studio UI: Polished Japanese and English localizations covering cinematography terminology.

What we learned

How effective Google's Gemini multimodal models are at deconstructing subtle cinematic elements like focal length, color contrast, and camera staging.

  • The power of combining high-speed columnar analytical databases (ClickHouse) with generative agents for instant memory retrieval.

What's next for CinemaDirector-AI: Agentic Cinema Director Studio

Full Storyboard & Timeline Assembly: Extending cut-by-cut refinement into full multi-scene sequence editing.

  • Automated Audio Stems Mixing: Combining Gemini Omni video output with AI-generated spatial foley and orchestral scoring.
  • Direct Export to NLEs: Exporting Final Cut Pro XML and DaVinci Resolve timelines directly from ClickHouse.

Built With

  • agentic-ai
  • clickhouse
  • gcs
  • gemini-3.8-flash
  • gemini-omni-1.1-flash
  • google-cloud
  • parallel-web
  • python
  • streamlit
  • vector-search
Share this project:

Updates

Submission history