CutTo — AI Video Director

Inspiration

I run a kids' educational YouTube channel. Every video needs animated visuals, voiceover, music, and editing — which means expensive software and 4-8 hours per upload. Most teachers and educational content creators have the same problem: great lesson ideas, no practical way to turn them into engaging videos. I wanted to make video creation as simple as describing a lesson in a conversation.

## What it does

CutTo is an AI Video Director that turns any idea into a finished cinematic video. You describe what you want (via text or voice), and CutTo:

1. Shapes the vision — A creative AI director collaborates on the concept with genuine enthusiasm and strong creative opinions
2. Plans the video — Gemini generates a scene-by-scene plan with inline preview images (interleaved text + image output) so you see the visual direction before committing
3. Lets you edit — An interactive scene editor lets you modify narration, visuals, speakers, reorder scenes, and ask the AI for revisions in natural language
4. Generates the video — Veo 3.0 creates cinematic video clips using FLOW-decomposed prompts, Cloud TTS produces natural voiceover, and FFmpeg assembles everything with crossfade transitions
5. Delivers the result — Watch, download, and upload the finished MP4

Supports 6 video categories: animated stories, science explainers, documentaries, tutorials, marketing videos, and motivational pieces.

## How we built it

• Frontend: React 18 + Vite 5, WebSocket communication, Web Speech API for voice input
• Backend: Python 3.11 + FastAPI, WebSocket server, pipeline orchestrator
• AI Conversation: Gemini 2.5 Flash (interleaved text + image) via google-genai SDK
• ADK Integration: Multi-agent architecture with 3 specialized agents (Creative Director, Storyboard Artist, Orchestrator) using Google ADK with 4 function tools
• Video Generation: Veo 3.0 with FLOW-decomposition prompt engineering and fallback chain to Imagen 4.0
• Prompt Engineering: Every visual prompt is decomposed into 6 elements (subject, action, camera, lighting, palette, texture) then reconstructed into a cinema-grade shot brief
• Voice Synthesis: Google Cloud TTS (Journey/Chirp3 HD voices) with edge-tts fallback
• Video Processing: FFmpeg for fps normalization, crossfade transitions, Ken Burns effect, and clip assembly
• Testing: 119 automated tests covering pipeline, services, agent, API, and ADK agents

## Challenges we ran into

• Veo rate limits required staggered batching and a multi-level fallback chain (Veo → Imagen → Gemini native image)
• Mixed frame rates (24fps vs 30fps) from different Veo generations broke crossfade transitions — solved by normalizing all clips before assembly
• Audio/video duration sync — implemented two modes with FFmpeg atempo chaining
• Gemini interleaved output requires specific model + modality configuration that differs from text-only calls

## Accomplishments that we're proud of

• End-to-end video generation from a single conversation — no editing tools needed
• FLOW prompt decomposition produces dramatically more cinematic Veo output than raw prompts
• Multi-agent ADK architecture with specialized roles
• Voice input, reference image upload, interactive scene editor, real-time progress tracking
• 119 automated tests — production-quality engineering

## What we learned

• Gemini's interleaved text + image output is the killer feature for creative tools — showing previews during conversation makes planning feel immediate
• Prompt decomposition (FLOW method) is critical for realistic Veo output
• The gap between "AI generates content" and "AI directs a production" is about persona and workflow design
• ADK's agent framework provides clean separation of concerns for multi-step creative workflows

## What's next for CutTo

• Veo 3.1 integration with native audio and reference images
• Multi-language support for global classrooms
• Template library for common educational topics
• Export to social media formats (vertical, square)

Built With

• docker
• fastapi
• ffmpeg
• gemini
• google-adk
• google-cloud-tts
• imagen
• javascript
• python
• react
• veo
• websockets