Studio Director

Inspiration

Most agent demos reason over a toy dataset. We wanted one that runs a real business.

One of us operates an AI video factory: seven YouTube channels (Greek-myth animation in three languages, economics explainers, football what-ifs, Mandarin-learning shorts) with 407 published videos. Every decision - which channel to feed next, which format, how long, when to publish - was made by hand, from weeks of spreadsheet analysis. That is exactly the kind of judgement an agent should make, IF it can remember everything the studio has ever shipped.

What it does

Ask the Studio Director what the factory should produce next. It does not guess: it writes its own SQL against its ClickHouse memory, studies what actually performed (by channel, title pattern, duration, publish hour, even per-scene QA telemetry), and returns a production brief where every choice cites a number.

The result that surprised us: the agent independently rediscovered conclusions that took the human operator weeks of manual analysis.

• ToneTutor's winning format is the ~30-second Chinese-idiom short (measured by hand at 4.7x the channel's other formats).
• What If FC's winning formula is the human-vs-human cross-sport comparison ("Mbappe vs Usain Bolt", 1,551 views, the channel's best).
• The proven publish slots are 13:00 and 01:00 UTC - the exact two slots the human settled on.

Three SQL queries. Ten seconds.

And then it starts the job. Ask it to produce rather than recommend, and after the research it calls start_production with the brief. The factory takes it from there, with nothing outside Google Cloud doing the thinking or the talking: a Screenwriter (Gemini 2.5 Flash with a JSON response schema) pitches eight idioms, the factory drops every one the channel has already published (by characters and by pinyin slug - 90 so far) and writes the full spec for the first survivor; the per-character tones come from the pypinyin dictionary, not the model, and the spoken tone line is rebuilt from them. Cloud Text-to-Speech (Chirp 3 HD) narrates it in English and Mandarin. The channel's HyperFrames template renders it. It goes up private with publishAt baked into the insert, and the schedule is read back.

The first real order, placed by the agent itself during the demo recording: 指鹿为马 ("point at a deer, call it a horse") - a 45-second Short, private, scheduled for 2026-09-08 13:00 UTC, youtube.com/shorts/uzuJsH1hwIs. 129.7 seconds from production order to scheduled upload. The record lives in productions/ and becomes part of the memory for the next order.

The Director only starts production on a channel the factory has a production line for (today: @tonetutor, the 30-second idiom Short). Ask it to produce for a channel without one and the tool answers no_production_line; it can still analyse and recommend for all seven.

How we built it

• Gemini 2.5 Flash + Google Cloud Agent Builder (ADK) - the agent is an ADK LlmAgent, deployed unchanged to Vertex AI Agent Engine as a managed runtime.
• ClickHouse via the official MCP server - the agent's only data tool is run_query exposed by mcp-clickhouse (HTTP transport), consumed through ADK's McpToolset over MCP streamable-http. The MCP server and a self-hosted ClickHouse ride in one Cloud Run container: MCP is the public surface, ClickHouse listens on localhost.
• The memory: studio.videos (407 published videos: views, likes, duration, publish time), studio.production_scenes (628 storyboard scenes with QA telemetry from the generation pipeline), studio.episodes (finished cuts across languages). Ingested from the YouTube Data API and the pipeline's own logs.
• The production line (factory/) behind start_production: screenwriter.py (Gemini, JSON schema, dictionary-checked tones, duplicate filter), tts_google.py (Cloud Text-to-Speech, Chirp 3 HD; word timings laid out inside each segment's measured span), build_short.py (HyperFrames template + headless render), publish.py (upload preflight - the channel's uploader must authenticate before a second of narration is paid for - then a private upload with publishAt), produce.py (one call: brief in, record out).
• Where the tool runs matters: with the factory on the same machine it runs the line synchronously and returns the video URL; in the hosted runtime on Agent Engine there is no render farm, so the same tool queues the order and returns the receipt.

Challenges we ran into

• The official ClickHouse image declares VOLUME /var/lib/clickhouse, and classic Docker builders silently discard build-time writes to volume paths - the image built fine locally (BuildKit keeps them) and came up with an empty database in the cloud. Fix: bake data to a plain path.
• Cloud Run's front end intercepts Authorization headers on public services, which breaks ClickHouse HTTP basic auth. Co-locating the MCP server with ClickHouse removed the problem entirely.
• Agent Engine requirements must be pinned to locally verified versions (mcp==1.29.0); a loose range resolved to a build that failed at import inside the managed runtime.
• The channel's existing idiom renderer narrated with Kokoro and Microsoft edge-tts and timed its captions with Whisper - three AI tools from outside Google Cloud. The production line replaces exactly that stage with Cloud Text-to-Speech and proportional word timing, and keeps the rest of the template untouched.
• First recorded take of act 4: the Director ranked channels by average views and chose one the factory cannot build for. The fix was not a prompt tweak but a concept - the agent now knows the factory's production lines, and the tool refuses the rest.
• Gemini is not allowed to decide tones. It got 叶公好龙 right in rehearsal and would probably get most right, but "probably" is not a standard for a language-learning channel; the dictionary overrides the model every time.

What we learned

An agent is only as good as its memory. The same model, given a real production history in ClickHouse instead of a prompt full of assumptions, stops inventing strategy and starts citing evidence.

And an agent that can only recommend is a report. The moment it can start the job, every recommendation has to survive contact with the factory: the channel must have a line, the idiom must not be a repeat, the tones must be right, the upload must authenticate before anything is spent. Most of the engineering went into those refusals, not into the happy path.

What's next

Wire the other production lines (the Greek-myth animation channels, the football what-ifs) behind the same tool, and let every night's uploads land back in ClickHouse before the next order.

Built With

• adk
• agent-builder
• clickhouse
• cloud-run
• cloud-text-to-speech
• gemini
• google-cloud
• mcp
• python
• vertex-ai