Direct-to-Mood Marketing Studio

An AI Creative Director that generates a complete marketing campaign — copy, images, and video in one click. Powered by Gemini's native interleaved output, it thinks, drafts, critiques, and delivers

Comment

Inspiration

I watched a friend spend an entire Saturday afternoon building a single Instagram campaign for her small bakery. She wrote copy in ChatGPT, generated images in Midjourney, edited them in Canva, scripted a video in a separate tool, and then spent another hour trying to make everything feel like it belonged to the same brand. The hook said "artisan warmth," but the image looked clinical. The video had a completely different energy.

That disconnect — five tools, no shared creative vision — is the problem. I wanted to build what a real creative director does: hold the entire campaign in their head, weave text and visuals together so each asset references and reinforces the others, critique their own work, and iterate until it's cohesive. Except this creative director works in thirty seconds instead of thirty hours.

When Google announced the Gemini Live Agent Challenge with native interleaved TEXT + IMAGE output, I knew this was the missing piece. Not chaining separate text and image calls together, but generating them in a single pass where the AI sees the text it just wrote while creating the image that accompanies it.

What it does

Interleaved output changes everything. When Gemini generates text and images in one generateContent call with responseModalities: [TEXT, IMAGE], the visual assets are genuinely informed by the copy context. A "warm and inviting" mood produces golden-lit photography that matches cozy copy. This is fundamentally different from generating text first and then prompting a separate image model.

Agentic workflows dramatically improve quality. My first version was a single prompt that generated everything at once. It worked, but the output was inconsistent. Adding an explicit pipeline — brief extraction, drafting, QA scoring, and auto-revision—transformed the results. Having the AI score its own cohesion and compliance, then revise if the score was below 8/10, produces campaigns that feel professionally art directed.

Veo's audio is not precisely controllable. I spent significant time trying to get Veo 3.1 to narrate a full script within an 8-second video without cutting off. The solution was to stop fighting the limitation: generate silent cinematic visuals with Veo, and layer browser-based speech synthesis for voiceover. The narration now always completes because we control it, and the video provides stunning visuals to accompany it.

JSON parsing from LLMs requires defensive engineering. Even with responseMimeType: 'application/json,' Gemini occasionally wraps output in markdown fences. I built the extractJsonBlock and safeJsonParse utilities, plus a heuristic QA fallback, so the app never crashes on malformed AI output.

How we built it

The architecture is deliberately simple — two files do all the work:

server.js — A Node.js/Express backend that implements the agentic pipeline:

Brief Extraction: Takes the user's business description (and optional uploaded logo) and uses gemini-2.5-flash to extract a structured brand brief—voice, audience, tone, color palette, constraints.

Interleaved Draft: Sends the brief to gemini-2.5-flash-image with Modality.TEXT and Modality.IMAGE enabled. Gemini generates alternating text and image assets in a single API call. The Creative Director system prompt instructs it to write a hook, generate a matching hero image, write storytelling copy, generate a lifestyle image, and close with a CTA.

QA Scoring: A separate Gemini-2.5-flash call evaluates the generated assets for cohesion (do they feel like one campaign?) and compliance (do they respect channel, reading level, and taboo word constraints?). Returns scores, issues, and suggested fixes.

Auto-Revision: If cohesion scores are below 8/10, the pipeline automatically regenerates with targeted fix instructions—transparent to the user, who just sees better output.

Video Generation: Calls ai.models.generateVideos with Veo 3.1 to produce a cinematic MP4 clip based on the campaign's CTA copy. The video generates asynchronously and populates automatically when ready.

Everything streams to the frontend via server-sent events—the user watches each step unfold in real-time: director notes, brief, assets appearing one by one, QA scores, and video status.

index.html — A single-file React 18 SPA (no build step) that parses the SSE stream and renders assets as they arrive. It also provides:

Per-asset controls: regenerate individual images, refine text with natural language, share to social media Multimodal input: upload a brand logo that gets sent as base64 alongside the text prompt Export: download everything as a zip bundle (JSON, Markdown, PNGs, MP4) Voiceover: Web Speech API reads all text assets aloud, synced to video playback Deployed on Google Cloud Run (Docker container, europe-west1) and Vercel (serverless).

Challenges we ran into

The empty campaign problem. Early on, the app successfully generated 4 assets, but the page appeared blank. The SSE event parsing was using evt.type for the asset kind, but that collided with the event routing field. Normalizing evt.assetType and || evt.type fixed the rendering.

QA is always scoring 0/10. The QA prompt used literal \n (escaped backslashes) instead of actual newlines, and the response came back wrapped in markdown fences that broke JSON.parse. Adding responseMimeType: 'application/json' to force raw JSON, plus extractJsonBlock as a safety net, resolved it.

Image regeneration failing silently. Requesting only Modality.IMAGE from generateContent is unreliable — Gemini sometimes returns nothing. Switching to [Modality.TEXT, Modality.IMAGE] and asking for an interleaved label + image, then extracting just the image part, made single-image regeneration consistent.

Video narration cutoff. Veo 3.1's internal speech synthesis truncated voiceover scripts longer than a few seconds. Rather than fighting an uncontrollable API behavior, I pivoted: Veo generates silent cinematic visuals, and the browser's Web Speech API handles narration — guaranteed to complete every word.

Auto-retry storm. When Veo hit a rate limit (HTTP 429), the useEffect that auto-triggered video generation kept re-firing because the failure reset isVideo Generating to false, re-satisfying the trigger conditions. Adding a videoAttempted flag ensured auto-generation fires exactly once, with manual retry available via button.

Each challenge reinforced the same lesson: when building on top of generative AI, defensive design is not optional—it is the product.

Accomplishments that we're proud of

True single-call interleaved generation. This isn't text-then-image chaining. Gemini generates copy and visuals in one generateContent call, so the hero image is literally born from the same context as the hook that precedes it. The result feels art-directed, not assembled.

A working agentic pipeline, not a demo. The app doesn't just prompt and return — it extracts a brief, drafts, scores its own work, and auto-revises if quality falls short. Every step is real, streamed live, and observable. No mockups, no hardcoded responses, no shortcuts.

One click, five asset types. A user types a business description, clicks Generate, and receives copywriting, hero images, lifestyle visuals, a cinematic video clip, and a QA scorecard — without touching a second button. The video populates automatically when Veo finishes.

Bulletproof narration. After discovering that Veo's internal voiceover cuts off unpredictably, we built a hybrid approach: Veo delivers stunning silent visuals, and the browser's Web Speech API delivers the full narration synced to playback. The voiceover completes every word, every time, on every device.

Zero-build frontend. The entire UI — React 18, SSE parsing, speech synthesis, zip export, social sharing — lives in a single index.html file with no build step. Clone, npm start, done. This made deployment to both Cloud Run and Vercel trivial.

Resilient AI integration. Gemini wraps JSON in markdown fences? Handled. Image-only modality returns nothing? Handled. Veo hits a rate limit? One attempt, clear error, manual retry button. QA parsing fails? Heuristic fallback kicks in. The app never crashes on unexpected AI output.

Real guardrails with real enforcement. Channel targeting, reading level, and taboo word filtering aren't just UI dropdowns — they flow into the prompt, get verified during QA scoring, and trigger auto-revision if violated. The compliance score means something.

Per-asset control after generation. Users aren't locked into all-or-nothing. Regenerate one image without touching the copy. Refine a headline with natural language. Share a single asset to Instagram. Download everything as a zip. The creative director generates the campaign; the user retains full editorial control.

What we learned

Interleaved multimodal output is a fundamentally different paradigm. Before this project, we treated text and image generation as separate steps — generate copy, then generate a matching image. Gemini's native TEXT + IMAGE modality proved that generating them together in a single call produces assets that are contextually entangled. The AI doesn't just "match" the mood — it carries the same creative thread through both mediums. That distinction between chaining and interleaving is subtle but the quality difference is obvious.

Agentic self-critique is the single biggest quality lever. Adding a QA scoring step where Gemini evaluates its own output transformed campaign quality more than any amount of prompt engineering on the generation step. The model is surprisingly good at spotting when a campaign lacks cohesion or violates constraints — often catching issues that aren't obvious to a human on first glance. Auto-revision based on that score made the difference between "decent AI output" and "this looks professionally produced."

LLM output is never as structured as the API promises. Even with responseMimeType: 'application/json', Gemini occasionally returns markdown-fenced JSON, extra whitespace, or partial objects. We learned to treat every AI response as untrusted input — extractJsonBlock, safeJsonParse, and heuristic fallbacks became as important as the prompts themselves. Defensive parsing isn't a nice-to-have; it's the foundation of a reliable AI product.

You can't control what you don't own. Veo 3.1 generates beautiful video, but its internal audio pacing is a black box. We spent hours crafting prompts with "CRITICAL VOICEOVER RULES" trying to force complete narration — none of it worked reliably. The breakthrough was accepting the limitation and moving narration to something we fully control: the browser's Web Speech API. The lesson generalizes — when an AI capability is unreliable, don't fight it, route around it.

Streaming transforms the user experience. The difference between waiting 30 seconds for a JSON blob and watching assets appear live — director notes, brief, images materializing one by one, QA scores populating — is enormous. Server-Sent Events added minimal backend complexity but made the app feel alive and transparent. Users trust the output more when they can watch the reasoning unfold.

Simple architecture scales better than clever architecture. Two files — server.js and index.html — power the entire application. No framework, no build step, no database, no state management library. This made debugging fast, deployment trivial (one Docker container, one static file), and the codebase accessible to anyone who reads JavaScript. The temptation to add complexity was constant; resisting it was the right call.

Rate limits are a feature, not just an error. Hitting Veo's 429 quota limit exposed a retry storm in our useEffect logic that would have degraded the experience for any user on a free tier. Building a videoAttempted guard taught us that every async side effect in a React app needs an explicit "attempted" state — not just "loading" and "done." Failure is a state that needs its own design.

What's next for Direct-to-Mood Marketing Studio

Multi-campaign memory. Right now every generation starts from scratch. The next step is persistent brand profiles — the Creative Director remembers your logo, tone, color palette, and past campaigns so each new generation builds on established brand identity rather than rediscovering it.

A/B variant generation. Instead of one campaign, generate two or three mood variants side by side and let the user compare. "Show me this campaign in Warm and in Energetic" — then pick the winner or merge the best elements from each.

Gemini Live (real-time voice). Replace the text input with a live conversation. Describe your business by talking, get director feedback spoken back, and refine the campaign through natural dialogue — a true creative brainstorming session with an AI partner.

Platform-native export. Go beyond zip downloads. Direct publishing to Meta Business Suite, TikTok Creative Center, LinkedIn Campaign Manager, and YouTube Studio via their APIs — one click from generated campaign to live ad.

Scheduled campaign series. Generate a week or month of coordinated content — Monday's Instagram post, Wednesday's TikTok, Friday's LinkedIn article — all maintaining brand cohesion across the series, not just within a single campaign.

Imagen 4 and Veo 4 integration. As Google releases next-generation image and video models, plug them directly into the pipeline. The agentic architecture is model-agnostic — swapping the generation model requires changing one line, not rewriting the workflow.

Collaborative editing. Let teams work on the same campaign. A founder generates the initial draft, a designer regenerates specific images, a copywriter refines the hook — all in the same live session with real-time sync.

Analytics feedback loop. After a campaign is published, feed engagement metrics back into the Creative Director's context. "Your warm-toned campaigns get 3x more engagement on Instagram" — so the next generation is informed by what actually works, not just what looks good.

Built With

  • auto-revision
  • babel-jsx
  • babel-standalone
  • cors-npm
  • docker
  • dotenv
  • express.js
  • file-operations-backend-node.js-18+-server-runtime-backend-framework-express.js-4-api-routing
  • gemini-2.5-flash
  • gemini-2.5-flash-image
  • genai-1.44+-sdk
  • generatevideos
  • git-github
  • google-cloud-run
  • google-font
  • jszip-3.10
  • markdown
  • middleware
  • multi-turn-refinement-ai-?-video-veo-3.1-(veo-3.1-generate-preview)-cinematic-mp4-video-clip-generation-ai-sdk-@google/genai-(v1.44+)-official-google-genai-sdk-for-node.js-?-generatecontent
  • no-build-step)-single-file-spa-with-hooks-(usestate
  • node.js
  • nodemon
  • pngs
  • qa-scoring
  • react-18+
  • responsive-design-typography-google-fonts-(outfit
  • server-sent-events
  • static-file-serving-frontend-react-18-(standalone
  • tailwind-css
  • usecallback)-frontend-?-jsx-babel-standalone-in-browser-jsx-transpilation
  • useeffect
  • useref
  • veo-3.1
  • vercel
  • web-share-api
  • web-speech-api
Share this project:

Updates

Submission history