StudioFX

Inspiration

Writing is often a lonely, linear process, but creativity is inherently branching and exploratory. I wanted to build a tool that felt less like a standard autocomplete and more like a collaborative "what if?" machine. Every time an author hits a story beat, there are dozens of different paths the plot could take. The goal behind StudioFX (StoryFX) was to visualize that branching narrative structure while using the full spectrum of Gemini's multimodal capabilities to make the exploration process cinematic - letting writers not just read their outline, but watch and hear it come to life.

What it does

StudioFX is a branching story outline engine. You start with a simple premise ("A detective wakes up with no memory"). Using Gemini 2.5 Pro, the engine generates structural "story beats" - concise, index-card style plot points. At the end of every beat, the AI provides two divergent paths forward, allowing you to branch the narrative. You can pick an AI suggestion, type your own path, or click back to any previous node in the visual story tree to explore an alternate timeline.

As you build the story, StudioFX is constantly generating interleaved multimodal outputs:

• Text: Streaming narrative beats via Gemini 2.5 Pro.
• Images: Every beat generates a cinematic scene visualization behind the scenes via Gemini 3.1 Flash Image.
• Voice: Gemini TTS Pro narrates the story using voice profiles mapped to the story's detected genre.

When you're ready, hit "Watch Story" to sit back and experience your branching outline as a fully narrated, visual storyboard that dynamically streams in.

How I built it

StudioFX is built on a modern React 19 + TypeScript frontend with a Node.js/Express backend, hosted on Google Cloud Run. The core of the application relies on a WebSocket connection between the client and server. When a user selects a narrative path, the backend passes the accumulated story memory to Gemini 2.5 Pro (using the new @google/genai SDK), which streams the story beat tokens back to the client in real-time.

Simultaneously, the backend uses Gemini 2.5 Flash for quick, inexpensive "utility" tasks: extracting memory from previous beats to keep the context window tight, detecting genre shifts, and generating specialized visual prompts. Once the text finishes streaming, the frontend immediately fires parallel HTTP requests to our proxy. The proxy hits Gemini 3.1 Flash Image to generate a cinematic still of the scene, and Gemini TTS Pro to generate the voice narration. The UI buffers these assets - displaying the AI's "thought process" on screen - and reveals the completed multimodal beat as one cohesive unit.

Challenges I ran into

The biggest challenge was LLM drift and repetition. I found that left to its own devices, the AI would generate very generic, cliché prose (e.g., "Little did they know..."). I spent significant time building an automated test harness (story-test.ts) to analyze the generated stories. I implemented a "health check" system with strict system prompt guardrails that bans clichés, enforces proper screenwriting pacing (like "Save the Cat"), and uses a three-tier semantic causality check to ensure the beat strictly follows the user's input rather than drifting off topic.

Another major challenge was UI latency for heavy media. Waiting 8 seconds for an image and voiceover to generate after reading a text block broke the pacing. I solved this by using WebSocket streaming for the text (so the user is engaged immediately) and then using that read-time to asynchronously pre-fetch the image and audio from Google Cloud, caching the results in IndexedDB for instant playback later.

Accomplishments that I'm proud of

The visual Node Tree. Building an interactive, SVG-based tree map that visually tracks the narrative branches as they grow, complete with smooth bezier curves and active state glowing, took a lot of finessing but completely changes how you interact with the story. I'm also incredibly proud of the "Watch Story" playback mode. Seeing Gemini's text, images, and voice all synchronized in a cinematic presentation truly sells the concept of a multimodal storyteller.

What I learned

I learned the profound difference between "Pro" and "Flash" models, and how to orchestrate them. I started using Gemini Pro for everything, but quickly realized we could drastically reduce latency and cost by offloading validation, memory extraction, and prompt rewriting to Gemini 2.5 Flash, reserving the powerful 2.5 Pro (with Thinking enabled) strictly for the creative heavy lifting of writing the story beats and paths.

What's next for StudioFX (StoryFX)

1. Character Consistency: Implementing a persistent "Character Bible" stored in Firestore that Gemini can reference to ensure visual consistency across the generated images for the same character in different beats.
2. Export to Script: A tool that takes the complete branching outline and expands it into standard screenplay format.
3. Multi-User Collaboration: Allowing writers to invite others to their tree, leaving comments on specific nodes, or proposing alternate branches asynchronously.

Built With

• 19typescriptnode.jsgoogle
• 2.5
• cloud
• react
• rungemini