Overview
Sketch2Story is a storyboard and pre-production tool that turns sketches, images, and text into structured visual sequences. It handles sketch-to-image conversion, selective frame editing, in-between shot generation, and production breakdowns. The system is built for creators who need fast iteration, consistent framing, and reliable scene interpretation.
What it does
Sketch2Story lets creators build storyboard panels from sketches, uploads, or text. It can convert rough drawings into clearer frames, enrich panels with notes, generate missing shots between key moments, and produce a production checklist that includes props, clothing, and character details. Users can also edit existing frames with simple text instructions and export the full sequence as a video.
Sketch2Story also includes a Production Breakdown view that transforms storyboard panels into actionable pre-production data. The system extracts props, wardrobe elements, and character details directly from the visual sequence and merges them into a clean, deduplicated checklist. This allows creators to move from rough sketches to practical planning materials without additional manual work. By grounding the output in structured scene analysis, the tool supports real workflows used in film, animation, and commercial production.
How we built it
The project is built with a local-first architecture using vanilla JavaScript, with all project data stored securely on the device through localStorage. This approach enhances security and privacy, as no user data, API keys, or passwords ever leave the browser. The UI is built to be a clean, intuitive front-end for a sophisticated AI engine operating behind the scenes.
Our core innovation is a multi-step, agentic workflow where an LLM acts as an "AI Director". This agent translates simple user interactions into the precise, structured inputs that FIBO requires for high-quality generation. For sketch-to-image conversion, we ensure the user's composition is respected by passing the drawing as a control image. For panel edits, the AI Director analyzes the user's text instruction and the current image to generate a targeted inpainting prompt, preserving the integrity of the rest of the frame.
The "in-between" generation showcases our most advanced technical chain. The system sends the image data from two keyframes, the start and end panel, to the AI Director. The agent performs a multi-frame visual analysis to reason about the logical action that must occur between them. It then composes a new, detailed narrative prompt describing this transitional shot. This AI-generated prompt, along with the most relevant of the two keyframes as a visual reference, is then sent to the FIBO API to generate a cohesive and contextually accurate in-between panel. This entire process requires zero textual input from the user, demonstrating true narrative inference.
For the Production Breakdown, we engineered a robust data extraction pipeline. The AI Director analyzes batches of panels, including both images and user notes, and uses a forced tool call to return a structured JSON object containing lists of props, clothing, and detailed character descriptions. This is more reliable than simple text parsing. To ensure a clean and practical final list for the user, we run these initial results through a second, specialized AI pass that aggressively deduplicates and merges similar items, transforming raw data into a production-ready checklist.
The system is designed so that all of the heavy technical detail happens behind the scenes. The AI Director automatically composes prompts that cover every relevant parameter, including angle, lighting, realism, shape, and other key descriptors, and then relies on FIBO’s built in natural language to JSON conversion to translate those instructions into fully controlled outputs. By keeping this complexity inside the AI layer, the interface stays focused on simple creative actions rather than exposing users to long prompt strings or technical controls. The result is a clean user experience on the surface, supported by a highly structured and advanced workflow underneath.
Why this UX is new
Traditional storyboard tools require creators to manually redraw variations, adjust camera angles by hand, and fill in missing moments between key frames. Sketch2Story introduces a different workflow: creators sketch or upload rough ideas, and the system handles the technical complexity of generating consistent, camera-aware, production-ready frames. The interface is built around natural creative actions: draw, edit, rearrange, etc. while the AI Director translates each interaction into structured FIBO controls behind the scenes.
The in-between generation feature also removes a major source of friction in storyboarding: missing transitional shots. With a single click, the system analyzes two panels, infers the implied action, and produces a coherent intermediate frame without user input. This creates a fluid, timeline-based creative experience that feels closer to professional pre-production software than to typical AI image generation tools.
Built With
- fibo
- openai
Log in or sign up for Devpost to join the conversation.