ClipForge

ClipForge turns valuable text into living video stories with scripts, shots, voice, generated media, and editable drafts.

Comment

Inspiration

Most AI tools today are built around one person talking to one assistant. That is useful, but it can also make creative work feel more isolated. So much valuable text never gets seen.

A good article, a product idea, a lesson, a speech, a research note, a personal story, or even a short paragraph can contain real value. But in today's content world, plain text is often too quiet. People scroll past it before they understand it.

ClipForge was inspired by a different idea: AI should help people communicate, collaborate, and create together. For video creation, the real problem is rarely just "generate a clip." Creators still need to discuss the story, agree on the hook, split the script into shots, keep visual style consistent, choose voice and music, review assets, and turn everything into an editable draft. ClipForge was inspired by a simple vision: make every valuable piece of text move, speak, and become something people can watch.

I built ClipForge to make AI act more like a creative production partner. It helps organize the messy parts of video making so humans can focus on the message, story, and final decisions. I wanted to build a tool that takes meaningful text and helps turn it into a visual story: with structure, scenes, images, narration, sound, motion, and an editable video draft. The goal is not to generate random videos. The goal is to help good ideas become easier to show, share, and remember.

What it does

ClipForge is an AI video creation workspace that turns an idea, URL, script, or visual reference into a structured short-video production plan.

It can help creators: It helps transform text into video-ready assets:

  • generate short-video scripts from a topic, URL, prompt, or reference material;
  • break scripts into characters, scene cards, shot lists, narration, dialogue, and sound cues;
  • keep shots connected through reusable scene information and visual style settings;
  • generate or manage images, video clips, narration audio, sound effects, and music;
  • use Qwen3-TTS through a Hugging Face Space adapter for narration-style audio;
  • use Gemma 4 for text planning, visual analysis, and transition prompt generation;
  • connect external media-generation tools through MCP providers;
  • save projects, media assets, scene cards, output archives, and agent runs;
  • move selected shots, subtitles, audio, and video assets toward a CapCut-style editable draft.
  • script polishing and rewriting;
  • story and scene breakdown;
  • characters, scene cards, and shot lists;
  • visual prompts and generated images;
  • narration, dialogue, sound effects, and music;
  • image-to-video and video assembly workflows;
  • saved media assets and output archives;
  • CapCut-style editable draft structure.

The goal is not to replace the creator. The goal is to let AI handle structure and production flow, while people keep control over the story, tone, visual direction, and final edit. In simple terms, the workflow is:

How I built it

$$ valuable\ text \rightarrow story\ structure \rightarrow scenes \rightarrow media \rightarrow moving\ presentation $$

ClipForge is built as a full-stack creator tool. The current project supports multiple AI providers and media tools. It includes Qwen3-TTS narration through a Hugging Face Space adapter, Gemma 4 for planning and visual reasoning, remote MCP tool support, and a web workspace for script, image, video, audio, music, media library, and output management.

The web app uses Next.js, React, TypeScript, Tailwind CSS, and Radix UI. The interface is organized around a real production workflow: Workbench, Script Creator, Image Studio, Video Studio, Audio Studio, Music Studio, Media Vault, Output Archive, and Configuration.

How I built it

The backend uses FastAPI, async SQLAlchemy, Pydantic Settings, and provider-based AI services. Instead of hardcoding one model into the product, the backend separates text, vision, image, video, audio, and music capabilities behind provider contracts. I built ClipForge as a full-stack AI creator workspace.

That provider layer currently supports Gemma 4, Gemini, Xiaomi MiMo, Hugging Face Spaces, OpenAI-compatible APIs, Anthropic, and remote MCP tools. The Qwen3-TTS adapter is used for narration generation through Hugging Face Spaces. The project also includes a native iOS SwiftUI client so the same backend can support mobile chat-based creation. The frontend is built with Next.js, React, TypeScript, Tailwind CSS, and Radix UI. The interface is organized around real creator tasks: Workbench, Script Creator, Image Studio, Video Studio, Audio Studio, Music Studio, Media Vault, Output Archive, and Configuration.

The most important design choice was to treat video creation as a pipeline, not a single prompt: The backend is built with FastAPI, async SQLAlchemy, Pydantic Settings, and a provider-based AI service layer. Instead of tying the product to one model, the backend separates text, vision, image, video, audio, and music capabilities into provider contracts.

$$ idea \rightarrow script \rightarrow shots \rightarrow media \rightarrow review \rightarrow editable\ draft $$ That architecture lets the project connect different tools for different steps:

That structure makes it easier for creators to discuss and adjust each step instead of accepting one black-box AI output.

  • text and visual reasoning for planning;
  • Qwen3-TTS-style narration generation;
  • image and video generation providers;
  • music and sound generation;
  • MCP tools for external media services;
  • storage for projects, scene cards, generated assets, and output archives.

The most important product decision was to treat video generation as a production pipeline, not a single prompt. Good videos need a sequence of decisions: what the text means, how it should be structured, what each scene should show, how it should sound, and how it should be edited.

Challenges I ran into

The hardest challenge was making many different AI outputs feel like one coherent workflow. The biggest challenge was turning unstructured text into structured video production data.

Different providers return different result formats. Some return file paths, some return URLs, some return nested JSON, and some return partially structured text. The backend needs adapters, result parsing, and fallback logic so creators do not have to think about those differences. A paragraph does not automatically become a good video. The system has to find the main message, split it into scenes, preserve continuity, generate shot-level descriptions, and prepare media prompts that still match the original meaning.

Another challenge was keeping scene continuity. A good short video needs characters, locations, mood, visual style, narration, and transitions to stay consistent across shots. ClipForge handles this by breaking scripts into reusable scene cards and shot-level production data. Another challenge was connecting many AI providers into one workflow. Different tools return different formats: text, JSON, local file paths, remote URLs, audio files, image assets, and video files. ClipForge uses provider adapters and parsing logic so the user can work with one creation flow instead of many disconnected tools.

Long-running media generation was also a challenge. Text planning is fast, but video, audio, and music generation can take longer. The system separates planning, provider calls, asset storage, and export orchestration so each part can improve independently. I also had to think about editability. A generated video is useful, but an editable draft is more useful. That is why the project keeps scripts, scene cards, shots, media assets, and output structures as separate pieces instead of hiding everything inside one final file.

What I learned

I learned that useful creative AI is less about one impressive generation and more about helping people move through a full workflow. I learned that AI video creation is not just about making pixels move. It is about preserving the value of the original text while changing the form of expression.

A creator does not only need an answer. They need structure, options, editable pieces, and a way to share or review the work with others. In that sense, the best AI products should not pull people away from each other. They should give people better shared material to talk about, revise, and build on. If the AI ignores the meaning, the video becomes decoration. If the AI only summarizes, the result feels flat. The useful middle ground is to turn text into a sequence of visual decisions that still carries the original idea.

I also learned that model flexibility matters. Script planning, visual understanding, narration, image generation, music, and video generation each need different strengths. A provider-based architecture makes the product more practical because the workflow can use the right model or tool for each task. I also learned that a practical creator tool needs flexibility. One model may be good at planning, another at narration, another at image generation, and another at video. A provider-based architecture makes the workflow more realistic because each step can use the right tool.

What's next

Next, I want to improve ClipForge in three directions: Next, I want to make ClipForge better at turning different types of valuable text into different visual formats:

  • a stronger timeline editor for trimming, previewing, and reviewing shots;
  • better collaboration features so multiple people can comment on scripts, shots, and generated media;
  • more model-specific presets for Qwen, Moonshot/Kimi, Gemma, Gemini, and MiMo workflows.
  • educational notes into explainer videos;
  • product copy into launch videos;
  • articles into short social clips;
  • stories into cinematic scenes;
  • presentations into narrated video summaries.

The long-term vision is a creator workspace where AI helps people build together, not just chat alone. I also want to improve the timeline editor, strengthen CapCut draft compatibility, and add better review tools so creators can refine the generated scenes before publishing.

The long-term vision is simple: if a piece of text is worth reading, ClipForge should help make it worth watching.

Built With

Share this project:

Updates