The Problem We Solved
Creating a short film from a written story requires a team and weeks of work.
A writer finishes a story. To turn it into a short film they need:
- A cinematographer to plan shots
- An illustrator for storyboards
- A video editor for animation and assembly
- A voice artist for narration
- A sound designer for effects
- A colorist for the final grade
That's 6 specialists, expensive software, and weeks of back-and-forth. Most stories never make it to screen.
InkFrame collapses this entire pipeline into a single paste-and-click.
You paste any story - a novel excerpt, a screenplay fragment, a creative brief. InkFrame's agentic pipeline does the rest: rewrites the story for the camera, breaks it into scenes, generates a storyboard, animates every frame, records narration, mixes audio, grades the footage, burns in subtitles, and exports a production-ready MP4. In minutes, not weeks.
What We Built
InkFrame is an 8-stage agentic film production pipeline built entirely on Alibaba Cloud infrastructure:
| Layer | Technology |
|---|---|
| LLM reasoning | qwen-plus via DashScope OpenAI-compatible endpoint |
| Text-to-image | wan2.6-t2i via DashScope Wan API |
| Image-to-video | wan2.6-i2v-flash via DashScope async task API |
| Text-to-speech | cosyvoice-v2 via DashScope TTS SDK |
| File storage | Alibaba Cloud OSS (oss2 SDK) |
| Video assembly | ffmpeg running on Alibaba Cloud ECS |
| Backend | FastAPI (Python 3.12) on Alibaba Cloud ECS |
| Frontend | React 18 + Vite + Tailwind CSS |
Every AI call goes through DashScope. Every generated file is stored on Alibaba Cloud OSS. Zero third-party AI providers.
Live Demo Flow
1. Register / login
2. Paste any story (min 80 characters)
3. Watch live story quality score (hook, emotional arc, visual richness)
4. Pick video model + CosyVoice narrator + aspect ratio
5. Click Generate - watch all 8 stages run with live SSE progress
6. Preview each scene: storyboard frame → animated clip
7. Download final MP4 with burned-in subtitles
8. Optional: one-click dub to Chinese, Spanish, French, Arabic, and more
9. Chat with the AI Director (qwen-plus) to refine any scene
Architecture
High-Level System Diagram
┌─────────────────────────────────────────────────────────────────────────────┐
│ USER BROWSER │
│ React 18 SPA · Vite · Tailwind CSS · SSE progress │
└──────────────────────────────┬──────────────────────────────────────────────┘
│ HTTPS / REST / SSE
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ ALIBABA CLOUD ECS │
│ │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ FastAPI Backend (Python 3.12) │ │
│ │ │ │
│ │ /api/auth JWT auth (register / login) │ │
│ │ /api/projects CRUD + pipeline trigger + SSE progress │ │
│ │ /api/director Qwen streaming chat director │ │
│ │ /api/progress Server-Sent Events task stream │ │
│ └────────────────────────────┬─────────────────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────────▼─────────────────────────────────────────┐ │
│ │ 8-Stage Generation Pipeline │ │
│ │ (services/pipeline.py) │ │
│ │ │ │
│ │ 1 story_parser.py qwen-plus → show-don't-tell rewrite │ │
│ │ 2 story_parser.py qwen-plus → continuity extract │ │
│ │ 3 story_parser.py qwen-plus → scene breakdown (4-8 scenes) │ │
│ │ 4 runway_client.py wan2.6-t2i → storyboard frame per scene │ │
│ │ alibaba_cloud_oss → upload frame to OSS │ │
│ │ 5 runway_client.py wan2.6-i2v-flash → animate frame → MP4 clip │ │
│ │ 6 audio_service.py cosyvoice-v2 → narration MP3 per scene │ │
│ │ audio_service.py edge-tts → SFX per scene (fallback) │ │
│ │ 7 polish_service.py ffmpeg → color grade + film grain │ │
│ │ 8 assembler.py ffmpeg → concat + mix + subtitle burn │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ SQLite DB · Redis + RQ (optional async queue) · /data volume │
└─────────────────────────────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌──────────────────┐ ┌─────────────────────────┐ ┌───────────────────┐
│ DashScope LLM │ │ DashScope Wan + Voice │ │ Alibaba Cloud │
│ │ │ │ │ OSS │
│ qwen-plus │ │ wan2.6-t2i │ │ │
│ (story, scenes, │ │ (text-to-image) │ │ inkframe/frames/ │
│ director chat, │ │ │ │ inkframe/clips/ │
│ translation) │ │ wan2.6-i2v-flash │ │ inkframe/audio/ │
│ │ │ (image-to-video, │ │ inkframe/refs/ │
│ cosyvoice-v2 │ │ text-to-video) │ │ │
│ (TTS narration) │ │ │ │ Permanent HTTPS │
└──────────────────┘ └─────────────────────────┘ └───────────────────┘
└─────────────────────────┬──────────────────────────┘
Alibaba Cloud DashScope API
dashscope.aliyuncs.com · oss-cn-hangzhou.aliyuncs.com
Mermaid Diagram
graph TB
subgraph Browser["User Browser"]
UI[React 18 SPA]
end
subgraph ECS["Alibaba Cloud ECS"]
API[FastAPI Backend]
subgraph Pipeline["8-Stage Pipeline"]
P1[1 · Story Intelligence<br/>qwen-plus]
P2[2 · Frame Generation<br/>wan2.6-t2i]
P3[3 · Animation<br/>wan2.6-i2v-flash]
P4[4 · Audio<br/>cosyvoice-v2]
P5[5 · Polish<br/>ffmpeg]
P6[6 · Assembly<br/>ffmpeg]
P1-->P2-->P3-->P4-->P5-->P6
end
DB[(SQLite)]
FS[/data/outputs/]
API --> Pipeline
end
subgraph Alibaba["Alibaba Cloud DashScope"]
LLM[qwen-plus]
T2I[wan2.6-t2i]
I2V[wan2.6-i2v-flash]
TTS[cosyvoice-v2]
end
OSS[(Alibaba Cloud OSS<br/>inkframe-qwen)]
UI -->|REST + SSE| API
P1 --> LLM
P2 --> T2I
P3 --> I2V
P4 --> TTS
P2 -->|upload PNG| OSS
P3 -->|store MP4| FS
style Alibaba fill:#FF6A00,color:#fff
style OSS fill:#FF6A00,color:#fff
style ECS fill:#1565C0,color:#fff
style Browser fill:#1a1a2e,color:#fff
How We Built It - Step by Step
Step 1 - The Core Problem: Story → Visual Scenes
The first challenge: raw story text is full of told emotion rather than shown action. "She was nervous" generates a bad image. "Her hands shook as she reached for the door handle" generates a great one.
We built a Show-Don't-Tell rewriter using qwen-plus:
# services/story_parser.py
SHOW_DONT_TELL_PROMPT = """Rewrite the story so every emotion is shown through
physical action, expression, or environment. Replace abstract words (sad, angry,
happy) with concrete visual actions a camera can capture."""
def show_dont_tell(story_text: str) -> str:
return _call([
{"role": "system", "content": SHOW_DONT_TELL_PROMPT},
{"role": "user", "content": story_text},
], temperature=0.7)
Then a continuity extractor pulls characters, colour palette, and visual style - so every scene stays consistent:
CONTINUITY_PROMPT = """Extract: characters (name + appearance), palette
(3-5 word colour description), setting_style, time_of_day. Return JSON only."""
Then a scene parser breaks the rewritten story into 4–8 cinematic scenes, each with a visual_prompt, motion_prompt, narration_text, and sfx_prompt.
Step 2 - Storyboard Generation (wan2.6-t2i)
Each scene's visual_prompt feeds into DashScope's Wan 2.6 text-to-image API. The new multimodal protocol differs from the old ImageSynthesis SDK - we built a wrapper that handles both:
# services/runway_client.py
def _t2i_sync(prompt: str, size: str = "1696*960") -> str:
if IMAGE_MODEL == "wan2.6-t2i":
# New multimodal protocol
message = Message(role="user", content=[{"text": prompt}])
rsp = ImageGeneration.call(model=IMAGE_MODEL, messages=[message],
n=1, size=size, prompt_extend=True)
return rsp.output.choices[0].message.content[0]["image"]
else:
# Legacy wanx protocol
rsp = ImageSynthesis.call(model=IMAGE_MODEL, prompt=prompt, ...)
return rsp.output.results[0].url
All 4–8 frames are generated in parallel using asyncio.gather with a semaphore to respect DashScope rate limits.
Generated PNGs are immediately uploaded to Alibaba Cloud OSS via oss2.Bucket.put_object() to get permanent URLs - DashScope's temporary URLs expire in 24 hours.
Step 3 - Animation (wan2.6-i2v-flash)
The OSS frame URL feeds into DashScope's image-to-video async task API. This is a 3-step flow: create task → poll until SUCCEEDED → download:
def _create_i2v_task(img_url: str, prompt: str) -> str:
body = {
"model": "wan2.6-i2v-flash",
"input": {"prompt": prompt, "img_url": img_url},
"parameters": {"resolution": "720P", "duration": 5, "prompt_extend": True},
}
r = httpx.post(f"{DASHSCOPE_BASE}/services/aigc/video-generation/video-synthesis",
headers={"X-DashScope-Async": "enable", ...}, json=body)
return r.json()["output"]["task_id"]
def _poll_dashscope_task(task_id: str) -> dict:
while True:
r = httpx.get(f"{DASHSCOPE_BASE}/tasks/{task_id}", headers=...)
status = r.json()["output"]["task_status"]
if status == "SUCCEEDED": return r.json()["output"]
if status == "FAILED": raise RuntimeError(...)
time.sleep(5)
All scene animations run in parallel via asyncio.gather.
Step 4 - Narration + SFX (cosyvoice-v2)
Each scene's narration_text is synthesised by CosyVoice v2 - DashScope's multilingual TTS. We use the dashscope.audio.tts_v2.SpeechSynthesizer SDK:
# services/audio_service.py
def _cosyvoice_tts_sync(text: str, voice: str, out_path: Path):
synthesizer = SpeechSynthesizer(model="cosyvoice-v2", voice=voice)
audio = synthesizer.call(text)
out_path.write_bytes(audio)
Available voice presets: longxiaochun, longxiaochun_v2, longwan, longfei, longjing, longhua.
SFX is generated via edge-tts (offline, open-source) as a lightweight fallback - describing the ambient sound aloud in a whispered voice.
Step 5 - Cinematic Polish (ffmpeg)
Rather than a separate API call, we replicate color-grading effects using ffmpeg filter chains. The polish_prompt is parsed for style keywords:
# services/polish_service.py
filters = ["unsharp=...", "eq=saturation=1.15:contrast=1.05"]
if "film grain" in prompt: filters.append("noise=alls=8:allf=t+u")
if "dramatic" in prompt: filters.append("eq=contrast=1.25")
if "warm" in prompt: filters.append("colorbalance=rs=0.05:bs=-0.05")
if "cinematic" in prompt: filters.append("vignette=PI/5")
This runs entirely on the ECS instance - no extra API cost.
Step 6 - Assembly + Subtitles (ffmpeg)
The assembler concatenates all clips, mixes narration (100% volume) with SFX (30% volume) per scene, and burns in subtitles from an auto-generated SRT file:
# services/assembler.py
# Mix narration + SFX per scene
filter = "[0:a]volume=1.0[narr];[1:a]volume=0.3[sfx];[narr][sfx]amix=..."
# Burn subtitles
vf = f"subtitles={srt_path}:force_style='FontSize=18,PrimaryColour=&HFFFFFF'"
Step 7 - AI Director Chat (qwen-plus streaming)
A persistent chat interface lets users talk to "Alex", a qwen-plus powered film director. The backend streams SSE chunks:
# routers/director.py
async def stream():
async with _async_client.beta.chat.completions.stream(
model="qwen-plus",
messages=[{"role": "system", "content": system_with_project_context},
{"role": "user", "content": user_message}],
) as s:
async for chunk in s:
delta = chunk.choices[0].delta.content
if delta: yield f"data: {delta}\n\n"
yield "data: [DONE]\n\n"
The director knows the full project - title, story synopsis, and every scene's description and prompts.
Step 8 - Multi-Language Dubbing (qwen-plus + cosyvoice-v2)
One-click dubbing: translate narration via qwen-plus, re-synthesise with CosyVoice, merge back with ffmpeg:
# services/dubbing_service.py
translated = await asyncio.to_thread(_translate_sync, narration_text, "zh")
await asyncio.to_thread(_tts_sync, translated, "longxiaochun_v2", dubbed_audio)
# ffmpeg: replace audio track in final video
Step 9 - Real-Time SSE Progress
Every stage reports progress via Server-Sent Events. The frontend subscribes on generation start and updates the UI live:
data: {"stage": "generating_frames", "done": 2, "total": 6, "msg": "..."}
data: {"stage": "animating_clips", "done": 4, "total": 6}
data: {"stage": "assembling_video", "done": 1, "total": 1}
Step 10 - Story Quality Scoring
Before any generation starts, qwen-plus scores the story and gives actionable feedback in the UI:
{
"hook_score": 8,
"visual_richness": 7,
"suspense_score": 9,
"emotional_arc": "isolation → desperate action → sacrifice",
"overall_score": 82,
"tip": "Add a physical detail in the opening to anchor the reader",
"ready": true
}
Qwen Cloud Services - Complete Reference
| # | Service | Model ID | Used For |
|---|---|---|---|
| 1 | DashScope LLM | qwen-plus |
Show-don't-tell story rewrite |
| 2 | DashScope LLM | qwen-plus |
Character & palette continuity extraction |
| 3 | DashScope LLM | qwen-plus |
Scene breakdown → 4–8 cinematic scenes |
| 4 | DashScope LLM | qwen-plus |
Live story quality scoring |
| 5 | DashScope LLM | qwen-plus |
AI Director streaming chat |
| 6 | DashScope LLM | qwen-plus |
Multi-language dubbing translation |
| 7 | DashScope Wan | wan2.6-t2i |
Text → storyboard frame (1696×960) |
| 8 | DashScope Wan | wan2.6-i2v-flash |
Image → animated video clip (5s, 720P) |
| 9 | DashScope CosyVoice | cosyvoice-v2 |
Per-scene narration TTS |
| 10 | Alibaba Cloud OSS | oss2 SDK |
Permanent file storage for all generated assets |
DashScope endpoint: https://dashscope.aliyuncs.com/compatible-mode/v1 (OpenAI-compatible)
OSS endpoint: https://oss-cn-hangzhou.aliyuncs.com
Alibaba Cloud Deployment Proof
File: backend/alibaba_cloud_oss.py
This is the designated proof file. It demonstrates direct use of the oss2 SDK for Alibaba Cloud Object Storage:
import oss2
from config import OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET, OSS_BUCKET_NAME, OSS_ENDPOINT
def upload_to_oss(local_path: Path, object_key: str = None) -> str:
auth = oss2.Auth(OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET)
bucket = oss2.Bucket(auth, OSS_ENDPOINT, OSS_BUCKET_NAME)
with open(local_path, "rb") as f:
bucket.put_object(object_key, f, headers={"Content-Type": ...})
return f"https://{OSS_BUCKET_NAME}.{OSS_ENDPOINT_CLEAN}/{object_key}"
All AI inference uses DashScope (dashscope.aliyuncs.com) - the Alibaba Cloud / Qwen Cloud AI platform.
The /api/projects/{id}/assets endpoint lists all OSS objects for a project, verifiable at runtime.
Project Structure
inkframe-qwen/
├── LICENSE ← MIT (open source, required by hackathon)
├── README.md
├── .env.example ← All environment variables documented
├── Dockerfile ← Production container (Python 3.12 + ffmpeg)
├── docker-compose.yml ← Local dev: backend + frontend + redis
├── docs/
│ └── architecture.md ← Full architecture diagram (ASCII + Mermaid)
│
├── backend/
│ ├── main.py ← FastAPI app, CORS, routers
│ ├── config.py ← All Qwen Cloud / DashScope config
│ ├── database.py ← SQLAlchemy models (User, Project, Scene)
│ ├── auth.py ← JWT authentication
│ ├── task_store.py ← Generation task state (SQLite-backed)
│ ├── worker.py ← RQ worker entry point
│ ├── alibaba_cloud_oss.py ← ★ Alibaba Cloud OSS proof file
│ ├── requirements.txt ← dashscope, oss2, openai, ffmpeg-python
│ │
│ ├── routers/
│ │ ├── auth_router.py ← /api/auth/*
│ │ ├── projects.py ← /api/projects/* (CRUD + pipeline trigger)
│ │ ├── stream.py ← /api/progress/:task_id (SSE)
│ │ └── director.py ← /api/director/chat (qwen-plus streaming)
│ │
│ └── services/
│ ├── story_parser.py ← qwen-plus: rewrite / continuity / scenes / score
│ ├── runway_client.py ← wan2.6-t2i + wan2.6-i2v-flash + OSS upload
│ ├── audio_service.py ← cosyvoice-v2 TTS narration + edge-tts SFX
│ ├── polish_service.py ← ffmpeg cinematic color grade + grain
│ ├── assembler.py ← ffmpeg concat + audio mix + subtitle burn
│ ├── pipeline.py ← 8-stage orchestration (asyncio)
│ ├── dubbing_service.py ← qwen-plus translation + cosyvoice dubbing
│ └── storyboard_export.py ← Pillow storyboard PDF
│
└── frontend/
├── package.json ← React 18, Vite, Tailwind, axios
├── index.html
└── src/
├── App.tsx ← Router
├── pages/
│ ├── LandingPage.tsx ← Marketing + pipeline overview
│ ├── HomePage.tsx ← Project list + new film form
│ ├── ProjectPage.tsx ← 3-panel: scene strip / preview / director chat
│ ├── LoginPage.tsx
│ └── RegisterPage.tsx
├── components/
│ ├── Botanicals.tsx ← SVG botanical decorations
│ └── Flowers.tsx
├── hooks/
│ └── useSSEProgress.ts ← SSE subscription hook
├── contexts/
│ └── AuthContext.tsx ← JWT auth context
└── utils/
└── api.ts ← Axios instance with auth interceptor
Setup & Installation
Prerequisites
- Python 3.10+
- Node.js 18+
ffmpeginstalled and in PATH- Alibaba Cloud account → Get DashScope API key
Option A - Docker Compose (recommended)
git clone https://github.com/YOUR_USERNAME/inkframe-qwen.git
cd inkframe-qwen
# Copy and fill in your keys
cp .env.example .env
# Edit .env: set DASHSCOPE_API_KEY at minimum
docker compose up --build
# Backend: http://localhost:8000
# Frontend: http://localhost:5173
Option B - Manual
git clone https://github.com/YOUR_USERNAME/inkframe-qwen.git
cd inkframe-qwen
# ── Backend ──────────────────────────────────────────
cd backend
pip install -r requirements.txt
cp ../.env.example ../.env
# Edit .env: set DASHSCOPE_API_KEY
uvicorn main:app --reload --port 8000
# ── Frontend (new terminal) ───────────────────────────
cd ../frontend
npm install
npm run dev
# → http://localhost:5173
Environment Variables
# ── Required ──────────────────────────────────────────────────────────────────
# Qwen Cloud / DashScope key - get from https://dashscope.console.aliyun.com/apiKey
DASHSCOPE_API_KEY=sk-...
# JWT secret (generate with: python -c "import secrets; print(secrets.token_hex(32))")
SECRET_KEY=your-random-64-char-hex-string
# ── Alibaba Cloud OSS (for persistent storage beyond DashScope 24h URLs) ──────
OSS_ACCESS_KEY_ID=your_access_key_id
OSS_ACCESS_KEY_SECRET=your_access_key_secret
OSS_BUCKET_NAME=inkframe-qwen
OSS_ENDPOINT=https://oss-cn-hangzhou.aliyuncs.com
OSS_REGION=cn-hangzhou
# ── LLM ───────────────────────────────────────────────────────────────────────
OPENAI_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
LLM_MODEL=qwen-plus # qwen-max for best quality, qwen-turbo for speed
# ── Image generation ──────────────────────────────────────────────────────────
IMAGE_MODEL=wan2.6-t2i # or wanx2.1-t2i-turbo (legacy)
# ── Video generation ──────────────────────────────────────────────────────────
VIDEO_MODEL=wan2.6-i2v-flash # or wanx2.1-i2v-turbo (budget)
VIDEO_DURATION=5 # seconds per clip
VIDEO_RESOLUTION=720P # 720P or 1080P
# ── TTS narration ─────────────────────────────────────────────────────────────
TTS_MODEL=cosyvoice-v2
# Voice options: longxiaochun · longxiaochun_v2 · longwan · longfei · longjing · longhua
TTS_VOICE=longxiaochun
# ── App ───────────────────────────────────────────────────────────────────────
RUN_TASKS_INLINE=true # false = use Redis + RQ worker
GEN_CONCURRENCY=5 # parallel DashScope calls
CORS_ORIGINS=http://localhost:5173
API Reference
Authentication
POST /api/auth/register { email, password }
POST /api/auth/login { email, password }
GET /api/auth/me
Story Analysis (before generation)
POST /api/projects/analyze-story { story_text }
→ { hook_score, visual_richness, suspense_score, overall_score, tip, ready }
Projects
POST /api/projects create project + choose model/voice/features
GET /api/projects list all projects
GET /api/projects/:id project detail + all scenes
DELETE /api/projects/:id
Scene Editing
PATCH /api/projects/:id/scenes/:index edit visual/motion/narration/sfx prompts
Reference Images
POST /api/projects/:id/style-ref upload style reference → stored to OSS
POST /api/projects/:id/char-ref upload character reference → stored to OSS
Generation Pipeline
POST /api/projects/:id/generate start 8-stage pipeline → returns task_id
POST /api/projects/:id/cancel cancel running generation
GET /api/projects/:id/task-status current pipeline stage
Real-Time Progress
GET /api/progress/:task_id SSE stream: { stage, done, total, msg }
Outputs
GET /api/projects/:id/video download final MP4
GET /api/projects/:id/storyboard-pdf download storyboard PDF
POST /api/projects/:id/dub { target_lang: "zh" } → dubbed MP4
GET /api/projects/:id/assets list Alibaba Cloud OSS files
AI Director
POST /api/director/chat SSE stream: qwen-plus director response
POST /api/director/tool/regenerate update scene prompts from director feedback
GET /api/director/health
System
GET /api/projects/usage DashScope model info
GET /health
Features at a Glance
| Feature | Implementation |
|---|---|
| Show-don't-tell rewrite | qwen-plus system prompt, temperature 0.7 |
| Story quality score | qwen-plus → JSON with 5 metrics |
| Character continuity | Extracted by qwen-plus, injected into all scene prompts |
| Storyboard generation | wan2.6-t2i - 1696×960, prompt_extend=True |
| Scene animation | wan2.6-i2v-flash - async task, 5s clips, 720P |
| Narration TTS | cosyvoice-v2 - 6 voice presets |
| SFX | edge-tts (offline, lightweight) |
| Cinematic polish | ffmpeg filter chain - grain, curves, vignette, warmth |
| Subtitle burn | ffmpeg subtitles= with auto-generated SRT |
| Video assembly | ffmpeg concat + amix narration/SFX blend |
| AI Director chat | qwen-plus streaming SSE, project-context-aware |
| Multi-language dub | qwen-plus translation + cosyvoice-v2 re-synthesis |
| Persistent storage | Alibaba Cloud OSS via oss2 SDK |
| Auth | JWT (PyJWT) with secure HttpOnly cookies |
| Async queue | Redis + RQ (optional, inline for local dev) |
| Storyboard PDF | Pillow - frame grid with scene descriptions |
Track & Submission
Hackathon: Global AI Hackathon with Qwen Cloud
Track: Track 2 - AI Showrunner
Alibaba Cloud Proof: backend/alibaba_cloud_oss.py
Architecture Diagram: docs/architecture.md
License: MIT (see LICENSE)
Built With
- python
- qwen
- typescript
Log in or sign up for Devpost to join the conversation.