StoryFlow - screen recordings into narrated tutorials

Turn a screen recording into a narrated tutorial. StoryFlow keeps your words, cuts the rambling, and re-voices it in sync with what's on screen - in English or a local language.

Comment

Inspiration

Shipping the product is only half the work. Every release leaves someone - product, support, enablement - rebuilding the same walkthrough by hand: record it, script it, re-record it because the timing drifted, then do it again next sprint.

The tools that exist mostly replace your screen with an avatar or your voice with a robot reading a script you still had to write. We wanted the opposite: keep the real product footage, keep the person's own words and their order, and fix only the delivery.

Two situations kept coming up, and they need genuinely different products:

  1. You talked through the recording - so the words already exist, buried in filler and false starts.
  2. You recorded silently - so the words don't exist at all, and only you know what the screen means.

StoryFlow has a mode for each.

What it does

Auto mode - the recording already has your voice. StoryFlow extracts the audio, transcribes it with word-level timestamps (ElevenLabs Scribe), and places segment boundaries at the pauses in your own speech, corroborated against detected scene cuts. It samples a mid-frame per segment so the model can see what was on screen. GPT-5.6 Luna then cleans each span to its time budget - filler and false starts removed, step order preserved, mis-heard product names repaired against a glossary. You review every line for free, and only then does it synthesize the voiceover and stitch it back.

Manual mode - the recording has no voice. You cut the timeline where the topic changes and write what each segment should say. As you type, StoryFlow estimates the spoken length against that segment's actual interval and tells you whether it will fit, before you spend anything. You pick a professional voice, and the same renderer builds the tutorial.

Both modes share the honest part. At render time the synthesized speech is measured, not assumed: each line is spoken, timed against its slot, rewritten to length if it overruns, and re-synthesized - keeping the best-scoring take. We never slow the voice down to fill a gap. Segments you leave blank keep the original audio rather than inventing narration. Output is available in English, Mandarin (Mainland and Malaysian), Malay, and Indonesian.

How we built it

Next.js 15 on Vercel for the app and API. Clerk for identity, Stripe for credits. Convex for state. Private S3 with presigned URLs via Vercel OIDC - no long-lived AWS keys. Long jobs go to SQS and run on a containerized Lambda worker with FFmpeg; after() is only the local fallback. All AWS resources are declared in Terraform.

The semantic layer is GPT-5.6 Luna on the OpenAI Responses API, reached through a reversible adapter (src/lib/ai-model.ts) that speaks either directly to OpenAI or through a gateway, normalizing model ids across both naming schemes. Every successful call records a sanitized trace - provider, api, model, response id - and the "GPT-5.6 Luna" badge in the UI renders only when such a trace exists. It reports the real model id if anything else served the request, so the badge cannot lie about provenance.

Codex did the implementation work across the event: the GPT-5.6 integration and Responses adapter, the sanitized trace plumbed through types/store/schema, the honest-failure refactor, auth and ownership hardening across the API routes, the live LLM UAT harness, and the Playwright automation that recorded the demo film itself - including the edit-decision-list renderer that cuts the picture to a per-beat narration manifest so audio and video cannot drift.

Challenges we ran into

A production bug that reported success. While filming, the review screen showed narration that was suspiciously close to the raw transcript. Every LLM credential on the worker had died at once - gateway 401, OpenAI 401, fallback provider returning nothing - so each segment fell through to a deterministic filler-strip that only deletes "um" and "uh". Users were getting their transcript minus filler instead of AI-rewritten narration, and nothing surfaced it: the unconfigured path and the call-failed path converge on the same fallback, and the pipeline reports success either way. The only tell was the absent badge. Word counts told the story: 30 → 22, 20 → 12, 10 → 8 once it was fixed.

Our own migration had quietly orphaned a model. The worker was still configured for a previous model that the Responses API migration had made unreachable - the gateway returns convert_request_failed for it on /responses while still serving it on /chat/completions. It had been dead config for days, failing into the same silent fallback. We only caught it by A/B-ing both models through the real production function.

Making a demo that proves rather than asserts. Our first cut narrated our own marketing page - circular, and it demonstrated "we clean up real rambling" using a voice that never rambled. We rebuilt the source material on real signed-in product surfaces, blurred the one-time API key secret in-page and revoked the key after the shoot, and added assertions to the recording script: it aborts rather than film a canned-transcript fallback, and verifies the raw-vs-cleaned contrast is actually within frame before banking the take.

Accomplishments that we're proud of

  • Failure states that are honest. A provider outage now surfaces as a visible error instead of a silent downgrade. The model badge is absence-shaped - it appears only on real recorded evidence.
  • Review before you pay. Analysis pauses at review and costs nothing; a credit is charged only when generation succeeds.
  • Timing that is measured, not estimated. The render loop times the actual synthesized speech against each interval and rewrites to length, rather than trusting a word-count heuristic.
  • A demo we can defend line by line. Every claim in the film maps to a specific code path, and the two capabilities that don't exist - voice cloning, and drafting a script from a silent video - are deliberately not claimed.

What we learned

A fallback that never raises is a bug with good manners. Our degradation chain was designed to keep users working during an outage, and it did exactly that - so well that a total credential failure looked like a successful render for days. Fallbacks need to be observable at the surface, not just in logs. The one thing that saved us was a badge that renders only on positive evidence; every other signal said success.

Config outlives the code that gave it meaning. An env var set for a good reason months ago became dead config the moment the API migration shipped, and nothing failed loudly enough to notice.

Filming a product is a test suite. Recording the demo end-to-end against production found a live user-facing bug, a false API verdict, and a silent English-passthrough on translation failure - none of which the unit tests caught, because each one is a degradation, not an exception.

What's next for StoryFlow

Guard the translation path so a provider failure can never emit source-language text labelled as translated. Publish the review step as a first-class API so agents can submit a recording and approve narration programmatically - the async endpoints and signed webhooks already exist. Then per-speaker separation for multi-presenter recordings, and user voice cloning, which today is a single role-gated voice rather than a real feature.

Built With

Share this project:

Updates