Overview
Streamers produce hours of live content, but audiences grow on short-form platforms (TikTok / Instagram Reels / YouTube Shorts). Manually reviewing a 2-hour VOD to cut several vertical clips + editing them for engagement takes creators half a day.
Viralflow AI ingests a livestream VOD along with its platform event log (chat) and automatically produces ranked, captioned, 9:16 highlight clips with platform-ready titles and hashtags β in minutes. The best part is: the creator is in control. Viralflow includes a built-in intuitive video editor that allows creators to review AI's automatic edits, and add their own personal touch to it.
The core idea: a highlight is a cross-modal agreement event β the chat erupts, the audio peaks, and the scene reacts at the same moment. We fuse engagement, audio/speech, and visual signals into one excitement curve, then a Bedrock "AI Director" turns statistical peaks into narrative clips (setup β payoff) with publish-ready metadata in δΈζ + English.
The journey, end to end:
flowchart LR
U[Upload VOD + chat log] --> P[Highlight detection<br/>AI processing<br/>signals Β· fusion Β· Director]
P --> H[Highlights gallery<br/>ranked 9:16 clips]
H --> E[Refine in editor<br/>AI auto-edit + agent]
H --> C["`**Create a compilation reel**<br/>(Highlight groups, ranked)`"]
C --> E
E --> X[Copy metadata Β· export]
π₯ Demo
βΆ Walkthrough video: (talked over the silent video during presentation/pitch)
Demonstrates:
- AI highlight detection with real-time process updates through 3 data modalities (live stream chat analysis, audio analysis, and video analysis)
- Highlight carousel for selecting detected highlights
- Built-in video editor, including AI auto-editing, auto-captioning, and seamless exporting
Screenshots
| Upload VODs Screen | Transparent AI Processing Screen |
 |
 |
| Highlights Gallery | Per-platform Metadata |
 |
 |
β¨ Features
- Multimodal highlight detection β chat engagement, audio loudness/onset, speech (Transcribe), and visual signals (Rekognition) fused on a common 5-second grid, with cross-modal validation (a candidate survives only if β₯2 modalities spike together).
- Bedrock "AI Director" β judges each candidate with transcript + chat + keyframes, extends boundaries to capture the narrative arc, and assigns a virality score + mood label.
- Face-guided 9:16 smart crop β 16:9 β vertical, crop window panned to the median face position from Rekognition face tracking.
- Burned-in captions β zh/en karaoke captions from Transcribe word timestamps (ASS subtitle pipeline), plus a β€12-char hook overlay on the opening seconds.
- Per-platform metadata β title, hook, caption, and hashtags rewritten per platform (TikTok / Reels / Shorts), one-click copy.
- AI auto-edit engine β a per-clip editing pass that auto-zooms on the reacting speaker during laughter/loud moments, flashes split-second onomatopoeia captions ("WHAT?!", "NO WAY") on vocal accents, and drops mood-matched SFX. A Bedrock (Claude) brain plans the edit; a deterministic scipy detector is the offline fallback. Every edit lands as an editable timeline element, not a baked-in overlay. (details)
- Compilation reels β auto-edit a themed group of clips into one multi-clip reel with transitions and per-clip emphasis matched to the reel's vibe (punchy whip-pans for "hype", gentle crossfades for "emotional").
- Agentic in-browser editor β a vendored OpenCut fork (Next.js + Rust/WASM GPU renderer) with an autonomous editing agent: describe a change in natural language and a Bedrock tool-calling loop trims, retimes, splits, adds captions/effects, and more across the timeline β plus one-click AI auto-edit and auto-caption.
- Bilingual UI β Traditional Chinese / English toggle across the app.
- Ranked output β clips sorted by virality score so creators publish the best first.
π How it works
Three modules turn a raw VOD + chat log into publish-ready clips:
1 Β· Intelligent content analysis (multimodal fusion)
- Engagement signal (unique data advantage): parse the platform event log β chat rate z-score, unique chatters, join surges, laughter/slang bursts (εεε, η¬ζ», 666), VIP-weighted message value (a whale reacting β a lurker spamming), and template-spam suppression (novelty weighting kills copy-paste fan promos).
- Audio + speech: Amazon Transcribe (zh-TW, word-level timestamps, speaker labels) for what was said; ffmpeg RMS loudness + onset detection for how loudly (screams, crowd pops, song climaxes).
- Visual: Rekognition Video shot-segment detection (clean cut points) + face detection with emotions (SURPRISED/HAPPY spikes; boxes reused for smart cropping).
- Fusion: all signals z-normalized on a 5s grid, combined with per-vertical weights (talk-show vs gaming presets); cross-modal validation + the Bedrock AI Director confirm clip-worthiness.
Multimodal fusion β how four signals become one excitement curve:
flowchart LR
S1[Chat engagement] --> Z
S2[Speech Β· Transcribe] --> Z
S3[Visual Β· Rekognition<br/>shots + faces] --> Z
S4[Audio RMS Β· onsets] --> Z
Z[z-normalize on 5s grid<br/>per-vertical weights] --> Curve[Excitement curve]
Curve --> V{Cross-modal validation<br/>β₯2 modalities spike?}
V -- yes --> Cand[Candidates] --> Dir[Bedrock AI Director<br/>score Β· boundaries Β· mood]
V -- no --> Drop[Discard]
Real output from a real 74-minute stream (Lang Live Stream ID 3654414) β the fused curve with the 11 kept highlight windows shaded and their peaks marked. Regenerate with python3 scripts/make_excitement_curve.py <streamId>.
2 Β· AI automatic editing engine
- Boundaries chosen for setup β payoff, snapped near Rekognition shot cuts.
- Face-guided 16:9 β 9:16 crop; burned-in zh captions from Transcribe timestamps; hook overlay on the first ~2.5s.
- A further AI auto-edit pass layers reaction zooms, onomatopoeia captions, and mood-matched SFX β see AI editing below.
3 Β· Multi-style production (platform adaptation)
- Per-clip Bedrock metadata pack (title, hook, caption, hashtags) in Traditional Chinese + English; platform presets for length, caption style, and hook placement.
π Algorithm deep dive:
docs/algorithm-decisions.mdβ fast vs. full visual mode, fusion-weight choices, cross-modal validation, and every algorithmic tradeoff (with the ideas deliberately set aside and why).
π¬ AI editing (auto-edit, reels & agentic editor)
Detection and cropping produce a clean clip. Then a second AI layer makes it feel edited. All three subsystems below emit the same EDL contract (docs/contracts/edl.schema.json) that the in-browser editor renders, so nothing is ever baked in irreversibly.
1 Β· AI auto-edit engine (pipeline/autoedit.py, pipeline/autoedit_llm.py)
After a highlight is cut, an editing pass adds "make it more viral" beats:
- Reaction zoom β punches in on the reacting speaker (Rekognition face box) during laughter or a loud-audio spike.
- Onomatopoeia flash captions β split-second "WHAT?!" / "NO WAY" overlays on vocal accents/emphasis.
- Mood-matched SFX β e.g. crickets after an awkward silence β chosen from the clip's vibe.
Two-tier brain (mirrors the AI Director): a Bedrock (Claude) call reasons over the clip's multimodal context (audio-energy peaks, chat laughter, face boxes) and emits a sequenced, timestamped EDL; a deterministic scipy detector (find_peaks) is the offline fallback, so the loop stays runnable without AWS. Every emitted edit is materialized as an ordinary, editable timeline element β adjustable, deletable, or extendable by the user afterward.
Auto-edit engine β two-tier brain with an offline fallback:
flowchart LR
Clip[Highlight clip + signals] --> Ctx[Build clip context<br/>energy peaks Β· chat laughs Β· face boxes]
Ctx --> Q{AWS configured?}
Q -- yes --> Bed[Bedrock EDL planner]
Q -- no --> Sci[Deterministic scipy detectors]
Bed --> Val[Validate vs EDL schema]
Sci --> Val
Val --> TL[Editable timeline elements]
TL --> R[Render 9:16 clip]
π¬ Demo: Auto-editing
2 Β· Compilation reels (pipeline/compile_edl.py, pipeline/compilations.py)
Turn a themed group of highlights into one multi-clip reel on a single timeline: a segment per clip laid end to end, transitions between them, and light per-clip emphasis (opening hook + reaction zooms) chosen to match the reel's dominant vibe β a "hype" reel gets punchy whip-pan cuts and reaction zooms; an "emotional" one gets gentle crossfades and fades. Same two-tier planner (Bedrock brain + deterministic vibe planner fallback) and same EDL contract as the single-clip engine.
Compilation reel β one timeline, vibe-matched:
flowchart LR
G[Themed clip group] --> Vibe[Detect reel vibe<br/>dominant mood]
Vibe --> Q{AWS configured?}
Q -- yes --> Bed[Bedrock reel planner]
Q -- no --> Det[Deterministic vibe planner]
Bed --> EDL[Multi-clip EDL<br/>segments Β· transitions Β· emphasis]
Det --> EDL
EDL --> Ed[Open in editor]
3 Β· Agentic in-browser editor (apps/editor/)
A vendored fork of OpenCut (Next.js + a Rust/WASM GPU effects renderer), wired to this project's clips + auto-edit EDLs via a thin "highlight-api" backend. Beyond one-click AI auto-edit and auto-caption, it runs an autonomous editing agent: you describe a change in natural language and a Bedrock tool-calling loop executes it against the real timeline, with 15 editing tools β
trim_element, retime_element, split_element, move_element, delete_elements, duplicate_elements, add_track, insert_text_element, add_blur_effect, remove_clip_effect, toggle_clip_effect, toggle_track_mute, toggle_track_visibility, toggle_elements_muted, toggle_elements_visibility.
Agentic editor β the tool-calling loop:
sequenceDiagram
actor User
participant Runner as agent-runner (browser)
participant API as agent-turn route
participant Bedrock
participant TL as Editor timeline
User->>Runner: natural-language edit request
loop until the agent signals done
Runner->>API: message history + prior tool results
API->>Bedrock: Converse with 15-tool catalog
Bedrock-->>API: toolUse, e.g. trim_element
API-->>Runner: tool call
Runner->>TL: apply tool
TL-->>Runner: toolResult
end
Runner-->>User: edited timeline
ποΈ GIF placeholder β agent editing the timeline from a prompt Β· add
docs/media/agent-edit.gif.
π Architecture
The demo runs local-first: a Node server (frontend/local-server/) orchestrates the real Python pipeline (python -m pipeline.run) against AWS AI services, or serves pre-rendered results for a zero-latency demo. A dormant AWS CDK stack (backend/) is included as the cloud-scale design β the "scales to cloud" story β and is not required to run the project.
Runtime topology (local-first):
flowchart LR
B[Browser SPA<br/>React + Vite] -->|REST + WebSocket| LS[local-server Β· Node]
LS -->|spawn| PY[pipeline.run<br/>Transcribe Β· Rekognition Β· Bedrock Β· S3]
LS -->|or serve| Cache[(cached out/ runs)]
PY --> Cache
LS -. dormant .-> AWS[AWS CDK<br/>Step Functions Β· Lambda Β· MediaConvert]
The data-flow / signal pipeline itself:
flowchart LR
A[VOD + chat log] --> B[Signal extraction]
B --> C1[Chat engagement]
B --> C2[Transcribe: speech]
B --> C3[Rekognition: shots + faces]
B --> C4[Audio RMS / onsets]
C1 & C2 & C3 & C4 --> D[Multimodal fusion<br/>5s grid Β· cross-modal validation]
D --> E[Bedrock AI Director<br/>score Β· boundaries Β· metadata]
E --> F[Render: 9:16 crop Β· captions Β· hooks]
F --> G[Ranked highlight clips + reels]
π Performance
Measured on a real 74-minute idol-show VOD:
- Speed: full-VOD visual analysis β 29 min; fast mode β 7.5 min β candidates detected from chat+audio+speech first, Rekognition face jobs run only on candidate windows, overlapped with the Bedrock Director.
- Cost: β $5 per VOD in fast mode (Transcribe β $1.8, Rekognition β $1, Bedrock β $0.3, render β $1) vs. hours of editor time.
- Precision: cross-modal validation eliminated 100% of chat-template-spam false positives on the demo VOD.
π§° Tech stack
| Layer | Tech |
|---|---|
| Frontend | React 19, TypeScript, Vite, React Router, i18n (zh-TW / en) |
| Local backend | Node.js server (frontend/local-server/) β runs the pipeline as a subprocess, streams progress over WebSocket, serves media |
| Pipeline | Python 3.12 Β· boto3 Β· numpy Β· scipy Β· pandas Β· ffmpeg (with libass) |
| Cloud AI | AWS Transcribe Β· Rekognition Β· Bedrock Β· S3 |
| Editor | Vendored OpenCut fork (Next.js + Rust/WASM GPU renderer) with an AI editing agent (Bedrock tool-calling) |
| AI auto-edit | Bedrock (Claude) EDL planner + deterministic scipy fallback (pipeline/autoedit*.py, pipeline/compile_edl.py) |
| Deploy | Docker Β· Fly.io (single-container cached demo) |
| Cloud design | AWS CDK (Step Functions / Lambda / MediaConvert β dormant) |
π Getting started
Prerequisites
- Python 3.12 with a repo
.venvandpipeline/requirements.txtinstalled - ffmpeg with libass + freetype (use the
homebrew-ffmpeg/ffmpegtap, not slim homebrew-core) - AWS credentials with access to Transcribe, Rekognition, Bedrock, and S3
Run locally
# 1. Python pipeline deps
python3 -m venv .venv && source .venv/bin/activate
pip install -r pipeline/requirements.txt
# 2. Frontend env
cd frontend
cp .env.example .env # only needed for the cloud build; the demo uses .env.demo
npm install
# 3. Start the app (local server + Vite dev server)
npm run dev # http://localhost:5173
# npm run dev:all # also launches the OpenCut editor
Run the pipeline directly:
python3 -m pipeline.run --stream-id <id>
Deploy (single-container cached demo)
A Dockerfile + fly.toml package the app as a one-container cached demo (no Python/ffmpeg/AWS needed at runtime β it replays a pre-rendered run). See docs/ for details. Deploy with fly deploy; run a single machine (job state is in-process).
π Repository structure
frontend/ React SPA + local-server/ (the real local-first backend)
pipeline/ Python highlight pipeline (signals, fusion, director, render)
apps/editor/ Vendored OpenCut editor (Next.js) for manual refinement
backend/ Dormant AWS CDK stack β the cloud-scale design
.kiro/ Design specs & steering docs
docs/ Additional documentation + media
πΊ Future work
- Live ingest (IVS / MediaLive HLS) for near-real-time clipping over rolling windows
- Additional platform presets and languages
- Hosted editor + one-click publish per platform
- Creator-tunable weight presets per content vertical
π₯ Team
The Waymakers β Jason Nishio Β· Nelson Nishio Β· Aaron Lin
π Awards
π Hackathon Winner
π₯ 3rd Prize - $10,000 NTD - AWS Summit Taipei AI Everywhere Hackathon
Built With
- css
- dockerfile
- html
- javascript
- powershell
- python
- rust
- shell
- typescript
- wgsl
Log in or sign up for Devpost to join the conversation.