Clippy

Inspiration

As a video editor, software engineer, and designer, my niche is creating visually stunning edits. But I quickly realized a massive problem: post-production grunt work completely kills the creative flow. Sifting through hours of raw footage to find one specific scene, manually hunting for viral hooks, and pacing out b-roll takes hours of tedious timeline dragging. I wanted to build my own "JARVIS" for video editing—an assistant that handles the tedious assembly so I can spend all my energy on the actual art of storytelling.

What it does

Clippy is a multi-modal AI post-production agent that edits for you. Instead of scrubbing through footage manually, you just chat with the agent in a custom Streamlit UI.

Semantic Scene Search: Give it a long video and ask it to find a specific action (e.g., "find the backflip"). It mathematically pinpoints the exact frame and drops it into a CapCut timeline.

Viral Hook Extraction: It transcribes audio, analyzes it for high-retention emotional peaks, slices out the best 15-to-60-second hooks, and auto-generates subtitles.

Generative B-Roll: Ask for a specific cinematic shot, and it will generate a 6-second video and inject it directly into your CapCut draft.

How we built it

Clippy was built for the Agents for Humans Hackathon using Python and the AWS Strands SDK.

The Brains: We used Amazon Nova Pro via AWS Bedrock to orchestrate tool calling and analyze transcripts for viral retention triggers.

The Senses: Amazon Nova Multimodal Embeddings handles the frame-by-frame cosine similarity for scene searching, while Claude 3.5 Sonnet evaluates visual frame quality. OpenAI Whisper runs locally for precise, cost-effective audio transcription.

The Hands: We used Amazon Nova Reel to generate b-roll in the cloud, and the pyCapCut library to programmatically assemble the JSON draft files that CapCut desktop reads.

The Interface: Everything is wrapped in a custom, Gemini-inspired emerald-themed Streamlit GUI with persistent, file-based chat memory and dynamic visual staging.

Challenges we ran into

Bridging cloud AI with local desktop software was tricky. Aligning the exact local file paths between the Python environment and CapCut's strict JSON draft structure required extensive debugging. Additionally, managing persistent state in Streamlit—specifically ensuring chat histories didn't crash the app when switching between sessions or encountering corrupted file saves—required us to build strict type-checking and callback-driven navigation into the UI.

Accomplishments that we're proud of

Bridging Cloud AI with Local Software: Successfully engineering a pipeline that connects cloud-based foundation models via AWS Bedrock directly to a local desktop application by programmatically manipulating CapCut's JSON draft files.

Resilient UI State Management: Building a sleek, Gemini-inspired Streamlit interface that features bulletproof state management, persistent local chat histories, and a callback-driven navigation system capable of auto-recovering corrupted files.

Multi-Model Orchestration: Seamlessly chaining together five different models—Amazon Nova Pro, Nova Reel, Nova Multimodal Embeddings, Claude 3.5 Sonnet, and local OpenAI Whisper—into one cohesive, autonomous workflow.

Solving a Real Problem: Merging a computer engineering background with a passion for video editing to move beyond a flashy tech demo, building a highly functional utility that genuinely eliminates hours of tedious post-production grunt work.

What we learned

I deepened my understanding of agentic tool-calling workflows and how to chain multi-modal foundation models together. Relying on an LLM not just for text, but to execute physical file manipulation and video rendering pipelines, completely shifted how I view software engineering. I also leveled up my Streamlit skills, moving beyond basic scripts into robust, state-managed applications.

What's next for Clippy

Expanding the agent's toolset to handle automated color grading logic, beat-syncing cuts to audio tracks, and integrating multi-track audio mixing directly from the prompt interface.

Built With

• ai-agents
• amazon-nova-embeddings
• amazon-nova-pro
• amazon-nova-reel
• aws-bedrock
• aws-strands-sdk
• capcut
• claude-3.5-sonnet
• machine-learning
• openai-whisper
• pycapcut
• python
• streamlit