ClipCraft

Inspiration https://vimeo.com/1142309034 Mobile creators today generate more short-form video content than ever—but editing requires cloud services, expensive subscriptions, or powerful desktops. We wanted to break that dependence. Our goal: enable creators to edit, enhance, and repurpose videos entirely on-device, using efficient AI models optimized for Arm processors. ClipCraft was born from the idea that creativity shouldn’t need a server—AI should run instantly, privately, and anywhere. What It Does ClipCraft is an on-device AI clip editing studio that transforms raw video into polished, share-ready content in seconds. Key features: AI Auto-Highlight Detection – Finds the most engaging segments using local action + scene recognition. Smart Clip Reframing – Tracks subjects and auto-adjusts framing for vertical, horizontal, and square formats. On-Device Voice Cleanup – Removes noise and enhances speech using Arm-tuned audio models. AI Captioning (Offline) – Generates accurate subtitles in the user’s language with a lightweight Whisper variant. Style Filters with Neural Effects – Applies cinematic color grading and motion-aware enhancement. ClipCraft Templates – Prebuilt sequences (intros, transitions, outro cards) generated locally. Everything runs fully offline on mobile devices powered by Arm architecture, ensuring: Speed Privacy Battery efficiency Zero dependency on cloud compute How We Built It ClipCraft is engineered around a fully Arm-optimized AI pipeline:

1. Models Highlight Detection → MobileNet + custom temporal transformer Face/Subject Tracking → BlazeFace + lightweight DeepSort Audio Enhancement → RNNoise variant compiled with NEON Subtitle/ASR → Whisper-Tiny (quantized with Arm NN and TFLite) Stylization → Fast Style Transfer (int8) Super-Resolution → ESRGAN-lite (NEON optimized)
2. Optimization Hardware-aware quantization for Arm Cortex-A and Hexagon DSP Use of Arm Compute Library for fast matrix ops Multi-threaded video IO aligned with big.LITTLE CPU clusters Model caching to reduce power usage On-device GPU acceleration where supported (Mali, Adreno)
3. App Architecture Flutter + Rust backend ML inference handled through a shared Rust/FFI module Zero-copy frame processing pipeline Offline-first data design Challenges We Ran Into Achieving real-time inference without overheating on mid-range devices Memory constraints when running multiple models (tracking + ASR + stylization) Balancing UI simplicity with advanced editing controls Ensuring video frames remained zero-copy to avoid latency Tuning quantized models while preserving quality Accomplishments We’re Proud Of Reaching 19–21 FPS real-time AI inference on a mid-range Arm device Running the entire editing workflow offline, including speech-to-text Building a fully modular pipeline that other developers can reuse Delivering a UI that feels simple enough for casual creators Producing truly high-quality highlight reels without cloud help What We Learned Arm devices are capable of far more advanced on-device AI than expected Model quantization and NEON vectorization are essential for mobile performance Efficient video memory management often matters more than model speed Users care deeply about privacy—offline AI is a major trust advantage Flutter + Rust is a powerful pattern for cross-platform mobile AI apps What's Next for ClipCraft AI Storyboard Generator – Auto-creates video sequences using text prompts On-Device Lip Sync & Face Retargeting (experimental) Edge-to-Edge Multi-Clip Editing with GPU-accelerated transitions Creator Packs → Custom LUTs, filters, templates distributed locally Community Model Slot → Load your own TFLite/ONNX model ClipCraft Studio for tablets → Larger UI, multi-track timeline ClipCraft will evolve into the fastest, most private mobile AI editing studio, powered entirely by on-device Arm intelligence.

Built With

• all