Frames of Us (FOU)

client-side AI storyteller that turns videos, voices, and photos of families, friends, reunions, and teams into cinematic, privacy-preserved stories — instantly and offline.

Comment

Inspiration

Every clip we capture holds a story — but most stay forgotten in old folders and drives. When we found Chrome’s built-in AI could process audio, video, and images locally, we saw a chance to give those moments new life. Frames of Us (FOU) was born from a single family video — a dropped cake, a laugh, and a sunset — and grew into a tool to celebrate togetherness in every form: 👨‍👩‍👧 Families, 👩‍💻 Office teams, 🎓 College reunions, 🏫 Alumni meets, 🎉 Community events. Our goal: make storytelling personal, private, and cinematic — without the cloud.

What it does

Frames of Us (FOU) is a client-side multimodal AI storyteller. Users can upload or drag-and-drop: 🎥 Old videos (like family_vacation.mp4) 🎧 Voice notes (e.g., “Remember when Amma dropped the cake…”) 🖼️ Images (e.g., beach_closeup.jpg, sunset.jpg) 📝 Text notes (themes or captions) FOU analyzes them with Chrome’s built-in AI suite — Prompt API, Summarizer API, Writer API, Rewriter API, and Proofreader API — and automatically creates: A structured storyblok: scene-wise outline, narration, and emotions A short cinematic video with AI-generated narration and captions Platform-ready exports for YouTube, Instagram, LinkedIn, X, TikTok, and Reels — each with tailored length, caption, hashtags, and aspect ratio Everything happens offline, inside the browser — fast, private, and free.

How we built it

Frontend: React + Vite + Tailwind for the responsive “FOU Studio” interface APIs: Chrome Built-in AI — Prompt, Summarizer, Writer, Rewriter, Proofreader APIs Processing Flow: Extract keyframes + transcripts from uploaded video/audio Use Prompt API to detect mood, characters, and themes Summarizer API creates concise scene summaries Writer API drafts narration, social captions, and CTAs Proofreader and Rewriter polish and adapt tone per platform Output: Storyblok JSON (editable timeline of scenes) Multi-platform video exports (YouTube/LinkedIn/Instagram formats) Auto captions + subtitles + thumbnails Storage: IndexedDB for offline caching; File System Access API for local project saves Rendering: HTML5 + MediaRecorder to create final videos without a server

Challenges we ran into

Handling multimodal synchronization — aligning transcript text, keyframes, and voice memo timestamps precisely. Ensuring AI understanding of emotional tone across different input types (humor vs nostalgia). Building a responsive, cinematic UI that feels like a creative studio but runs entirely in-browser. Keeping large video files performant using client-side memory optimization. Balancing creative control (user edits) with automation speed.

Accomplishments that we're proud of

Created the first client-side multimodal storytelling tool using Chrome’s built-in AI — no cloud required. Generated auto-edited videos from personal archives in under 2 minutes. Produced platform-specific creative outputs (LinkedIn, Instagram, YouTube, X) automatically. Achieved real emotional connection — judges and testers smiled, teared up, and rewatched their own memories. Demonstrated true privacy-by-design storytelling — no upload, no tracking, just pure creation.

What we learned

Chrome’s Gemini Nano + Built-in AI APIs make private creativity possible — you can truly run meaningful AI workloads locally. The Prompt API’s multimodal support can understand context across text, voice, and visuals better than expected. Storyblok structures are powerful — they allow AI outputs to stay editable and transparent, not just "black-box videos.” Emotionally driven UX (color, copy, pacing) matters as much as technical accuracy in human-AI storytelling.

What's next for Frames of Us (FOU)

Group Collaboration Mode: allow multiple users (family, classmates, teams) to co-create storybloks remotely and merge clips together. Memory Timeline: automatically organize videos/photos by date and theme — birthdays, reunions, Milestones. Voice Personalization: let users keep their own voice for narration using local fine-tuning (privacy-safe). Smart Story Prompts: suggest themes like “A Journey Together”, “From Chaos to Celebration”, or “Then & Now”. FOU for Organizations: internal storytelling for company retrospectives, CSR moments, or anniversary reels. Offline Mobile Version: extend to Android (WebView + local Gemini Nano). Open Storyblok API: allow developers and creators to plug in new render templates or storytelling styles.

❤️ “FOU isn’t just about videos. It’s about remembering who we are — together.” ❤️ ❤️ ❤️ “Gemini is the chef of our memories. FOU is the heart. Together, we cook raw moments into timeless stories.”❤️ ❤️ ❤️

FOU System Architecture

INPUTS

🎥 Old Videos (e.g., family_vacation.mp4) 🎧 Voice Notes (e.g., note1.m4a) 🖼️ Images (e.g., beach_closeup.jpg, sunset.jpg) 📝 Text Notes (“Theme: coming-of-age...”)

INGEST & PREPROCESS

▪ File Upload UI / Chrome Extension Capture ▪ Extract transcripts, keyframes, and image metadata ▪ Speech-to-text & OCR ▪ Chunking, timestamp mapping, and normalization

AI PROCESSING (Gemini Nano — Client-Side)

🌐 Prompt API — Multimodal understanding (video + voice + text + image) 📄 Summarizer API — Key ideas, scene outlines ✏️ Writer API — Narration, titles, and social captions 🖊️ Rewriter API — Adjust tone (cinematic, emotional, casual) 🔤 Proofreader API — Grammar, flow, and clarity ➕ Local inference — No cloud, privacy-first

STORYBLOK GENERATOR

▪ Scene detection and segmentation ▪ JSON storyblok (title, scene, mood, narration) ▪ Editable timeline for user refinement ▪ Tagging (emotions, moments, people)

RENDER ENGINE (Client-Side)

▪ HTML5 Canvas + MediaRecorder + WebCodecs ▪ Applies transitions, subtitles, and TTS narration ▪ Generates story preview & exportable video

EXPORT & SOCIAL OUTPUTS

▶ YouTube (Trailer / Short) ▶ Instagram / TikTok (Reels) ▶ LinkedIn (Team Stories / Reunions) ▶ X / Twitter (15s teasers) ▪ Thumbnails, captions, hashtags, subtitles (SRT) ▪ All client-side, privacy-respecting outputs

LOCAL STORAGE & PROJECT SAVE

▪ IndexedDB / File System Access API ▪ Optional encryption for user privacy ▪ No external uploads

Built With

  • chromebuilt-in-ai
  • firebase
  • framermotion
  • gemininano
  • indexeddb
  • promptwriter
  • proofreader
  • react
  • rewriter
  • summarizer
  • tailwindcss
  • vite
  • webcodecs
Share this project:

Updates