Frame Talk

Transform silent app demos into broadcast-ready video podcasts with dynamic frame holds and zero drift, powered by Gemini 3.7 Flash, Google Cloud ADK, and ClickHouse.

Comment

Inspiration

Developer screencasts and silent software walkthroughs are ubiquitous, but watching a silent 3-minute video while cross-referencing a dense README is friction-heavy and uninspiring. While tools like NotebookLM can generate audio summaries from text transcripts, they are completely blind to video pixels—they cannot visually observe UI transitions, terminal outputs, or button clicks.

Worse, traditional video creation tools suffer from severe audio-video timing drift: whenever AI narrators spend extra time explaining a complex architecture or terminal output, the audio falls seconds behind the screen—resulting in hosts talking about past screens or previewing future screens before they appear.

We built Frame Talk to solve this: pairing Google Gemini 3.7 Flash visual video comprehension with a proprietary Chronos Sync Engine that dynamically freezes video frames when explanations go deep—guaranteeing 100% video-audio lockstep with zero drift.

What it does

Frame Talk is an autonomous multimodal studio that ingests a silent screencast (.mp4) alongside repository documentation (README.md) to generate an engaging, broadcast-ready technical video podcast hosted by two AI presenters (Mark & Sarah):

  1. Native Pixel-Level Scene Comprehension: Ingests raw video pixels directly into Gemini 3.7 Flash via the Google GenAI File API, detecting on-screen UI actions, terminal logs, and code edits without transcripts.
  2. Autonomous Multi-Agent Dialogue & QA Audit: Orchestrated via the Google Cloud Agent Development Kit (ADK v2.8.0), our Scriptwriter and QA Auditor agents craft natural developer banter with a high-level product introductory hook and zero robotic timestamps.
  3. Chronos Sync & Dynamic Frame Holds: Synthesizes uncompressed 24 kHz 16-bit Mono PCM speech via gemini-3.1-flash-tts-preview, meters spoken duration down to the millisecond, and dynamically stretches the video timeline with exact frame holds during deep technical explanations.
  4. 1-Click Autopilot (YOLO Mode): Allows users to either inspect each step in human-in-the-loop mode or run the entire 5-step pipeline end-to-end with one click to production MP4 download.
  5. Production Observability: High-throughput telemetry streamed to ClickHouse and monitored in real time on Grafana Labs.

How we built it

  • Google Cloud AI Stack:
    • Google Cloud Agent Development Kit (google-adk v2.8.0): Autonomous FrameTalk_Director orchestrator with sub-agent delegation (ScriptwriterPersonaAgent, QAPacingAuditorAgent).
    • gemini-3.7-flash: Native multimodal video token execution analyzing temporal UI actions directly from raw video pixels.
    • gemini-3.1-flash-tts-preview: Dual-speaker raw PCM synthesis (Puck for Mark, Kore for Sarah) with exact duration metering.
    • Google Prompt Caching: Automatic extraction of cached tokens applying Google Cloud's 75% discount ($0.0375 / 1M cached tokens).
  • Model Context Protocol (MCP):
    • Custom Chronos MCP server (/mcp) exposing stream_sync_events, get_pipeline_metrics, and inspect_scene_drift tools over SSE and JSON-RPC.
  • Partner Track (ClickHouse + Grafana Labs):
    • Columnar time-series logging across castops.sync_events, castops.llm_calls, and castops.user_activity via clickhouse-connect and the official mcp-clickhouse integration.
    • Hosted Grafana dashboards tracking real-time timeline drift, multi-model token telemetry, prompt cache savings, and trace logs.
  • Chronos Precision Mathematics: Audio duration is measured directly from 24 kHz 16-bit mono PCM bytes: $$\text{duration_ms} = \frac{\text{pcm_bytes}}{48}$$ When spoken commentary exceeds the visual scene duration, Chronos injects an exact hold frame at 70% scene depth: $$\text{required_freeze_ms} = \max(0, \text{total_speech_ms} - \text{scene_duration_ms} + 300)$$
  • Media Engine & Full-Stack:
    • Fast FFmpeg-powered video stitcher and 1080p compiler.
    • FastAPI (Python 3.12+) backend with cryptographic HMAC-SHA256 user tokens and Sybil attack prevention.
    • Lightweight frontend using Vanilla JS/CSS and the Bricolage Grotesque design system.

Challenges we ran into

  1. Solving Timeline Drift without Robotic Silence: Traditional AI dubbing forces dialogue into arbitrary video cuts by inserting awkward 5–10s silence buffers. We overcame this by building the Chronos Sync Engine to dynamically stretch video frames rather than compressing or chopping speech.
  2. Preventing Prompt Injection from Untrusted Documentation: Ingesting raw external README files exposes LLMs to indirect prompt injection. We engineered hardened XML isolation boundaries (<untrusted_documentation>) and strict refusal gates (ACCESS DENIED) to prevent jailbreak attempts.
  3. High-Throughput Telemetry in Async Pipelines: Capturing token usage, latencies, and video frame hold events concurrently without slowing down media compilation. We implemented an asynchronous ClickHouse event logger with in-memory fallback ring buffers.

Accomplishments that we're proud of

  • 100% Automated Test Suite: 56 out of 56 unit and integration tests passing cleanly across security guardrails, Chronos math, quota service, and API endpoints.
  • 1-Click Autopilot Mode: A frictionless "YOLO" execution path that auto-cascades through all 5 multimodal stages while visibly showing each agent's cards and scores.
  • Sub-Dollar Production Cost: Full video analysis and two-host podcast compilation costs under $0.15 USD per screencast, with prompt caching saving up to 75%.
  • Zero-PII Observability: GDPR-compliant user analytics using pseudonymized cryptographic hashes (X-FrameTalk-User-Hash).

What we learned

  • Gemini 3.7 Flash's native video token processing eliminates the need for expensive external speech-to-text transcriptions or OCR pipelines.
  • Building natively with the Google Cloud Agent Development Kit (ADK) provides an exceptionally clean foundation for multi-agent tool execution and Model Context Protocol (MCP) integration.

What's next for Frame Talk

  • Multi-Language Dubbing: Automatic multi-speaker translation with localized voice personas.
  • GitHub Release Bot: A GitHub Action that automatically produces a 60-second video podcast summary whenever a new software release is tagged.

Built With

  • adk
  • agent
  • antigravity
  • clickhouse
  • gemini.3.7-flash
  • github
  • grafana
  • python
Share this project:

Updates

Submission history