ScenePulse

Autonomous Multimodal Video Frame & Audio Cultural Taste Ad Engine — Powered by Qloo Cultural AI

Comment

Inspiration

Standard video ads interrupt viewers and rely on shallow object matching ("see a coffee cup → show a generic coffee ad"). Even when no commercial product appears on screen, every scene carries a distinct visual atmosphere, architectural style, and acoustic mood.

We asked: What if pausing a video didn't trigger an annoying pop-up, but instead awakened an autonomous Cultural AI Agent that captures the exact video frame, analyzes the last 10 seconds of audio, and dynamically queries Qloo's Taste Graph across 9 cultural domains?

That idea became ScenePulse.

What it does

ScenePulse turns any paused video moment into a real-time, cross-domain cultural discovery and contextual commerce experience:

  1. Real-Time Frame & 10s Audio Capture (<canvas> + Audio Telemetry):
    • When a viewer pauses any of the built-in streams (or uploads a custom .mp4/.webm video), ScenePulse captures the exact paused video frame via HTML5 <canvas> alongside the 10-second acoustic profile.
  2. Schema-Aware Agentic URN Planner (9 Qloo Domains):
    • Instead of using hardcoded recommendation templates, our Multimodal LLM Agent inspects the scene signals and dynamically selects the 4 most contextually relevant Qloo /v2/insights URNs from the 9 supported domains (urn:entity:brand, urn:entity:place, urn:entity:destination, urn:entity:artist, urn:entity:movie, urn:entity:book, urn:entity:tv_show, urn:entity:podcast, urn:entity:videogame) plus dynamic city routing (filter.location.query) for physical venues.
  3. Live Two-Stage Qloo Taste AI Pipeline (/search → /v2/insights):
    • Stage 1 (GET /search): Resolves extracted cultural seeds (e.g., Blue Bottle Coffee, Espresso, Kyoto Tower, Andy McKee, Taylor Guitars) into official Qloo Entity UUIDs.
    • Stage 2 (GET /v2/insights): Queries Qloo's Taste Graph using signal.interests.entities across the 4 agent-selected URNs to return verified cross-domain matches, affinity scores, imagery, and descriptions.
  4. 3-Screen Interactive Verification Studio:
    • 01 · Scene Showcase: 3 real short-form video streams with live Qloo seed UUIDs and cross-domain insight feeds.
    • 02 · Live Frame Capture & Proof: Interactive video player with floating in-video HUD pins, 4 aligned Qloo insight cards, captured JPEG frame preview, full UUID/affinity causality, JSON proof export, and a Live Qloo API Explorer.
    • 03 · How It Works: Built-in Remotion video walkthrough and full 9-URN Qloo API orchestration specification.

How we built it

  • Backend Orchestration (Laravel 11 + PHP 8.3):
    • QlooTasteService: Full-schema client for https://hackathon.api.qloo.com supporting GET /search seed resolution and all 9 GET /v2/insights entity URNs with location-aware filtering.
    • SceneMultimodalService (High-Velocity NVIDIA GPU Inference via Nebius): Because a pause-to-discovery experience demands instant responsiveness, we selected Nebius AI Studio's NVIDIA GPU infrastructure specifically for its high-throughput, low-latency token generation. We pair google/gemma-3-27b-it (27B Vision)—which excels at recognizing fine-grained visual composition, lighting, and objects in <canvas> video frames—with deepseek-ai/DeepSeek-V4.1-Flash / Qwen/Qwen3-30B-A3B-Instruct-2507 for 9-URN Qloo schema planning (plus seamless OpenRouter support). Because Qloo Taste AI handles the deep cross-domain cultural correlation, each full multimodal scene analysis completes rapidly at ~$0.0003 per frame.
  • 5-Second Time-Bucket SQLite Cache (scene_pulse_impulses):
    • Video timelines are indexed into 5-second buckets (floor(t / 5) * 5) so public visitors and judges get instant verified responses with zero login friction, while the "Capture Frame & Run Live Qloo API" button executes a fresh end-to-end multimodal + Qloo pipeline on demand.
  • Frontend & Infrastructure (Vanilla ES6 + HTML5 Canvas + HTTP 206 Range Proxy + Docker):
    • Lightweight, zero-bloat studio with partial-content video streaming (HTTP 206) for instant seeking and <canvas> frame extraction, containerized via Docker Compose.

Challenges we ran into

  1. Translating Raw Pixels & Audio into Validated Qloo Entity UUIDs:
    • Vision models naturally output descriptive adjectives ("warm lighting, wooden guitar body"), whereas Qloo /v2/insights requires concrete entity UUIDs. Building a two-stage agent pipeline that first extracts canonical cultural entities, resolves them via GET /search, and feeds verified UUIDs into GET /v2/insights solved this cleanly.
  2. Dynamic Context-Appropriate Domain Selection:
    • A specialty espresso ritual calls for brands, cafes, acoustic artists, and books, while a Kyoto night skyline calls for destinations, local venues, cinema, and design brands. Teaching the LLM planner the full 9-URN Qloo schema ensured every scene receives domain categories that genuinely fit its aesthetic.
  3. CORS-Free Frame Extraction & Fast Seeking:
    • Extracting base64 JPEG frames on pause required implementing proper HTTP 206 Partial Content byte-range streaming in our frontend server so <canvas> snapshots work instantaneously at any timestamp.

Accomplishments that we're proud of

  • 100% Real Qloo Graph Provenance: Every recommendation card surfaces its live Qloo Entity UUID, affinity percentage, URN subtype, and the exact /search seed UUIDs that produced it.
  • Schema-Aware Agentic Routing: Dynamically planning 4 distinct Qloo /v2/insights endpoints per scene rather than relying on static categories.
  • Production Unit Economics (~$0.0003 per Scene Analysis): Demonstrating that pairing lightweight open-weight Vision and 3B-active-parameter MoE models (Gemma-3-27B-IT + Qwen3-30B-A3B) with Qloo's deterministic Taste Graph delivers sub-second cultural precision without expensive frontier LLM token costs.
  • Instant Verification UX: Combining <canvas> live frame capture, custom video uploads, a zero-token Live Qloo API Explorer, and an embedded Remotion architectural walkthrough in one cohesive interface.

What we learned

  • Qloo's Taste Graph Eliminates the Need for Heavyweight LLMs: Items that share zero visual keywords—such as percussive fingerstyle guitar, luthiery brands, and intimate listening venues—share deep cultural affinity in Qloo's Taste Graph. Offloading cultural correlation to Qloo means the LLM only needs to perform fast visual/acoustic extraction and schema routing, allowing small, ultra-fast open-weight models to outperform giant standalone LLMs at a fraction of the cost.
  • Pause Moments Are High-Intent Discovery Windows: Surfacing non-intrusive, culturally aligned pins the moment a viewer pauses turns passive viewing into active exploration.

What's next for ScenePulse

  • Drop-In Player SDK (@scenepulse/player-sdk): Packaging our <canvas> pause-capture and Qloo insight overlay as a lightweight plugin for Video.js, Shaka Player, and web streaming platforms.
  • Live Stream & HLS Segment Indexing: Extending the 5-second time-bucket engine to continuous live broadcasts for real-time cultural discovery during music sessions, travel streams, and live events.

Built With

Share this project:

Updates

Submission history