Neuro Scene

Real-Time Agentic Multiverse Orchestration - Where audience telemetry instantly dictates the narrative and cinematography of a movie through an AI agent swarm.

Comment

NeuroScene

Real-Time Agentic Multiverse Orchestration & Spatial Intelligence
*Where millions of concurrent audience telemetry votes instantly dictate cinematic narrative, camera optics, and volumetric lighting through a parallel swarm of Google Cloud

💡 Inspiration

For over a century, cinema has been a strictly one-way broadcast. Even modern experiments in "interactive video" (like Bandersnatch or video-game branching) are fundamentally limited: they force a single viewer to make a binary choice between pre-recorded MP4 video clips every ten minutes, while live-streaming audiences on Twitch or YouTube are relegated to chaotic chat boxes that creators cannot possibly parse or act upon in real time.

We asked an ambitious question: What if a live movie was genuinely alive?

What if hundreds of thousands—or millions—of concurrent viewers could collectively steer the emotional direction, moral dilemmas, screenplay dialogue, camera angles, and three-point lighting of an ongoing film in sub-second real-time?

Historically, two architectural bottlenecks prevented this from existing:

  1. The Database Row-Locking Bottleneck: Ingesting 50,000+ continuous emotional votes per second instantly crashes traditional relational databases (PostgreSQL, MySQL) due to row-level ACID write lock contention and connection pool exhaustion.
  2. The AI Generation Latency Bottleneck: Monolithic LLM chains and video generation models take 15 to 60 seconds per prompt, completely destroying the collective momentum and live immersion of an audience.

We built NeuroScene to eliminate both bottlenecks. By marrying the astronomical ingestion and tumbling analytical power of ClickHouse Cloud with a hierarchical, sub-second Gemini 2.5 Pro & Flash Agent Swarm on Google Cloud Run, NeuroScene introduces the world's first real-time agentic multiverse orchestration engine and spatial intelligence platform.

🎬 What It Does

NeuroScene transforms passive viewers into an active collective directorial mind:

1. Massive Audience Telemetry Ingestion (50,000+ EPS)

Viewers in the live auditorium cast continuous emotional telemetry (Dread, Combat, Irony, Empathy), vote on high-stakes character dilemmas, or inject collective sentiment surges. Every vote is captured through an in-memory double-buffered micro-batcher and streamed into ClickHouse Cloud with async_insert=1, completely bypassing row-level locks.

2. Continuous 5-Second Tumbling Aggregation

ClickHouse Cloud continuously aggregates incoming telemetry into a 5-second tumbling Materialized View (scene_sentiment_mv). Instead of running heavy table scans during playback, the system maintains an instantaneous 1-row statistical summary (avg_fear, avg_action, avg_humor, avg_romance, winning_vote) ready for instant retrieval in under 10 milliseconds.

3. Hierarchical Swarm Intelligence (Promise.all() < 480ms)

  • Director Agent (Gemini 2.5 Pro): Reads the 1-row sentiment summary and active storyline state, performing deep dramatic reasoning to determine target tension pacing, emotional stakes, and narrative trajectory.
  • Parallel Specialist Sub-Agents (Gemini 2.5 Flash): Triggered concurrently via Promise.all() to formulate the next cinematic beat in parallel:
    • Script Agent: Writes contextual character dialogue, emotional subtext, and delivery cues aligned with the crowd's chosen dilemma path.
    • Lighting Agent: Computes three-point lighting color theory (key light hex, fill intensity, shadow density, volumetric haze) to evoke the crowd's dominant mood.
    • Camera Agent: Coordinates focal length (mm), camera angle (Dutch tilt, high-angle crane, ground eye-level), framing, and movement characteristics (steadicam sprint, handheld tremor).
    • Acoustic Agent: Generates 3D binaural spatial audio reverberation profiles synced to the camera's spatial coordinates.

4. Four-Tab Multi-Studio Architecture

The client studio provides four specialized command tabs for viewers, critics, and production directors:

  1. Live Cinema & Dilemma: View the live 60 FPS viewport, active dilemma countdown timer, Path A vs Path B voting buttons, and quick emotional expression triggers.
  2. Agent Swarm War Room: Inspect live telemetry breakdowns, Director narrative reasoning logs, and the parallel sub-agent execution waterfall (Script, Lighting, Camera execution timings).
  3. ClickHouse Real-Time Analytics: Real-time columnar schema inspector (scene_sentiment_mv), tumbling window metrics, and a simulated crowd surge injector to stress-test 10,000 concurrent votes.
  4. Audience Community & Live Chat: Real-time audience chat stream with sentiment badges, viewer avatar identification, and collective commentary.

5. Interactive 2D Telemetry Mood Pad

A real-time 2D Cartesian coordinate pad allowing viewers to drag and cast multidimensional emotional vectors across four quadrants:

  • Upper-Left: Dread / Suspense (High Tension, Low Action)
  • Upper-Right: Chaos / Combat (High Tension, High Action)
  • Lower-Left: Calm / Mystery (Low Tension, Low Action)
  • Lower-Right: Hope / Empathy (Low Tension, High Action)

6. Director Manual Override Console

Empowers studio directors to intervene in the automated narrative flow:

  • Atmosphere Lighting Presets: 1-click override between Neon Cyan (#00F0FF), Alert Crimson (#FF1E56), Matrix Emerald (#10B981), and Solar Amber (#F59E0B).
  • Camera Optics Controls: Dynamic lens focal length slider (18mm wide to 85mm portrait telephoto) and handheld camera shake intensity (0% to 100%).
  • Emergency Crisis Trigger: Injects an immediate high-tension plot twist crisis into the swarm state machine.

7. Multiverse Branch Explorer (Track & Nodes Views)

Visualizes narrative divergence across branching storylines with two dedicated viewing modes:

  • Track View: A horizontal timeline powered by Ant Design Steps with custom left/right navigation buttons, non-wrapping beat titles, and click-to-inspect telemetry snapshots.
  • Nodes View: A compact vertical card stack optimized for sidebar inspection without vertical text compression.

8. 429 Quota Circuit Breaker

Built-in automated rate-limit protection (quotaManager.ts) that intercepts HTTP 429 quota exhaustion errors from Vertex AI, gracefully falling back to deterministic procedural generation until the quota window resets.

9. Firebase Authentication Gate

Enforces authentication across all studio actions with support for Google 1-click OAuth, email authentication, and an instant 1-click Guest Critic Mode for frictionless evaluator access.

10. Ultra-Cinematic Cyberpunk-Luxury Landing Page

  • Interactive 3D WebGL/Canvas Neural Point-Cloud: Over 240 particles spinning in 3D coordinate space that dynamically react to mouse velocity (dx/dt, dy/dt), accelerating rotation and drawing glowing bioluminescent synaptic filaments.
  • Split-Text Cinematic Headline: "SYNTHESIZE REALITY FROM PURE THOUGHT" paired with live telemetry HUD tags (LATENT SEED, RENDER ENGINE, 60 FPS LOCKED, 50K+ EPS).
  • Interactive 3D Latent Space Viewport: Orbit-controllable 3D procedural manifold with 4 environment presets (Cybernetic Alley, Deep Sea Biosphere, Hyper-Dimensional Lattice, Neural Synapse Network), dynamic sliders (Density, Entropy, Coherence, Bloom), and an interactive 4-stage generation scrubber (Tokens -> Latent Noise -> Volumetric Mesh -> 4K Photoreal Radiance).
  • Asymmetric Bento Box Feature Grid: Showcasing Neural Radiance Fields, live CUDA NeRF code sandbox, Gemini 2.5 parallel swarm status, ClickHouse Cloud telemetry metrics, and an interactive Web Audio frequency visualizer.
  • Interactive Architecture Pipeline: 4-stage technical walk-through with latency budgets and execution calls.
  • Infinite Multiverse Showcase: Live cinema streams with technical camera metadata (Aperture, Lens, ISO, Shutter, Latent Seed).
  • Studio Tier Matrix: Clear access tiers for Indie Visionaries ($0 / Free Critic), Studio Production, and Autonomous Enterprise.

⚙️ How We Built It

[ Audience Clients (Telemetry, Dilemma Votes, Mood Vectors) ]
                             │
                             ▼  HTTP/2 POST / WebSockets
     [ Google Cloud Run: Telemetry Micro-Batcher ]
     (In-Memory Double-Buffering, 250ms Flush, async_insert=1)
                             │
                             ▼  Native TCP / 8443 TLS
          [ ClickHouse Cloud (Managed Cluster) ]
       ├── default.raw_telemetry (MergeTree Engine, LZ4)
       └── default.scene_sentiment_mv (5s Tumbling Materialized View)
                             │
                             │ 1-Row Statistical Summary (< 10ms)
                             ▼
     [ Google Cloud Run: Multiverse Orchestration Engine ]
                             │
            ┌────────────────┴────────────────┐
            ▼                                 ▼
   [ Director Agent ]             [ Multimodal Synthesizers ]
    (Gemini 2.5 Pro)              ├── Vertex AI Imagen 3
            │                     └── Cloud Text-to-Speech
   ┌────────┼────────┬────────┐
   ▼        ▼        ▼        ▼  Promise.all() (< 480ms)
[Script] [Light]  [Camera] [Audio]
(Gemini 2.5 Flash Swarm)
   └────────┬────────┴────────┘
            │
            ▼  Server-Sent Events (SSE) Stream
[ Audience Interactive Auditorium & Multiverse Studio ]

Backend & Swarm Architecture

  • Node.js, Express & TypeScript: High-concurrency asynchronous runtime hosting the telemetry buffer, SSE stream dispatcher, and state machine.
  • Unified Google Gen AI SDK (@google/genai): Built with the official modern SDK, using strict responseSchema JSON validation for deterministic agent outputs.
  • ClickHouse Cloud Integration: Connects to a managed ClickHouse Cloud cluster over secure TLS (port 8443) using @clickhouse/client.
  • Automated Circuit Breaker: Detects API quota saturation and seamlessly engages backup procedural generators to prevent system halts.

Database Engine & Schema Design

  • default.raw_telemetry (MergeTree): High-velocity event table partitioned by date and ordered by (scene_id, timestamp, user_id) for maximum columnar compression.
  • default.scene_sentiment_aggregates (SummingMergeTree): Pre-aggregated state table maintaining rolling sums of audience metrics.
  • default.scene_sentiment_mv (Materialized View): A continuous tumbling 5-second window view grouping by toStartOfInterval(timestamp, INTERVAL 5 SECOND) to calculate avg(sentiment_fear), avg(sentiment_action), and dilemma vote tallies on write.

Frontend & Visual Architecture

  • React 19 & Vite 6: Ultra-fast reactive UI with hot-module replacement and optimized asset bundling.
  • Ant Design 5/6: Enterprise-grade component system configured with custom high-contrast dark theme tokens (colorText: "#f8fafc", colorTextSecondary: "#cbd5e1", colorBgContainer: "#111217").
  • Tailwind CSS: Utility-first CSS powering the spatial glassmorphism and bento grid layouts.
  • HTML5 Canvas & WebGL: Zero-dependency, 60 FPS procedural rendering for the 3D neural point cloud and procedural 3D manifold.
  • Web Audio API: Native client-side procedural sound synthesizer generating sub-bass drops, mechanical clicks, telemetry chirps, and crisis alarms without downloading external audio files.

💥 Challenges We Ran Into

  1. The Row-Lock Write Bottleneck:
    During early prototyping with standard relational databases (PostgreSQL/MySQL), simulating 5,000+ concurrent votes generated massive row-level write locks, causing transaction queues to stall and connection pools to exhaust within seconds.
    Solution: Migrating to ClickHouse Cloud with in-memory double-buffering and async_insert=1 eliminated row locking entirely. ClickHouse writes data in compressed columnar parts, easily ingesting 50,000+ EPS with negligible CPU overhead.

  2. Materialized View Dependency Ordering & Idempotent Migrations:
    ClickHouse requires that target aggregate tables exist prior to creating a Materialized View with a TO <target_table> clause. In a cloud environment, automated migrations can fail if tables are evaluated out of order.
    Solution: We built an automated, idempotent migration runner (initDb.ts) that executes pre-flight checks, verifies table dependencies in topological order, and validates the connection before application startup.

  3. Sub-Second SLA vs. AI Generation Latency:
    A single monolithic prompt asking an LLM to generate narrative dialogue, lighting parameters, and camera coordinates simultaneously took between 8 and 15 seconds. Waiting 15 seconds per story beat killed live audience engagement.
    Solution: We decoupled the architecture into a hierarchical swarm. A high-reasoning Gemini 2.5 Pro model serves as Director Agent, setting the dramatic goal. It immediately fans out to specialized Gemini 2.5 Flash Sub-Agents running in parallel via Promise.all(). This collapsed the swarm execution time from 15 seconds to under 480 milliseconds.

  4. Ant Design Steps Layout Squishing in Narrow Grids:
    When rendering Ant Design's <Steps> component inside 4-column or 5-column dashboard sidebars, the container width (~350px) compressed 10+ history nodes, causing beat titles and descriptions to wrap character-by-character into vertical 1-letter stacks.
    Solution: We enforced whiteSpace: "nowrap", text-overflow ellipsis, dynamic minimum container width (Math.max(560, history.length * 155)), smooth horizontal scrolling with left/right nudge buttons, and introduced a Segmented Mode Switcher allowing users to switch between a horizontal track and a compact vertical card stack.

  5. Dark Mode Text Contrast & Readability:
    Standard Ant Design dark algorithm tokens set secondary text to low opacity (rgba(255, 255, 255, 0.45)), making dilemma options, live chat items, and war room telemetry faint on dark card surfaces.
    Solution: We augmented the global theme tokens with explicit high-contrast values (#f8fafc, #cbd5e1, #94a3b8) and refactored component styles to ensure crisp legibility across all four studio tabs.

  6. SDK Deprecation & API Evolution:
    Mid-development, Google announced the deprecation of legacy @google-cloud/vertexai packages in favor of the unified @google/genai SDK.
    Solution: We migrated all agent invocations to @google/genai, refactoring schema structures and error handling to ensure production readiness.

  7. Zero-Config Local Testing vs. Strict Cloud Production:
    We needed to ensure that evaluators, judges, and developers could run and verify the project locally even if they did not have active GCP service account keys or ClickHouse cloud passwords configured.
    Solution: We engineered a Dual-Engine Architecture: in local development (NODE_ENV=development), if credentials are absent, the server seamlessly boots an in-memory high-fidelity mock engine; in production (NODE_ENV=production), it enforces strict cloud-only verification and will abort if live connections cannot be established.

🏆 Accomplishments That We're Proud Of

  • Sub-480ms Parallel Swarm Latency: Executing specialized Gemini 2.5 Flash sub-agents in parallel under the guidance of a Gemini 2.5 Pro Director in less than half a second.
  • Live ClickHouse Cloud Integration: Establishing an operational cluster, ingesting 25,000+ telemetry events in benchmark bursts, and maintaining an active 5-second tumbling Materialized View with zero query overhead during playback.
  • True Agentic Creative Direction: Building an AI system that doesn't just output text, but coordinates multi-disciplinary cinematic crafts—lighting color theory, cinematographic camera physics, spatial acoustics, and dramatic screenplay structure—synchronously.
  • Ultra-Cinematic Landing Page: Creating a dark cyberpunk-luxury landing experience featuring a 3D neural point cloud that reacts in real-time to mouse velocity, an interactive 3D latent space explorer, and an asymmetric bento grid.
  • Model Context Protocol (MCP) Integration: Exposing our ClickHouse Cloud cluster through clickhouse-mcp.json, enabling AI agents to inspect, query, and verify live audience telemetry using natural language.
  • Production-Ready Serverless Deployment: Delivering single-command automated deployment scripts (deploy-cloud-run.ps1 and deploy-cloud-run.sh) that containerize and deploy the full-stack system to Google Cloud Run.

🧠 What We Learned

  • Columnar Analytics Reinvents Real-Time AI: Materialized Views in ClickHouse Cloud transform heavy query-time analytics into near-zero-cost write-time aggregations. By pre-aggregating mood averages on insert, the AI Director agent can read a single row in under 10ms instead of scanning raw tables.
  • Hierarchical Swarms Beat Monoliths: Fanning out targeted tasks to lightweight models (Gemini 2.5 Flash) supervised by a heavy reasoning model (Gemini 2.5 Pro) consistently yields higher creative fidelity, fewer hallucinations, and vastly superior latency than a single massive prompt.
  • Zero-Asset Procedural Synthesis: In high-speed generative applications, combining generative AI keyframes with client-side procedural engines (Web Audio API for sound, HTML5 Canvas for lighting stardust) eliminates network bandwidth bottlenecks while delivering zero-latency feedback.

🔮 What's Next for NeuroScene

  • Live Streaming Platform Bridges (Twitch / YouTube): Ingesting live chat sentiment, emotes, and super-chats directly into ClickHouse Cloud via WebSockets, allowing streamers to co-direct live movies with their communities.
  • Voice Cloning & Speech Synthesis with Cloud TTS: Integrating custom neural voice models to vocalize dialogue dynamically in characters' unique vocal signatures with real-time emotional inflection.
  • Asynchronous Video Generation with Veo & Imagen Video: Spawning asynchronous video generation jobs in the background while the client renders instant procedural 3D keyframe transitions, merging live speed with full video fidelity.
  • Audience Multiverse Provenance & Archival: Storing unique timeline cuts as permanent records, allowing audience members who collectively steered a specific storyline to revisit or collect their unique narrative branch.
  • Game Engine SDK (Unreal Engine 5 & Unity): Releasing an open-source C++ / C# plugin that allows game developers to connect multiplayer game worlds to the ClickHouse + Gemini Swarm telemetry pipeline.

🛠️ Built With

Languages, Frameworks & Libraries

  • Runtime: Node.js (v22 LTS)
  • Language: TypeScript 5.x
  • Backend Framework: Express.js, CORS, Dotenv
  • Frontend Framework: React 19, Vite 6
  • Component Architecture: Ant Design (antd, @ant-design/icons)
  • Styling & CSS: Tailwind CSS, Custom Spatial Glassmorphism Tokens
  • Process Management: Concurrently, Nodemon, ts-node

Database & Analytics Infrastructure

  • ClickHouse Cloud: Managed Serverless Columnar Database (Google Cloud region)
  • Storage Engines: MergeTree, SummingMergeTree
  • Real-Time Aggregations: Continuous Tumbling Materialized Views (scene_sentiment_mv)
  • Node.js Driver: @clickhouse/client (Official HTTP/TLS Driver over port 8443)
  • Agent Integration: Model Context Protocol (MCP)

Artificial Intelligence & Models

  • Google Gen AI SDK: @google/genai (Unified Official SDK)
  • Director Agent: Vertex AI Gemini 2.5 Pro (gemini-2.5-pro)
  • Script Sub-Agent: Vertex AI Gemini 2.5 Flash (gemini-2.5-flash)
  • Lighting Sub-Agent: Vertex AI Gemini 2.5 Flash (gemini-2.5-flash)
  • Camera Sub-Agent: Vertex AI Gemini 2.5 Flash (gemini-2.5-flash)
  • Keyframe Visuals: Vertex AI Imagen 3
  • Voice Synthesis: Google Cloud Text-to-Speech (Neural2 / Journey)

☁️ What Google Cloud Products Did You Use in This Project?

  1. Google Cloud Run: Hosts the containerized backend orchestration server and frontend client, providing serverless auto-scaling, HTTP/2 termination, and zero-maintenance infrastructure.
  2. Google Cloud Vertex AI (Gemini 2.5 Pro): Powers the central Director Agent, analyzing aggregated audience mood telemetry, setting pacing goals, and making high-level narrative decisions.
  3. Google Cloud Vertex AI (Gemini 2.5 Flash): Powers the parallel swarm of specialized sub-agents (Script, Lighting, Camera) operating simultaneously with sub-second response times.
  4. Google Cloud Vertex AI (Imagen 3): Synthesizes cinematic keyframe visuals depicting the newly branched scene based on the Camera and Lighting agents' specifications.
  5. Google Cloud Text-to-Speech: Generates character speech and dialogue audio streams from the Script Agent's generated screenplay lines.
  6. Google Firebase Authentication: Manages user identity, audience authentication, and session handling, supporting Google 1-click OAuth and guest critic mode.
  7. Google Cloud Artifact Registry: Stores production Docker container images for deployment to Cloud Run.
  8. Google Cloud Secret Manager: Secures sensitive environment variables and API credentials in production.
  9. Google Cloud Logging & Monitoring: Tracks telemetry throughput, request latencies, and agent execution times across Cloud Run instances.

🧰 All Other Tools and Products Used

  • ClickHouse Cloud on GCP: High-speed columnar analytics cluster hosted on Google Cloud Platform.
  • Model Context Protocol (MCP): ClickHouse MCP server (clickhouse-mcp.json) allowing AI coding assistants to query and inspect tables natively.
  • Web Audio API: Native browser API utilized for zero-latency procedural audio synthesis (bass drops, chirps, clicks, alarms).
  • Docker & Docker Compose: Containerization engine for cross-platform local development and Cloud Run container builds.
  • Google Antigravity CLI: AI pair programming agent utilized during development for architecture synthesis, iterative refactoring, and verification.
  • Git & GitHub: Version control and source code management.

Built With

Share this project:

Updates

Submission history