ICVFX

Inspiration On virtual production (ICVFX) LED volume stages (The Mandalorian), physical camera tracking and the virtual background must stay synced at 120Hz. A boom mic blocking tracking markers, network clock drift, or a dropped render frame causes severe visual tearing—halting $50k/hour soundstages. We built an autonomous AI crew member that detects, diagnoses, and heals stage desync in real-time before the director calls "Cut!".

What it does 120Hz Edge Ingestion & EKF: Ingests real-time FreeD tracking packets, runs a 6-DoF Extended Kalman Filter (EKF) with dead-reckoning, and streams live dolly coordinates to the Mission Control HUD. Grafana Cloud Observability: Streams Prometheus metrics to Grafana Cloud Mimir, tracking packet jitter, PTP clock drift, jerk breaches, and EKF covariance across 5 live panels. Multimodal AI Arbiter (Gemini on Vertex AI): Analyzes the witness video/snapshot, rendered frustum buffer, and telemetry manifest to isolate root causes in seconds. Autonomous MCP Remediation: Executes Model Context Protocol (MCP) tool calls (switches to Kalman dead-reckoning, recalibrates PTP, or adjusts frustum overscan) and verifies recovery under a 1.0ms SLA. Mission Control HUD: Renders side-by-side compensated videos with real-time HUD overlays and fault-injection controls. How we built it Vertex AI (Gemini 2.5 Flash): Rapid multimodal root-cause arbitration over video and sensor feeds. Grafana Cloud Mimir: Remote-write telemetry pipeline with Snappy-compressed Protobuf payloads. Google Cloud Run: Serverless containerized deployment with scale-to-zero compute (₹0.00 idle cost). Core Stack: C++20 zero-copy kinematic engine, Python FastAPI backend, React 18 / Vite HUD, and MCP client/server tooling. Challenges we ran into Prometheus Remote-Write Wire Format: Building zero-dependency, snappy-compressed Protobuf serialization in Python to meet Grafana Cloud Mimir's strict ingestion specs. Sub-10s Video Pipeline: Transcoding high-framerate video and streaming it into Vertex AI within serverless request timeouts. Cloud Run Native Integration: Packaging C++ binaries, FFmpeg, and Google Cloud ADC credentials inside a lightweight serverless container. Accomplishments that we're proud of 100% Diagnostic Accuracy: 60 automated unit, load, and edge-case tests passing cleanly across all fault scenarios. Sub-3s Incident Recovery: End-to-end detection, diagnosis, MCP remediation, and closed-loop verification executing under strict broadcast SLAs. Live Hybrid Mesh: Physical edge UDP crane tracking simultaneously relaying to Google Cloud Run and Grafana Cloud. What we learned Correlating high-frequency time-series telemetry with visual witness frames allows Gemini to effortlessly distinguish physical sensor occlusions from digital network drift. Standardizing remediation through Model Context Protocol (MCP) decouples the AI from stage hardware, making it camera- and engine-agnostic. What's next for ICVFX Unreal Engine LiveLink Plugin: Native nDisplay plugin for sub-frame frustum warping inside Unreal 5.5. Multi-Camera Volumes: Simultaneous multi-camera arbitration across A/B/C camera witness rigs. Edge TPU Integration: Compiling the EKF and incident gating to Google Coral TPUs on camera heads for zero-latency local fallback.