Genlock Sentinel: Autonomous ICVFX SRE Agent

Autonomous In-Camera VFX SRE agent built with Google ADK, Gemini 3.7/3.1, and Grafana Cloud MCP to eliminate 150µs frame-drift & prevent catastrophic virtual stage burn.

Comment

🛡️ Genlock Sentinel — Autonomous ICVFX nDisplay SRE Agent

Sub-Second Frame-Lock Recovery • Non-Dismissible HITL Safety Gate • Silence-Over-Guessing Protocol
Built for Google Cloud Agentic Cinema: The Blockbuster Hackathon — Enterprise Agent Platform & Grafana Labs Tracks

YouTube Demo Video GitHub Repository Tests Passing

Genlock Sentinel Hero Banner

🌟 The 4 Core Blockbuster Hackathon Pillars

1. 🎯 Why This Use Case is a Strong Fit for Google Cloud & Grafana Labs

In modern Hollywood In-Camera Visual Effects (ICVFX), massive LED volumes are driven by multi-node Unreal Engine nDisplay GPU clusters (16+ nodes, render-01 to render-16). The physical camera shutter, tracking systems, and render engines are phase-locked via SMPTE/PTP IEEE 1588 genlock pulses.

When frame synchronization drifts by just 150 microseconds (150 µs), the physical camera captures catastrophic scanline tearing, motion-blur phase disparity, and rollbars permanently baked into the raw camera footage. At an average virtual stage burn rate of $1,800/minute ($50,000–$150,000/hour), manual log grepping wastes critical minutes and tens of thousands of dollars in downtime.

Genlock Sentinel makes the virtual production cluster an autonomous, self-healing runtime. Built on Google Agent Development Kit (ADK) 2.8.0 and connected to Grafana Cloud via Model Context Protocol (MCP), the agent inspects high-frequency Prometheus genlock telemetry, triages live Loki logs and Tempo distributed traces, and restores frame lock in under 800 milliseconds.

2. ⚡ How It Creates a Better Operational Experience (Sub-Second Healing)

Traditional virtual production visualizers force on-set SREs into multi-dashboard context switching, manual log grepping, and speculative guesswork while production burn ticks away. Genlock Sentinel transforms passive observability into an autonomous, agent-native workspace:

  • Sub-Second Mathematical Diagnosis: Pairs Gemini 3.7 Flash for sub-second telemetry triage with zero-temperature Gemini 3.1 Pro native reasoning tokens for causal root-cause correlation in $< 800\text{ms}$.
  • Autonomous Low-Blast Remediation: Reversible cluster mitigations (failover_cluster_leadership, deprioritize_texture_streaming, force_genlock_resync) fire deterministically without human delay when confidence $\ge 0.75$, dropping drift back to green baseline ($38.2\mu\text{s}$) before camera roll.
  • Broadcast-Grade Mission Control: Driven by AG-UI Server-Sent Events (SSE) streaming RFC 6902 JSON Patch state deltas at 60 FPS to a reactive React 18 / Vite Carbon Cockpit featuring a 16-node LED radar matrix and live microsecond waveform telemetry.

3. 🤝 Human-Agent Co-Creation & Safety Boundaries (HITL Gate)

Genlock Sentinel establishes mechanically enforced, bidirectional safety boundaries between on-set human supervisors and autonomous agents:

  • Isolated Actuator Boundaries: Cognitive reasoning nodes have zero actuator bindings. High-stakes cluster mutations (halt_live_take, fallback_to_greenscreen) are strictly isolated behind dedicated post-approval nodes.
  • Non-Dismissible Human-in-the-Loop (HITL) Gate: Whenever complex telemetry indicates high-blast-radius failures (concurrent GPU thermal throttling at 94°C and network packet loss on render-12), autonomous execution halts deterministically at an ADK LongRunningFunctionTool checkpoint.
  • Live Financial Exposure Projection: The engine projects a non-dismissible executive card displaying accrued financial exposure ($2,450.00 stage halt cost) and visual impact score (9.0/10), requiring binary supervisor sign-off (Approve / Deny) before physical actuation occurs.

4. ⚙️ How We Implemented Google ADK & Grafana MCP

  • 7-Node ADK 2.8.0 Directed Workflow Graph: Engineered an authoritative 7-node directed graph (stream_watch ➔ evidence_triage ➔ root_cause_correlation ➔ autonomous_dispatch / hitl_card_generation ➔ hitl_pause ➔ post_approval_handling) with compile-time decision edge routing.
  • Dual-Model Cognitive Topology: gemini-3.7-flash powers high-throughput triage (Node 2) and structured HITL card generation (Node 5); gemini-3.1-pro at strict temperature=0.0 with native reasoning tokens executes deep causal correlation (Node 3).
  • Bi-Directional Grafana Cloud MCP Integration: Native MCP tools querying Grafana Loki (query_loki_logs) and Tempo (find_slow_requests, get_trace_by_id) over authenticated endpoints.
  • Constitutional Silence-Over-Guessing Protocol: If Loki queries fail or timeout after exponential backoff (1s, 2s, 4s), the agent explicitly records logs_available: false, outputs proposed_action: "none", and escalates to human diagnostics with zero fabricated log lines.
  • Enterprise Checkpoint Durability: All 10 state fields of GenlockSentinelState are persisted continuously across Google Cloud SQL PostgreSQL 16 via non-blocking asyncpg with optimistic locking and automatic StaleSessionError retry handling.
  • Google Model Armor Defense: Telemetry streams are screened against OWASP Top 10 for LLM Applications (OWASP LLM01 prompt injection overrides and OWASP LLM02 credential leakage).

💡 Inspiration

Virtual production incident rooms are chaotic: when an LED volume tears during a live filming take, studio capital burns at $1,800 a minute ($30/second). Stage engineers face thousands of lines of asynchronous Unreal Engine logs, GPU junction temperatures, and PTP clock traces across siloed dashboards. We asked ourselves: What if an autonomous AI agent could live directly inside the ICVFX production network, pull Grafana Loki logs and Tempo traces via MCP in microseconds, correlate root causes with Gemini reasoning tokens, and heal the render cluster before the director calls "Action"? Google ADK 2.8 and Grafana Cloud made this vision reality.

🔍 What It Does

Genlock Sentinel is an enterprise-grade autonomous SRE agent pre-loaded with a live 16-node Unreal Engine nDisplay cluster (render-01 through render-16):

  1. Non-LLM Stream Watcher (Node 1): Ingests high-frequency Prometheus sync metrics ($60\text{Hz}$) with zero LLM latency overhead, detecting drifts exceeding the $150\mu\text{s}$ threshold within $2\text{ms}$.
  2. Sub-Second Observability Triage (Node 2): Dispatches Gemini 3.7 Flash to query live Grafana Cloud Loki logs and Tempo trace spans simultaneously through Model Context Protocol (MCP) tools.
  3. Deep Causal Correlation (Node 3): Executes zero-temperature Gemini 3.1 Pro with native reasoning tokens over extracted telemetry, achieving $>95\%$ classification confidence.
  4. Autonomous Cluster Failover (Node 4): Deterministically executes pre-approved, reversible mitigations in $< 800\text{ms}$, restoring sub-$50\mu\text{s}$ frame-lock before the camera roll begins.
  5. Human-in-the-Loop Risk Gating (Nodes 5–7): Intercepts ambiguous or high-blast-radius failures, freezes the ADK 2.8 workflow durably, projects an executive sign-off card ($2,450 risk), and executes post-approval actions (halt_live_take, fallback_to_greenscreen).

🖥️ Visual Grounding & Live Engine Showcase

1. Hollywood Carbon Cockpit Overview

Hollywood Carbon Cockpit Overview Full operations dashboard featuring 16-node LED cluster radar (R01–R16), live $1,800/min nuclear burn ticker, real-time sync-offset microsecond area chart with 150µs breach line, and 7-node AG-UI workflow bus.

2. Autonomous Self-Healing via Node 4 (<800ms Recovery)

Autonomous Self-Healing via Node 4 Node 4 dispatches pre-approved reversible failover for render-07 (185.4µs sync drift) back below the 150µs perimeter within 480ms without human intervention, maintaining active live camera rolling state.

3. Enterprise Human-in-the-Loop (HITL) Safety Gate

Enterprise Human-in-the-Loop Approval Gate Non-dismissible supervisor sign-off modal for high-stakes drift on render-12, displaying financial risk ($2,450 halt estimate), visual impact score (9.0/10), and binary Approve/Deny controls.

4. Silence-Over-Guessing & Telemetry Gap Protocol

Silence-Over-Guessing Protocol When Loki telemetry drops out, the engine sets logs_available: false and routes to ambiguous classification without fabricating log lines, exposing live Gemini 3.1 Pro thinking tokens.

5. Live Grafana Cloud Loki Telemetry Ingestion (Partner Track)

Grafana Cloud Loki Explorer Real-time Grafana Cloud (magentaparfait3455.grafana.net) Loki log stream explorer with active LogQL {cluster="ndisplay"} telemetry ingestion, volume histograms, and PTP sync jitter diagnostics.

6. Google Cloud SQL PostgreSQL Enterprise Instance

Google Cloud SQL PostgreSQL Instance Enterprise-tier Google Cloud SQL PostgreSQL 16 instance (genlock-sentinel-db in us-central1) providing durable session checkpointing alongside live Grafana Cloud Loki log streams.

🛠️ Registered Agent Tools Specification Matrix

Tool Name Type Bound Node Mathematical Algorithm / Target Safety Precondition & Guardrail
query_loki_logs Read / MCP Node 2 (Evidence Triage) LogQL {cluster="ndisplay"} Ingestion Model Armor OWASP LLM01/02 screening; 3-retry backoff (1s, 2s, 4s)
find_slow_requests Read / MCP Node 2 (Evidence Triage) Sift Trace Outlier Isolation Isolates render barrier execution delays >= 100ms
get_trace_by_id Read / MCP Node 2 (Evidence Triage) Tempo Distributed Trace Traversal Keyed to camera frame_id; extracts render span timings
failover_cluster_leadership Reversible Actuator Node 4 (Autonomous) Virtual IP & PTP Master Re-election Requires category == "network_jitter" and confidence >= 0.75
deprioritize_texture_streaming Reversible Actuator Node 4 (Autonomous) Texture Pool Throttle De-escalation Requires category == "asset_streaming_stall" and confidence >= 0.75
force_genlock_resync Reversible Actuator Node 4 (Autonomous) PTP Hardware Phase Sync Reset Requires category == "thermal_throttle" and confidence >= 0.75
halt_live_take HITL Actuator Node 7 (Post-Approval) Camera Roll Cut & Stage Burn Freeze Requires supervisor approval (approval_state == "approved")
fallback_to_greenscreen HITL Actuator Node 7 (Post-Approval) Solid Green Panel Drive (100 IRE) Requires explicit supervisor approval; overrides LED volume
execute_threshold_exceeding_failover HITL Actuator Node 7 (Post-Approval) High-Cost Cluster Node Swap Gated when remediation cost exceeds $500 threshold

🛠️ How We Built It

Runtime & Orchestration: Built with Python 3.11+ and Google Agent Development Kit (ADK) 2.8.0 Workflow Runtime, orchestrating a strictly isolated 7-node directed graph with typed state transitions.

Dual-Model Cognitive Layer: Integrated Vertex AI models using Pydantic V2 strict schemas (extra="forbid") with gemini-3.7-flash for high-throughput triage and gemini-3.1-pro at temperature=0.0 with native reasoning tokens for causal classification.

Observability & Telemetry Bus: Native Grafana Cloud Model Context Protocol (MCP) client connecting directly to Loki and Tempo HTTP endpoints, supplemented by OpenTelemetry 4-level GenAI distributed trace spans exported to Google Cloud Trace.

State & Persistence Architecture: Central 10-field state (GenlockSentinelState) governed by pure functional reducers (reduce_immutable, reduce_merge_by_key, reduce_append_only, reduce_last_write_wins), durably checkpointed to Google Cloud SQL PostgreSQL 16 via non-blocking asyncpg.

Defense-in-Depth Security: Google Model Armor pre-screens all incoming cluster logs against prompt injection (OWASP LLM01) and credential leaks (OWASP LLM02). AST import boundaries structurally prevent cognitive nodes from binding actuator tools (OWASP LLM06 Excessive Agency).

Hollywood Carbon Cockpit Frontend: Built with React 18, TypeScript 5.9, Tailwind CSS, and Vite 6, driven by an AG-UI SSE client streaming RFC 6902 JSON Patch state deltas at 60 FPS.

🚧 Challenges We Ran Into

Vertex AI Protobuf Schema Sanitization: Vertex AI's REST generation endpoint rejected Pydantic V2 schemas with extra="forbid" (additionalProperties: False) with HTTP 400 errors. We built a recursive schema sanitizer (_clean_schema_for_gemini) stripping additionalProperties and title fields while maintaining strict schema structure.

Cloud SQL Checkpoint Concurrency under Rapid Drift: Concurrent synthetic drift events caused StaleSessionError optimistic concurrency collisions. We cached the DatabaseSessionService singleton and implemented an exponential reload-and-retry loop (up to 3 attempts) in save_checkpoint.

Enforcing Silence-Over-Guessing in LLM Inference: AI models instinctively attempt to guess missing log lines when telemetry times out. We codified structural Python preconditions: when Loki queries fail after 3 retries, the client returns logs_available: false and empty results [], mathematically forcing the Gemini reasoning loop to refuse speculation.

Center-Locked Native SVG Waveform Dynamics: Standard CSS transforms caused telemetry pulse rings and breach badges to detach from their SVG origins during dynamic window resizing. We replaced CSS scaling with native SVG <animate> elements directly animating radius r, locking indicators to exact pixel coordinates.

AST Isolation of High-Blast Actuators: To prevent developer import regressions, we implemented static AST inspection in CI/CD verifying that cognitive nodes (evidence_triage.py, root_cause_correlation.py) never import Tools 7–9 (halt_live_take, fallback_to_greenscreen).

🏆 Accomplishments That We're Proud Of (228 / 228 Tests Passing)

228 Tests Passing Terminal Proof

  • 228 / 228 automated tests passing (100% pass rate) across 20 test suites in under 8 seconds.
  • Sub-800ms Autonomous Remediation: Ingesting Prometheus drift, triaging Loki logs, executing Gemini correlation, and dispatching leader failover completes in under 800 ms.
  • Zero TypeScript Errors & Clean Production Bundle: 1,868 frontend modules compiled cleanly in 2.28s with zero warnings.
  • Full OWASP Top 10 for LLM Applications Compliance: Verified defense against Prompt Injection (LLM01), Sensitive Information Disclosure (LLM02), and Excessive Agency (LLM06).
  • Zero Fabrication Guarantee: 100% verified citation enforcement where every root-cause assertion maps to concrete telemetry evidence.

📚 What We Learned

Virtual production SRE requires absolute predictability. In mission-critical environments with $1,800/minute burn rates, non-deterministic chatbots and unbounded ReAct loops are fatal liabilities. Structuring AI workflows as deterministic directed graphs with typed state reducers, Model Armor boundaries, and mechanical Human-in-the-Loop gates proves that generative AI can be safely deployed in high-consequence enterprise environments.

🚀 What's Next for Genlock Sentinel

  • SMPTE ST 2110 IP Video Fabric Integration: Direct hardware packet timing analysis across uncompressed SMPTE ST 2110 IP broadcast networks.
  • Automated nDisplay Config Patching: Generating verified, diff-checked Unreal Engine .ndisplay configuration updates committed directly to stage GitOps repositories.
  • Multi-Stage Studio Federation: Federation of Genlock Sentinel agents across multi-stage Hollywood studio lots with unified Grafana Cloud fleet dashboards.

🧪 Live Evaluation Instructions for Judges

Judges can clone the public repository and verify the complete agent runtime, test suite, and live scenarios locally:

# 1. Clone repository & install backend dependencies
git clone https://github.com/piyushxlabs/genlock-sentinel.git
cd genlock-sentinel/backend
uv sync

# 2. Run the Exhaustive 228-Test Verification Suite
uv run pytest tests/ -v
# Verified Result: 228 passed, 2 skipped in ~8s (100% pass rate)

# 3. Launch Backend & Frontend Cockpit
# Terminal 1 (Backend):
uv run uvicorn src.main:app --port 8000 --reload

# Terminal 2 (Frontend Console):
cd ../frontend && pnpm install && pnpm dev
# Opens http://localhost:3000

# 4. Trigger Live Judge Scenarios via CLI:
# Scenario A: Autonomous Healing (<800ms failover on render-07)
uv run python scripts/simulate_drift.py --scenario simple

# Scenario B: Human-in-the-Loop Safety Gate ($2,450 risk on render-12)
uv run python scripts/simulate_drift.py --scenario complex

# Scenario C: Silence-Over-Guessing Telemetry Outage (Zero Hallucination)
uv run python scripts/simulate_drift.py --scenario edge

# Reset stage to pristine baseline at any time:
uv run python scripts/reset_session.py

Built With

Share this project:

Updates

Submission history