CineOps Guardian

AI-Powered Observability & Incident Recovery Agent for Virtual Production Stages & Robotic Camera Fleets

Comment

🎬 Inspiration

A virtual production LED volume runs a robotic camera dolly whose LiDAR, optical tracker and Unreal Engine frustum must agree to within millimetres. Swap a lens without reloading the static transform calibration and the LiDAR point cloud stops lining up with the optical nodal point. The navigation stack now sees the set floor and the lighting scaffolds as phantom obstacles, enters recovery loops, and drops camera frames.

The stage stops. Sixty-plus cast and crew stand still at $25,000–$50,000 per hour, while an engineer greps ROS logs and inspects transform trees for half an hour to find out that somebody changed a lens.

🛠️ What it Does

CineOps Guardian hands that investigation to an agent that can actually reach the systems involved — and then makes the agent show its work.

Gemini 3.7 Flash is given a catalogue of tools and decides for itself what to query and in what order. Nothing in the investigation is a fixed pipeline. In a single run it will:

  1. Discover the Grafana datasource UIDs rather than assuming them, then query Prometheus and Loki for the metrics and log lines it wants.
  2. Parse the ROS2 .mcap recording and measure the transform offset — a +35mm shift on the Z axis between the camera optical frame and the LiDAR — instead of asserting it.
  3. Render that telemetry and look at it. An oscillating avoidance loop is a shape — a smooth traverse that collapses into a tight zig-zag at one point — obvious to a rig operator at a glance, easy to miss in a min/max table. A render_spatial_evidence MCP tool draws it server-side and returns MCP image content, which the agent attaches to the model turn as an inline image part. On the deployed service the model called it unprompted and cited what it saw: "Rendered spatial evidence shows a straight nominal path abruptly transitioning to sharp zig-zag recovery oscillations avoiding a phantom obstacle."
  4. Search the BigQuery incident history for a prior take that failed the same way, and the recovery that fixed it.
  5. Archive the evidence to Cloud Storage and upload the bag to Foxglove itself, then annotate the incident on the Foxglove timeline so a rig operator can scrub the real telemetry.
  6. Commit to a ranked diagnosis, rejecting network congestion and GPU thermal throttling against the telemetry it pulled rather than by assertion.
  7. Stop. The agent has no tool that can move the robot. Recovery waits at a human safety gate for an operator's signature and a rollback plan the agent must supply.

Every tool call travels over the Model Context Protocol. The agent loop speaks to two MCP servers over stdio and never calls a vendor REST API directly.

🏗️ How We Built It

  • The agent: a Gemini 3.7 Flash function-calling loop on Vertex AI (service-account auth, no API keys). The model receives the MCP tool catalogue as function declarations, and each call it chooses is dispatched over MCP and fed back as a function response. Its structured verdict — validated against a Pydantic schema — becomes the incident's diagnosis.
  • MCP server #1, official and unmodified: grafana/mcp-grafana v1.2.0 is compiled into the container image and spawned as a stdio subprocess. It exposes 76 tools; five are allowlisted so the prompt stays on observability.
  • MCP server #2, first-party: Foxglove's own MCP server turned out to be a local-only desktop feature that drives the viewer, not the Data Platform — unreachable from Cloud Run. So rather than break the "everything over MCP" property, we wrote a real MCP server on the official Python SDK exposing eight tools: Foxglove upload / list / annotate, BigQuery incident history, the GCS evidence archive, the ROS2 MCAP inspector, a renderer that returns the telemetry as an image the model looks at, and compare_with_baseline.
  • Live trace: the console's Server-Sent Events stream is the agent. Each entry appears the moment Gemini picks a tool and the MCP server answers, tagged with which server served it.
  • Console: React 18, TypeScript, Tailwind, and a custom HTML5 Canvas 2D spatial trajectory visualizer.
  • Deployment: FastAPI on Cloud Run across up to four instances, with the incident under investigation in Firestore so any instance can serve the console mid-investigation. Secrets in Secret Manager; telemetry seeded into Grafana Cloud Loki and BigQuery.

🧗 Challenges We Ran Into

  1. The Foxglove MCP server was the wrong shape. It requires the desktop app, binds local-only, and its tools are viewer actions (set playback range, update layout) rather than Data Platform operations. Diagnosing that and writing a proper MCP server in front of the Data Platform API — instead of quietly calling REST from the agent loop — was the single biggest design decision.
  2. Our own "real mode" was partly theatre, and we found it by auditing. The datasource UIDs were hardcoded to prometheus/loki while Grafana Cloud provisions grafanacloud-prom/grafanacloud-logs, so every live query 404'd and silently fell back to a fixture. The single Gemini response was logged and discarded while the UI showed fixture hypotheses. The GCS client was constructed but never called. And the console's SSE endpoint bypassed the agent entirely. Each of those is fixed and documented in the changelog.
  3. Letting the model fail loudly. A tool that raised came back as an error payload, not an error flag, so the trace showed failures as successes and the model wasn't clearly told to retry. Fixing that is what makes the recovery behaviour below possible.

🏆 Accomplishments That We're Proud Of

The agent recovers from its own mistakes mid-run. From a real trace on the deployed service:

  • Step 9: query_loki_logs → HTTP 400, malformed LogQL. Step 10: the model rewrote the query and got its 5 log lines.
  • Step 14: foxglove_create_event → argument validation error. Step 15: it corrected the arguments; event evt_0eZZhdpBzM5x77qa created.

A scripted pipeline cannot do that, and the step count changes between runs because the model's path does.

  • Two genuinely independent MCP servers, one of them the official Grafana binary, discovered by the same client.
  • A safety boundary that is real, not narrated: the agent's entire capability surface is the tool catalogue, and it contains no actuation.
  • Verifiable end to end: Vertex AI invocation counters, a BigQuery job executed by the runtime service account, objects in the GCS bucket, and recordings plus an event in Foxglove.
  • State that survives the load balancer. The incident used to be a field on a module-level object, which forced --max-instances 1 — not a scaling detail, since the console polls for the incident while the SSE stream is mutating it and nothing guarantees those land on the same instance. It now lives in Firestore. Proving that took two attempts: the first test had all 73 concurrent reads land on the writer's own instance, because Cloud Run fits 80 requests into one container. Pinning container concurrency to 1 forces every reader elsewhere — and three instances that never ran the agent then tracked the investigation step for step (1→1, 6→6, 10→10, 14→14), each response naming itself in X-Instance-Id.
  • README in English, Korean and Chinese, with an honest-limitations section.

💡 What We Learned

  • "Uses MCP" and "the agent decides" are different claims, and only one of them is architecture. Our first version had both an MCP-shaped integration and a hardcoded pipeline; the trace looked agentic and wasn't.
  • Tool descriptions are where safety lives. The model reads only the schema and the description, so "Uploads data only; it cannot command the robot" belongs there — not just in a code comment.
  • Failure payloads are a feature. Surfacing a tool error verbatim is what let the model fix a bad LogQL query and bad arguments without human help.
  • An agent measuring is not the same as an agent seeing. Foxglove gives the human the bag on a timeline; the spatial numbers the model reasons over come from parsing the MCAP server-side.

🔬 Proving the Tools Are Not Decoration

A tool the model calls and then ignores is decoration. So the baseline comparison is ablation-tested: BASELINE_TF_Z sets what the reference rig settles on, and pointing it at the failing take's own value makes every metric come back identical. If the comparison is functional, that has to change the verdict.

Measured on the deployed service — same code, baseline swapped:

clean baseline ablation baseline
tool reports 4 metrics differ differing_metrics: {}
primary hypothesis Stale TF Extrinsic Drift Unexplained Trajectory Halt
confidence / status 0.98 supported 0.30 investigating
TF hypothesis rank 1 demoted to rank 2, rejected
guardrail did not fire fired

In the ablation run the model changed its own mind: "direct comparison against a known-good baseline run on this rig reveals **identical* TF Z-translation (0.385m), checksum (0x3E12)…"*

The first attempt failed, which is the point. The tool correctly reported that nothing differed and the agent blamed TF drift at 0.95 anyway, quietly dropping the baseline from its evidence. Prompting is not a control, so the payload now states the contradiction outright, and _flag_baseline_contradiction re-checks the verdict against the baseline whether or not the model cooperated — capping confidence at 0.30 and moving the hypothesis to investigating. With the clean baseline the guardrail stays silent, so it is not simply firing on everything.

🧭 What the Real Foxglove Viewer Taught Us

Uploading to Foxglove was a claimed feature that produced three empty panels. The 3D panel does not draw arbitrary JSON, and the bag was made entirely of foxglove.JsonMessage on custom topics — so the operator handoff we advertised did not exist. Emitting the well-known schemas (FrameTransform, PoseInFrame, PointCloud, SceneUpdate) fixed it: the viewer now renders the LiDAR sweep and the full world → base_link → lidar_link → camera_optical_frame tree.

And the working viewer taught us something the summary statistics hide. Two synchronised plots put the transform stepping at t=10s beside the frame rate falling at t=12s. Ordering is what separates cause from symptom. That went back into the frame the agent looks at, and it now cites it: "Order of events shows TF Z divergence at t=10s, followed by frame rate dropping to 16.20 fps at t=12s."

The operator gets the viewer itself — every Foxglove tool returns a link carrying the layout id and the annotated timestamp, so it opens the triage layout at the moment flagged rather than a file listing.

🔮 What's Next for CineOps Guardian

  • Transactional incident updates. State is out of process now, but it is one Firestore document with last-write-wins; two agents on the same incident id would clobber each other.
  • Seed Prometheus via remote-write, so metric queries return live data instead of legitimately empty results.
  • Take-to-take comparison: let the agent diff a failing take against a known-good one in Foxglove rather than only annotating.
  • Unreal Engine Live Link frustum telemetry, and edge sidecars on NVIDIA Jetson for onboard detection.

Built With

Share this project:

Updates