CinePilot

Inspiration

Aerial cinematography is unforgiving. Drone pilots often burn through precious battery cycles capturing redundant angles, struggling with horizon drift, or missing key composition requirements on a shot list. Without a seasoned director standing by the monitor calling cues, solo operators are forced to multitask between complex manual flight stick inputs and high-level visual storytelling.

CinePilot was born from a simple question: What if every drone pilot had an intelligent, multimodal director speaking real-time framing cues directly into their earpiece while orchestrating production shot lists?

How We Built It

CinePilot bridges live aerial video feeds directly into an autonomous multimodal agent loop powered by the Gemini Live API:

Live Video Streaming Pipeline: We ingest high-definition video from a physical DJI Mini 4K drone via local RTMP streaming into MediaMTX, sampling frames at ~1.2 FPS with OpenCV.
Embodied Reasoning & Live Directing: Frames are streamed asynchronously over a bidirectional WebSocket to the Gemini Live API. CinePilot evaluates composition (rule-of-thirds, horizon pitch, lighting, and pacing) and maintains long-horizon shot memory.
Deterministic Tool Calling: When CinePilot observes that cinematic criteria have been met or require correction, it triggers structured function calls:
- update_shot_list(shot_id, status, feedback): Dynamically transitions shot states ($PENDING \rightarrow IN_PROGRESS \rightarrow COMPLETED$).
- speak_director_guidance(instruction, priority): Synthesizes direct vocal guidance into the pilot's headset via the browser Web Speech API.
Enterprise Telemetry with Grafana: Every visual decision, latency metric ($\Delta t \approx 1.2\text{s} - 2.1\text{s}$), and tool execution is pushed via the Loki HTTP Push API (/loki/api/v1/push) to a real-time Grafana dashboard.

Challenges We Faced

Hardware SDK Constraints: The DJI Mini 4K does not support direct programmatic SDK flight control. We turned this constraint into an architectural strength by building CinePilot as a "Human-in-the-Loop Co-pilot" that directs the human operator through audio cues rather than attempting unstable automated flight.
Low-Latency Bidirectional Streaming: Synchronizing real-time RTMP video decoding with bidirectional Gemini tool responses required an asynchronous producer-consumer pattern in Python to keep response latency under 2 seconds.
State Drift & Memory: Ensuring the model accurately tracked shot completion without getting confused by repeating visual backgrounds required strict schema validation and application-side state persistence.

What We Learned

The Gemini Live API excels at dynamic temporal reasoning across continuous video streams, spotting subtle tilt deviations and visual blockages far faster than conventional static vision models.
Real-time audio cues combined with visual dashboards dramatically reduce cognitive load for field operators.

What's Next for CinePilot

Integrating direct gimbal control adapters and depth-map projection for 3D ground bounding box raycasting.
Multi-camera coverage orchestration pairing aerial drones with ground-based rovers for synchronized film sets.