Twin City

Bringing AI to the Film Industry.

Comment

TwinCity: AI Movie Directing & Cinematic Shot Simulator

🎬 Inspiration

Planning high-stakes movie scenes, stunts, and complex multi-camera shots in real cities costs film studios millions of dollars in location permits, road closures, and pre-production time. Directing teams usually rely on static 2D storyboards that cannot accurately predict real-world sightlines, camera movement, or building occlusion.

We built TwinCity to give directors and cinematographers a virtual sandbox: describe any movie scene or stunt in natural language, pick any real-world city location, and immediately simulate, choreograph, and record cinematic camera shots in photorealistic 3D.

🎥 What It Does

TwinCity is an AI-powered cinematic pre-visualization platform:

  • Natural Language Scene Directing: Type cues like "A police cruiser chases a sports car through the intersection while a drone tracks from above" to automatically spawn 3D actors, calculate trajectories, and simulate vehicular and stunt kinematics.
  • Real-World Photorealistic 3D Sets: Anchor scenes in real urban environments (New York, San Francisco, Tokyo, London, and Paris) using high-fidelity 3D geospatial tiles.
  • Virtual Camera Rail Director: Draw custom 3D flight paths and camera rails across the city skyline with customizable tracking modes (follow action, track impact, or dynamic aerial overview).
  • In-Browser Film Recording: Capture 60 FPS video footage directly from the virtual camera rig to preview shots before ever stepping on set.

🛠️ How We Built It

We engineered TwinCity with a modern full-stack architecture tailored for real-time spatial simulation:

  • Frontend & 3D Engine: Built with React and Google Maps Photorealistic 3D Tiles (<gmp-map-3d>), coupled with Three.js GLB asset rendering for cars, drones, and articulated stunt actors.
  • Cinematic Camera Rail Math: We implemented continuous cubic Bézier splines and Hermite camera interpolation so directors can smoothly track dynamic subjects:

$$B(t) = (1 - t)^3 P_0 + 3(1 - t)^2 t P_1 + 3(1 - t) t^2 P_2 + t^3 P_3 \quad \text{for } t \in [0, 1]$$

Dynamic look-at camera orientation calculates heading $\psi$ and tilt $\theta$ relative to the action target $(x_a, y_a, z_a)$:

$$\psi = \operatorname{atan2}(y_a - y_c, \, x_a - x_c), \quad \theta = \operatorname{atan2}(z_a - z_c, \, \sqrt{(x_a - x_c)^2 + (y_a - y_c)^2})$$

  • Backend AI & Physics Engine: Built with FastAPI in Python. An intelligent screenplay compiler parses natural language directorial prompts into deterministic multi-stage actor timelines, speed envelopes, and coordinated stunt paths.
  • Video Capture Pipeline: Utilizes WebGL stream interception (captureStream) and MediaRecorder to generate download-ready MP4 cinematic previews right in the browser.

🧗 Challenges We Faced

  1. Camera Angle Alignment: Standard 3D asset coordinates author forward-facing vehicles along the $+Y$ axis, whereas 3D tile maps align heading $0^\circ$ to $+X$. We had to implement automatic coordinate-frame transformations so vehicles steer and drift along their travel heading without sliding sideways.
  2. Smooth 60 FPS In-Browser Capture: Synchronizing real-time 3D tile rendering with custom camera waypoints during canvas video recording required careful management of the WebGL drawing buffer and shadow-DOM canvas access.
  3. Intuitive Directing UX: Making advanced 3D choreography effortless for directors without forcing them to learn complex animation software.

💡 What We Learned

  • How to merge generative natural language directing prompts with deterministic kinematic trajectory models.
  • Techniques for manipulating 3D geospatial camera rails in urban canyons with skyscraper occlusions.
  • How much pre-visualization fidelity improves when directors can visualize real geographic scale and lighting conditions early in production.

🚀 What's Next for TwinCity

  • Multi-Camera Switcher: Real-time multicam recording (simultaneously rendering Wide, Over-the-Shoulder, and Action-Cam angles).
  • Time-of-Day & Atmospheric Lighting: Dynamic golden-hour sun positions, rain simulation, and night-time volumetric fog.
  • Custom Location Import: Allowing production crews to upload phone LiDAR scans of private soundstages and indoor sets directly into the city simulation.

Built With

Share this project:

Updates

Submission history