FireSight: 3D Tactical Digital Twin

Turning raw drone video into navigable 3D thermal twins and AI tactical briefs for Incident Commanders

Comment

Inspiration

Structural fires and wildfires create incredibly chaotic, fast-moving environments. When a building is burning, Incident Commanders currently rely on radio chatter or flat, 2D drone video feeds. But a flat video cannot accurately convey the volume of a fuel load, the true vector of a smoke plume, or the exact height of a compromised roof.

We wanted to bring Military-Grade Spatial Intelligence specifically to fire rescue teams. We realized that by combining drone footage, Multimodal Vision LLMs, and Gaussian Splatting, we could give Fire Commanders a "God-Mode"—a fully navigable 3D digital twin of the fireground, complete with AI-estimated risk vectors, allowing them to make life-or-death tactical decisions without risking human lives.

What it does

FireSight is an intelligent ingest engine and visualization dashboard built exclusively for fire missions.

Ingest & Sync: A drone pilot captures a <60s orbit of the burning structure and uploads it. The system generates a zero-login "Shared Mission Session" URL. The pilot drops the link in Slack, and the remote Fire Commander sees the exact same live dashboard instantly.

AI Fire Analyst: While uploading, our backend extracts keyframes using OpenCV and feeds them to a local Vision LLM (Qwen 2.5). The AI estimates fire growth rate, wind speed, and structural integrity, outputting a structured tactical brief (e.g., "Evacuate Sector 4").

3D Thermal Twin: In the background, our server orchestrates Nerfstudio to process the video into a 3D point cloud.

God-Mode Dashboard: The web frontend renders the 3D twin, stripping away natural colors to apply a procedural Thermal Heatmap based on height/density. It then overlays a real-time Computational Fluid Dynamics (CFD) simulation to visualize wind and fire spread vectors directly on the burning structure.

How we built it

We built a highly asynchronous, cloud-native architecture using a powerful Linode equipped with a RTX6000 from Akamai technology:

Frontend: React 18, Tailwind CSS, and @react-three/fiber. We wrote custom Three.js logic to apply pseudo-thermal gradients and implemented a real-time Euler-advection particle system for the wind/fire simulation.

Backend: Python FastAPI orchestrating asynchronous task queues. We used SQLite for mission state persistence, allowing multiple users to sync to the same URL seamlessly without heavy WebSocket overhead.

AI/ML (The Analyst): We utilized Ollama to run Qwen 2.5 locally. We aggressively prompted the Multimodal LLM to return strict JSON schemas representing tactical physics estimates.

3D Pipeline (The Muscle): We integrated Nerfstudio CLI commands (ns-process-data and ns-train) using COLMAP and Splatfacto directly into the backend workflow to convert .mp4 videos into .ply point clouds.

Challenges we ran into

The reality of a fireground is that fire and smoke move. Standard photogrammetry (COLMAP) frequently fails on moving subjects because it assumes a static world.

We solved this by engineering a custom fallback pipeline. If the primary ns-process-data fails to create a sparse model, our FastAPI backend catches the error and automatically re-runs only the COLMAP mapper with highly lenient, custom tolerances (e.g., ba_global_function_tolerance=1e-4, lower inlier requirements). This allowed us to successfully reconstruct dynamic fire scenes that standard pipelines would outright reject.

Accomplishments that we're proud of

Real-Time CFD Flow Overlay: Building a divergence-free, curl-noise turbulence simulator directly in the browser using React Three Fiber. It visualizes wind advection and fire plume buoyancy without lagging the UI.

The "Zero-Friction" Architecture: Realizing that firefighters don't have time to create accounts or remember passwords. We built the entire platform around persistent, shareable URL sessions backed by a lightweight SQLite queue.

Seamless Orchestration: Successfully integrating heavy GPU subprocesses (Nerfstudio) and a local Vision LLM into a single, cohesive FastAPI backend that provides real-time progress updates to the frontend.

What's next for FireSight

Edge deployment. We want to move the FastAPI ingestion engine directly onto the drone's edge computer (like an NVIDIA Jetson). This way, the .ply point cloud and AI JSON brief are generated locally in the sky and streamed over low-bandwidth radio directly to the Fire Commander, bypassing the need to upload heavy .mp4 video files from a remote incident site.

Built With

Share this project:

Updates