DRIFTx

Single-Pass Drone Video to Accurate 3D Model Generation System AI-Enabled Georeferenced 3D Reconstruction from a Single UAV flight

Comment

Inspiration

When a disaster hits, responders need a 3D picture of the site fast. Traditional drone mapping can't give it to them. Reliable photogrammetry typically needs 66-85% image overlap, which means multiple flights, hundreds of images and hours of processing. Reported case studies show how heavy this gets: the Kotropi landslide in Himachal Pradesh needed 98 images plus 9 videos for one 3D model, and the Dhanera flood in Gujarat needed 4 separate UAV survey blocks for 10 km².

In reconnaissance and disaster zones you often get exactly one flight. We asked a simple question: what if one continuous video from one pass were enough?

What it does

DRIFTx turns a single-pass UAV video into a georeferenced, metrically scaled, textured 3D model. The same pipeline can be used for disaster response, infrastructure inspection, urban planning and archaeology.

  • Video in, 3D out: 1080p/4K aerial video becomes a point cloud and mesh of terrain, buildings, roads, vegetation and obstacles.
  • Placed on the real map: GPS/GNSS, IMU and flight metadata are fused with the visual trajectory, so models are aligned to world coordinates with recovered metric scale.
  • Export and inspect: results export as GLB, PLY, georeferenced NPZ and CAD-ready geometry. A web viewer supports orbiting, point picking, and distance, area and volume measurement.
  • Multimodal extensions: thermal person detection and geolocation, LiDAR and night-vision inputs, and archaeological feature detection, all in a unified dashboard.
  • DRIFTx Mounted Vibration Sensor (MVP): an ESP32 + MPU IMU sensor that samples at 200 Hz, computes RMS/peak vibration against a baseline, and flags a persistent EVENT after 3 consecutive high-vibration windows. It streams to a live Supabase dashboard for condition monitoring and predictive maintenance.

How we built it

  1. Preprocessing: frame extraction with stride sampling, SLERP pose interpolation, blur rejection (Laplacian variance), near-duplicate removal (pHash + SSIM) and optional dynamic-object masking.
  2. Frozen backbone: Depth Anything 3 (Large 1.1), fine-tuned for aerial views, estimates dense depth and camera pose in a single forward pass. Inference runs on CUDA with BF16.
  3. Geometry refinement: confidence masking, floater and noise filtering, reprojection checks and multi-view depth consistency.
  4. Reconstruction and georeferencing: depth is back-projected to a fused XYZ+RGB point cloud, cleaned with statistical outlier removal, then meshed with Poisson reconstruction. Overlapping windows are merged with Sim(3) alignment, and coordinates go from EPSG:4326 to UTM.
  5. Export and QA: automated reprojection and GPS-alignment checks before export.
  6. Web platform: a FastAPI REST API with async Celery/Redis jobs, Docker, and a React + Three.js WebGL viewer (uploads up to 512 MB).
  7. Hardware: the ESP32 + MPU accelerometer, with an HTTPS POST of JSON payloads to a Supabase (PostgreSQL + Realtime) dashboard.

Challenges we ran into

  • Scale and georeferencing from a single pass: monocular depth is only relative, so recovering real metric scale and aligning to GPS without heavy ground control points took the most iteration.
  • Stitching long videos: processing in overlapping windows meant drift had to be controlled through Sim(3) alignment and overlap checks.
  • Noisy frames: blur, vibration and near-duplicate frames from moving drones hurt reconstruction until we added quality filtering.
  • Accuracy vs. our target: our goal is ≤1 m spatial accuracy. Current GPS alignment RMSE is 1.92 m, so we're still optimizing.
  • Keeping a multi-part system coherent: reconstruction, viewer, thermal/LiDAR modules and the vibration sensor each needed their own pipeline and had to work together.

Accomplishments that we're proud of

  • A working end-to-end pipeline from raw video to downloadable GLB/PLY models, with a live web viewer.
  • Measured results on UAV videos of 4:11-11:18: mean DSM surface coverage of 85.8% (80.6-91.9%), mean reprojection error of 0.543 px, mean GPS alignment RMSE of 1.92 m, and a mean of 90.4% of frames registered.
  • Efficient inference: roughly 3.88 GB of GPU memory on our test runs.
  • A working hardware prototype, with the vibration sensor streaming to a live dashboard.
  • Support from a drone-industry partner (Drona Aviation) through an MoU, and a team that built the full stack end to end.

What we learned

  • Foundation depth models like Depth Anything 3 can replace much of the traditional multi-view pipeline, but they only become useful for mapping once geometry and GPS are fused around them.
  • Data quality control (blur, redundancy, confidence filtering) often improves results more than model changes.
  • Honest validation matters. Reporting reprojection error, GPS RMSE and coverage separately showed us exactly where we stand and what to fix.
  • Building for emergencies means designing around constraints (one flight, limited time, no new hardware) from day one.

What's next for DRIFTx

  • Independent survey validation against ground control to confirm and push toward our ≤1 m accuracy target.
  • Faster processing: TensorRT/FP16 optimization to hit our goal of under 15 minutes per 10-minute video.
  • Large-area and autonomous-drone support for wider coverage.
  • Digital twins and predictive maintenance by combining 3D models with vibration monitoring to catch issues before failure.
  • VR headset visualization for faster decisions in the field.
  • Deeper thermal, LiDAR and multimodal fusion for search and rescue and archaeology.

Built With

Share this project:

Updates

Submission history