WallHAX

Drone Vision AR - Hackathon Tech Stack Summary

Project Overview Real-time drone localization and AR navigation system using visual odometry, ArUco marker detection, and WebXR on mobile devices.

Core Technologies

Backend (Python) Flask + Socket.IO – Real-time bidirectional communication with mobile AR client OpenCV – Visual feature tracking, ArUco marker detection, pose estimation (solvePnP) NumPy – Coordinate transformations and matrix operations Depth Anything V3 – Monocular depth estimation for point cloud generation MediaPipe – Human pose landmarks & segmentation masks Visual Odometry Pipeline Optical Flow (Lucas-Kanade) – Track feature points across frames Essential Matrix + RANSAC – Camera motion estimation Camera Calibration – Intrinsic parameters from MATLAB calibration matrices Thread-based Processing – Asynchronous VO updates without blocking UI AR Frontend (Web/Mobile) Three.js – 3D scene management and virtual object placement WebXR API – Immersive AR mode on iOS/Android DeviceOrientation API – Gyroscope/compass sensor fusion HTML5 Canvas – Real-time camera feed capture and overlay Computer Vision Algorithms ArUco Marker Detection (Calibration Ground Truth) Quaternion ↔ Rotation Matrix Transforms (3D coordinate frames) Epipolar Geometry (Essential Matrix from optical flow) solvePnP (Marker pose in camera frame) Key Features ✅ Real-time VIO tracking at ~30 FPS ✅ Sub-meter accuracy via ArUco marker recalibration ✅ WebXR pass-through camera AR on mobile ✅ Contactless biometrics optional (SmartSpectra API) ✅ Bird's-eye debug visualization (matplotlib live plot) ✅ Multithreaded streaming (camera capture + JPEG encoding)

Built With

• aruco
• opencv
• python
• three.js
• webxr