Inspiration

The inspiration for Project WingMan stemmed from the critical gap in fully autonomous aerial navigation for GPS-denied, indoor, or subterranean environments. Traditional drone systems rely heavily on GPS and cloud-connected telemetry, making them useless in disaster response, cave exploration, or GPS-spoofed military scenarios. We wanted to build a resilient, edge-computed autonomous mission control system that combines real-time visual simultaneous localization and mapping (SLAM), edge-processed depth perception, and PX4 flight control—all managed through an intuitive web-based interface guided by a local Large Language Model (LLM).

What it does

Project WingMan establishes a fully autonomous drone navigation and monitoring pipeline within a simulated Gazebo environment:Real-Time Visual Mapping: Ingests RGB-D camera streams (/rgbd/image, /rgbd/depth_image) and processes them via RTAB-Map to generate a live 2D occupancy grid map. GPS-Denied EKF2 State Estimation: Fuses external vision-based odometry directly into the PX4 autopilot using specialized sensor bitmasks to ensure stable local position locking without GPS dependency.Autonomous Mission Control Dashboard: Aggregates live MJPEG camera streams, interactive 2D map rendering (ros2djs), and real-time PX4 telemetry coordinates into a unified HTML control panel communicating over WebSockets (rosbridge_websocket on port 9090).LLM-Driven Mission Execution: Translates high-level natural language objectives (e.g., "explore the maze and return to home") from a local LLM into structured JSON navigation commands for autonomous execution.

How we built it

We engineered a multi-layered robotics and web stack from the ground up:Middleware & Networking: Configured ROS 2 Humble with cyclonedds and ROS_DOMAIN_ID=0 to ensure ultra-low latency data transfer between the simulation and control nodes. Perception & Localization Stack: Implemented custom launch scripts combining RTAB-Map (rtabmap_slam), vslam_odometry, and ros_gz_bridge to bridge Gazebo sensor topics to standard ROS 2 message types.Autopilot Integration: Configured PX4 SITL parameters (EKF2_GPS_CTRL = 0, EKF2_HGT_REF = 0, and noise variance tuning) to stabilize external vision data feeds.Frontend Dashboard: Developed a modular web interface using roslibjs, EaselJS, and ros2d to render spatial maps and monitor vehicle health metrics in real time.Challenges we ran intoMiddleware & Daemon Desynchronization: Encountered intermittent ROS 2 CLI daemon timeouts and silent message drops caused by mismatched RMW implementations between background services and user terminals. We resolved this by standardizing environment variables across all active nodes.EKF2 State Convergence Without GPS: Achieving green state estimation flags (xy_valid, z_valid, v_xy_valid, heading_good_for_control) was extremely tricky. Minor mismatches in vertical vision control bitmasks caused EKF2 to silently reject position updates until we isolated horizontal vision fusion and properly tuned sensor noise parameters.Topic Synchronization Bottlenecks: Aligning timestamp queues between high-bandwidth RGB-D streams and custom odometry topics required strict interval tuning (approx_sync_max_interval: 0.05) to prevent frame drops in RTAB-Map.Accomplishments that we're proud ofSuccessfully achieving a stable, fully verified local position: 1 state lock in a completely GPS-denied simulation.Building a responsive, zero-dependency web dashboard that renders live occupancy maps and telemetry side-by-side with camera video streams.Integrating local LLM task generation into a functioning ROS 2 pipeline, laying the groundwork for true cognitive autonomy.What we learnedDeepened our mastery of ROS 2 Humble node architectures, QoS profiles, and CycloneDDS tuning.Gained hands-on experience tuning PX4 EKF2 multi-sensor fusion parameters for external vision-based navigation.Learned how to effectively bridge C++ robotics middleware (rosbridge) with lightweight web technologies (roslibjs, HTML5 canvas) for real-time mission telemetry.What's next for Project WingManHardware Deployment: Transitioning the software stack from simulation (Gazebo/PX4 SITL) onto a physical drone platform equipped with an onboard companion computer (e.g., Jetson Orin Nano) and an Intel RealSense depth camera. Autonomous Exploration Algorithms: Implementing frontier-based exploration combined with local LLM reasoning to dynamically map unknown maze environments without human intervention. Advanced Safety Protocols: Adding automated fail-safes, dynamic obstacle avoidance, and one-click emergency return-to-home commands directly into the web dashboard.

Built With

• autonomous
• c++
• cyclonedds
• dashboard
• drones
• ekf2
• gazebo
• git
• html5
• javascript
• json
• linux
• microxrceagent
• nemotron
• px4
• python
• robotics
• ros2
• roslibjs
• rtab-map
• slam
• telemetry
• visual
• web
• websockets