One-handed Drone Simulation

One-handed drone control 3D simulator with using IMU sensor with ESP32 board and gesture input, powered by computer vision, WebSockets.

Comment

One-Handed Drone Simulation

Inspiration

Traditional drone control systems require two hands and complex controllers, which can be limiting and unintuitive. We wanted to redesign this experience to make drone interaction more natural, accessible, and efficient.

We reimagined drone control by enabling users to operate the drone with one hand, while using the other hand for gesture-based interactions. This opens up new possibilities for intuitive control, especially in scenarios like emergency response, firefighting, and multitasking environments.

We also focused on solving a key robotics problem: unsafe and expensive real-world testing. This led us to build a full simulation environment for safe training and validation.

What it does

One-handed Drone Simulation is an interactive system that allows users to control a drone using:

  • One-handed physical control (ESP32 + IMU + joystick + buttons)
  • Gesture recognition using the other hand
  • Real-time WebGL drone simulation

Key features

  • Control drone movement (pitch, roll, yaw, throttle) with one hand
  • Use the other hand for gesture-based actions
  • Detect gestures (e.g., holding a fist for 2 seconds) to trigger actions like water dropping
  • Simulate real-world scenarios such as firefighting or rescue missions
  • Capture simulation moments and mint them as NFTs via IPFS (Pinata)
  • Reward users with STARK tokens after successful simulation events

How we built it

Hardware

  • ESP32 board
  • IMU sensor for motion tracking
  • Push buttons and joystick for control input

Simulation

  • Next.js + WebGL for real-time drone simulation and multiple scenes

Gesture Recognition

  • RubikPi running Python
  • MediaPipe + OpenCV for real-time gesture detection
  • Custom logic to detect a fist held for ~2 seconds to trigger water-dropping action

Edge Control and Sensor Fusion (ESP32)

We implemented real-time motion processing on the ESP32 using the BNO08x IMU sensor.

To improve stability and usability, we applied:

  • Low-pass filtering to reduce noise and jitter
  • Deadband and offset correction to reduce drift
  • Score-based classification for directional intent
  • Sensor fusion across IMU, joystick, and button inputs

Motion scoring model

$$ S_{direction} = w_1 \cdot IMU + w_2 \cdot joystick + w_3 \cdot bias $$

The system selects the highest scoring direction as the output, producing smooth and natural control.

IoT Gesture Controller Visualizer

The ESP32 streams processed motion data over WiFi using WebSockets.

The browser-based dashboard receives live data and renders a WebGL drone:

  • The drone stays fixed in position
  • It rotates based on roll, pitch, and yaw
  • Updates happen in real time with low latency

This creates a digital twin for training and simulation.

Gesture Recognition (Qualcomm Robotics Integration)

On RubikPi, we use:

  • MediaPipe for real-time hand tracking
  • OpenCV for gesture processing
  • Custom logic for timed gesture detection

A fist held for ~2 seconds triggers a high-level action such as water drop in simulation.

We combine multiple input streams:

  • IMU motion sensing
  • Joystick + button control
  • Vision-based gesture recognition

This forms a unified robotics control system using sensor fusion and real-time decision making.

Blockchain Integration (Solana)

  • Created a custom token (STARK) using Token-2022
  • Uploaded NFT assets (images + metadata) to IPFS via Pinata
  • Minted NFTs based on simulation events
  • Built a reward system where users receive STARK tokens after completing actions

MLH Track - Best Use of Solana

Solana Implementation Flow

Flow of Solana Implementation

We used Solana to connect real-time simulation events with on-chain assets:

  • Minted NFTs representing simulation captures using IPFS + Solana
  • Created and deployed a custom token (STARK) with metadata
  • Built a reward system where users earn tokens after successful simulation events
  • Demonstrated real-time wallet updates through Phantom

spl-token 2022 command-line "STARK" Coint minting
"STARK" on Solana Devnet
Metadata of "STARK" Token
Inspection Scene NFT Minted on Solana
Inspection Gallery

Espressif Innovation in Smart Home / City

Built on ESP32 hardware, our system demonstrates a full IoT robotics pipeline.

  • Real-time IMU sensor fusion
  • Low-latency WebSocket streaming
  • Edge filtering and motion stabilization
  • WebGL digital twin visualization

Smart city applications

  • Firefighting drone coordination
  • Search and rescue systems
  • Infrastructure inspection
  • Emergency response training

Qualcomm Robotics

We implemented a robotics control system using multi-modal input fusion.

  • Vision: MediaPipe + OpenCV gesture detection
  • Motion: IMU-based orientation tracking
  • Manual input: joystick + buttons

Robotics principles

  • Sensor fusion across multiple input streams
  • Real-time control loops
  • Edge inference on RubikPi
  • Gesture-to-action mapping

Analog Devices Sensor Fusion

We use sensor fusion to stabilize motion control:

  • IMU filtering for noise reduction
  • Deadband for drift suppression
  • Offset correction for calibration
  • Weighted classification for direction selection

Best Software Hack

A full-stack robotics simulation platform combining:

  • Embedded hardware control
  • WebGL simulation
  • Gesture recognition
  • WebSocket streaming
  • Blockchain reward system

Most Likely to Be a Successful Startup

A scalable robotics training platform for:

  • Drone pilot training
  • Public safety simulation
  • Industrial inspection
  • Robotics education

Best AI Wearables

A wearable-style robotics interface using:

  • IMU motion control
  • Gesture-based commands
  • Hands-free interaction
  • Natural human-machine control

Challenges we ran into

  • ESP32 serial instability
  • IMU calibration and drift correction
  • Multi-input synchronization
  • Gesture recognition optimization
  • Solana Token vs Token-2022 complexity
  • NFT metadata consistency

Accomplishments that we're proud of

  • One-handed drone control system
  • Full hardware → simulation → blockchain pipeline
  • Real-time gesture interaction system
  • Digital twin simulation environment
  • NFT minting and token reward system
  • Verified blockchain integration

What we learned

  • Embedded systems with ESP32 and IMU sensors
  • Computer vision with MediaPipe and OpenCV
  • Sensor fusion and robotics control systems
  • WebGL simulation development
  • Solana blockchain development
  • Human-computer interaction design

What's next

  • Improve gesture recognition with ML models
  • Expand simulation scenarios
  • Add leaderboards and scoring systems
  • Integrate with real drones
  • Enhance NFT replay metadata
  • Improve system latency and robustness

Built With

Share this project:

Updates