Drones are ubiquitous across many domains and are often used for delivery, rescue, cinematography and entertainment purposes. Given that most UAVs are controlled via traditional remote controllers with limited customizability, we aimed to develop a universal controller for drones that would offer an additional degree of convenience and freedom for customers.
What it does
Using LeapMotion controller, drones can be controlled via simple hand gestures, eliminating the need for an external remote controller. In addition, other technologies such as voice command recognition and image recognition can be added in the future to extend the functionalites of the universal drone controller.
How we built it
Using the DJI Mobile Developer SDK, we extract class files instance fields and methods related to flight control data from the drone. The fields include the aircraft principal axes (pitch, roll, yaw) and x, y, z coordinates in 3-D system.
The LeapMotion and the vision processing are built using Python, and run off a laptop connected to the network and the LeapMotion. The realtime data from the motion detector are then processed in an algorithm that translates hand position into float pX, pY values and sent to the Android application connected to the drone using Spark Java Framework. In the end, the Andriod application re-processes the data into coordinates in aircraft principal axes and the x, y, z axes and feed them into the imported functions in the SDK to set the flight controller data on the drone. The live video feed from the drone's camera is decoded using methods in the SDK as well.
Challenges we ran into
One of the biggest challenges we faced was developing a communication system between the DJI Spark drone and Leap Motion, in regards to Wifi and Bluetooth functionality. Since the Spark drone communicated largely through a mobile SDK that used WiFi as its medium, it was difficult to find a way to have it connect both with the drone and with the computer which controlled the LeapMotion as well as the image processing. Eventually, we solved that by finding new ways to configure our communication framework so that the mobile device could use the same network as the computer, allowing a simple Web Server/HTTP Request communication style, with fairly low ping.
Another problem we've encountered was the instability and low compatibility of the DJI SDK. Its Android documentation was confusing and not specific, and a lot of the sample codes provided in the SDK were bugged (missing interface implementation or imported classes) which we have to edit accordingly. In addition to that, the application can only be run on certain versions of the Andriod system and SDK without crashing. Due to the fact that none of the group members has an Android device, we were originally using AVDs or emulators from various sources including the one in Android Studio. Configuring the SDK on different emulators was time-consuming and unnecessary. Luckily, we got access to a compatible Andriod device in the end and therefore solved this issue.
Accomplishments that we're proud of
We are proud of just how well we were able to utilize the LeapMotion. Though it is a somewhat confusing device at first that seems to randomly disconnect, in the end, we were able to figure it out and utilize its true power.
What we learned
Learning better ways to communicate between multiple computer systems as well as in between processes on the same system was quite informative. Dealing with the many different forms of Bluetooth communication also revealed just how complicated some technologies are under-the-surface. This knowledge will definitely help us in future hackathons, especially if we pursue more ideas that have a physical, robotic aspect.
What's next for Universal Drone Controller
Since most consumers carry their cell phones with them everywhere, we would like to integrate this universal drone controller into a user-friendly mobile app for both Android and iOS. Additionally, we would like to include voice-control functionality and more varied object detection, so the drone could detect and navigate to desired targets autonomously.