Instant Payments Using Voice-Activated Drones
Using Houndify Speech To Text to Understand Intent and Commit Takeoff Action
Using Houndify Speech To Text to Understand Intent and Commit Touchdown Action
Using Houndify Speech To Text to Understand Intent and Commit Transfer Action
Receiving $100 of (fake) cash instantaneously in my bank account
Since the days of yore, we have heard how an army of drones is going to bridge the last-mile gap of accessibility in the supply chain of various products for many people in the world. A good example of this can already be seen with Zipline in Rwanda and Ghana, which is already on its way to success with this model. While the quest to bridging the last-mile gap is well on it's way to getting completed, there are significant challenges in the overall supply chain that have had no evolution. As technology gets more and more advanced, people get more disillusioned by the possible use cases that are not being explored.
The inspiration for this hack very much stems from the fact that the Treehacks Hardware Lab provided us with a drone that did not come with an SDK, or any developer support. And therefore, I decided to build a whole new approach to voice recognition and instant payments that not only tries to address this growing disillusionment with technology, but also provides a use case for drones, a technology that is no longer "unimaginable" or "out of this world" that has not yet been explored: refunds and accessbility.
What it does
With Dronations, you can ask a voice-activated drone to:
TAKE A PICTURE(using gesture)
TRANSFER MONEY TO PERSON IN FRAMENow, you can receive instant refunds for a product or an item you have returned after buying. And people with limited mobility can send payments reliably to people they trust using just simple voice commands delivered to a drone.
How I built it
Since the Snaptain drone provided to me did not come with an SDK, I was left with no choice but to find a way to hack into it. Fortunately, the flimsy drone did not have that much going on in terms of protection from adversarial attacks. After fiddling with the drone for a few hours, it was clear that the best way of trying to break into the drone in order to control it with a computer or connect it to a server would be through the unsecured WiFi it uses to connect to an app on my phone.
I spent another few hours reverse engineering the drone protocol by sniffing packets on the channel and frequency used by this drone to get commands from my phone. For this purpose, I used Airmon-NG on a computer running a distribution of Kali-Linux (thankfully provided by my senior project lab at Stanford). Having figured out how to send basic takeoff, touchdown and image commands to the drone by bypassing its WiFi gave me an immense of control of the drone and now it was time to integrate voice commands.
Out of all the voice recognition softwares on offer, Houndify came through as my top choice. This was because of the intuitive design of adding domains to add more functionality on their app and the fact that the mentors were super helpful while answering questions to help us debug. I build three custom commands for takeoff, touchdown and sending money.
Finally, I integrated Checkbook's API for sending money. The API didn't quite work as expected and it took a while for me to finagle it into making it do what I wanted. However, eventually, everything came together and the result is an end-to-end solution for instant payments using voice-activated drones.
Challenges I ran into
- Wifi reliability issues on the cheap hardware of a Snaptain drone
- Checkbook API's refusal to accept my POST requests
- Sleep Deprivation?
- The endless bad humor of the team sitting next to me
Accomplishments that I'm proud of
- Literally hacking into a drone with little to no prior experience hacking an embedded system
- Sniffing packets over the air to reverse engineer drone protocol
- Going through tens of videos showcasing obscure details of packet sniffing
- Making a voice-activated drone
- Integrating with Checkbook's API
- Making new friends in the process of this hackathon
What I learned
- How to sniff packets on an unsecure HTTP connection
- How to write server level Node.js commands that communicate directly with the terminal
- How to spoof my MAC address
- How to spoof my identity to gain unauthorized access into a device
- How to build an app with Houndify
- How to integrate voice commands to trigger Checkbook's API
- If it exists, there is an AR/VR project of it
What's next for Dronations: Enabling Instant Refunds for Purchase Returns
- Integrate more features from Checkbook's API to provide an end-to-end payment solution
- Understand more edge use cases for drones - I have barely scratched the surface
- Do user research with people with low accessibility to understand if this can become a viable alternative to send payments reliably to people they trust
- Getting in touch with Snaptain and letting them know how easy it is to hack their drones