Dispatcher: Natural Language Drone Package Pickup

Inspiration

Iron Man's Jarvis, Cortana, R2D2. I really want to see AI come to life in the form of robotics not just sitting behind a screen

What it does

Dispatcher is an Aerial Intelligence platform that lets you gather information and interact with the real world using an UAV. For example if you want to go pickleball but aren't sure there are courts open you can say "go check if the tennis courts at shoreline park are open", and it will break down the task, fly to the location, use visual understanding to inform you if the courts are open.

How we built it

Using Azure AI we were able to leverage Low Latency Large Language models to interpret natural language commands into actionable function calls and real world scene understanding. There is also a Jetson Nano running onboard the drone itself that does the real time object detection and tracking allowing for object counting. As well as a simple to use frontend that allows you to have a video call with a drone!

Challenges we ran into

Most challenging part was reliable responses and function calls from large language models that required a lot of careful prompt tuning

Accomplishments that we're proud of

Especially proud of the realtime speech to open-set object tracking, while it has its limitations this is the first example we've seen in real life.

What we learned

How to leverage the correct model for each task, while GPT 4o is great for function calling and scene understanding it is not great at bounding boxes. So this project was all about finding the correct models to use for each use case

What's next for Dispatcher: Natural Language Drone Package Pickup

I want to expand the potential use cases even more and eventually get to a point where any conceivable request from a human can be executed from the drone

Built With

• azure
• jetson-nano
• mqtt
• opencv
• pipecat
• px4
• python
• yolo