Inspiration

We wanted to solve the problem of people vanishing in crowds, wether that be little children getting seperated from their parents, and the authorities not able to find them, or scanning a burning building to see if there are still people present. We spent some time, and came up with a solution that forgoes any human risk, and still is able to locate any missing people. This system also is able to advance machine learning, and areal vision, and UAV.

We are proud of this project, and read more below.

What it does

Zwarm is the layer that allows your agents to control real drones in a drone swarm. Zwarm comprises of two stages, you have your "mapping" stage, and you have your "agent controlled" stage. While these are two separate stages, they act coherently with one another. If you prompt your agent to "Find a person wearing a white shirt" it will begin exploring its environment until it either finds a object matching the description, or it maps out the entire room. The mapping is ALWAYS happening in the background (with the use of Neural Radiance Fields (NeRF), a photogrammetry method) where the drone's cameras footage is being streamed in, with bad images automatically filtered out, and used to make an additive 3D "map" of the environment the drone is in. Each drone has 1 camera, and our system is able to be scaled with the number of drones you have.

With Zwarm, you end up being able to allow your models to control a real drone and have it explore areas, find objects, and plan out trajectories, all autonomously. This acts as a MCP to the aerial realm, and pushes the entire field of agents controlling physical objects, UAV research, and addetive photogrammetry forward.

How we built it

To build Zwarm, we first needed drones. We bought 4 of these relatively cheap drones from Amazon (https://www.amazon.com/Beginners-Foldable-Quadcopter-Gestures-Batteries/dp/B0D8LK1KJ3) the Hiturbo S20 drone. This drone was chosen because A: it was the cheapest option available (we also looked into a much smaller drone that was esp32 based, however this wasn't chosen due to the fact that we needed to purchase cameras for all of them, and the addition of the weight and complexity would make it uncertain to actually fly) and B: this drone has a camera as well as an optical flow sensor, which allows us to use the sensors for reconstruction, localization and for drone stabilization. Now you may raise the question, how do you control said drone? Normally you need to use the provided RC controller, however we needed a way to control programatically. To do this, we have a white paper (https://abhiramasonny.com/s2x-protocol-reversal.pdf) of the specific methodology we used to reverse engineer the drones wifi protocol, to be able to control all functions, and read nearly all data, from a computer external to the drone. We then wrote a simple program which allowed us to use the drone through a raspi, which was connected to the drones wifi through its wifi chip, and connected to our macbook through a USB cable, and allowed us to have a link into the drone. After this, we expanded the system to support multiple drones in parallel (with multiple raspberry pis) allowing one Macbook and 4 Raspis to control all 4 of our drones.

After we could control all drones, we put nemotron (hosted locally on my macbook pro) as an agent with vision capabilities where it would take in camera information, as well as a context layer which gathers context from the 3d reconstruction (i.e if it already saw a object, and in the future the user asked to go to that object, it would 'remember' the object as well as the entire path to get to that object) to perform a task optimally. A task could be "find a person whose face looks like (reference image)" for the rescue of a missing person, or a task could be "explore this arena" to build out a 3d map and perform calculations on that map in the future, etc. The sky is the limit here, and the agent can really do ANYTHING on the drone you would want it to.

Challenges we ran into

Photogrammetry is not perfect, regardless of how good your camera is, or how powerful your compute is, getting a very accurate reconstruction of an environment is a very non-trivial problem. To solve this problem, one of our friends (Kathrine Zhang) mentioned the existence of a photogrammetry method, which works with a very few amount of images, called NeRF (Neural Radiance Fields). Compared to the method we were using prior (Gaussian Splatting), NeRF (https://www.matthewtancik.com/nerf) is able to create a much richer reconstruction with much fewer images, which is perfect for our needs. Regardless, a constraint is with any photogrammetry method, you will get a rough image of what the scene is, not an exact 3d model. this is fine for our usecases since we are using them more as a model of history.

Built With

Share this project:

Updates

Submission history