VisionAiry

Inspiration

Drones are fun to fly but hard to fly well. Sticks, trims, drift and walls all get in the way. We wanted to just tell a drone what to do, the way you'd tell a person: "take off, fly a 1 m square, turn right, land." We also wanted it safe enough to do that indoors on a $30 toy drone. The catch is that an LLM can come up with a plan that sounds right and still flies the drone into a wall. So we decided the AI never gets to fly anything we haven't simulated first, and that the drone should get better every time it flies.

What it does

VisionAiry is a natural-language drone pilot. You type a command in plain English and an LLM turns it into a full flight plan through tool calls. Before a single packet reaches the drone, VisionAiry runs that plan through a live physics simulation. It predicts the path the AI is about to fly and refuses dangerous plans before they happen: leaving the safe area, going through the ceiling, or diving toward the floor. When it refuses a plan, it tells the LLM why, and the LLM replans.

It learns from every flight. VisionAiry records each flight and feeds what it learned into the next one:

the drone's drift, which it trims out automatically;
its real speeds, which go into the AI's planning;
quirks like reversed axes, which it corrects for;
how its controller behaved, so it can retune itself.

The more you fly, the more accurate its predictions and plans get.

While the drone is flying, a real-time safety layer runs on every video frame:

Obstacle guard: object detection plus monocular depth give every object a name, a distance and a time-to-collision (CHAIR #7 1.8 m). Anything in the flight path that you'd reach in under 1.5 s stops the plan instantly.
Position hold: it measures drift from the video and pushes back against drafts and fans. It also adapts its own gains if it starts to wobble.
Self-correction: it detects a reversed axis, a gust it can't beat, lost video or a dying battery, and lands on its own when it has to.
Vision for the LLM: the AI gets labelled camera frames, so commands like "fly to the chair" or "go around anything in the way" work.

How we built it

Reverse engineering: we decompiled the drone's Android app and recovered its Wi-Fi protocol: 8-byte stick frames every 50 ms, the video stream, and the flags for take off, land and emergency stop. Then we wrote our own open client in Python.
AI pilot: it works with any OpenAI-compatible LLM. The model controls each stick as a % for an exact duration, with ramps, and plans a whole command in one sequence. That way it thinks once on the ground instead of hovering and drifting while it waits for the next answer.
Physics simulation: every plan is integrated step by step at 20 Hz using the drone's stick-to-velocity, climb and yaw-rate model. That gives the predicted trajectory, which is checked against the flight envelope before it's committed. A higher-fidelity simulator adds momentum, wind and gusts so we can stress-test AI-generated paths and plot them.
Flight memory: every flight writes a detailed log covering LLM calls, plan steps, guard stops, drift, corrections and telemetry. From that, VisionAiry builds up a profile of the drone:
- its learned trim;
- its calibration (axis directions, camera orientation, video lag, altitude hold);
- its measured speeds.

The profile is loaded on every start. Measured speeds go straight into the LLM's instructions, so its plans and our simulation both reflect how this drone really flies.

Perception: OpenCV for the video and object tracking, plus an ONNX depth model with on-the-spot calibration to get metric distances and 3D positions.
Safety supervisor: a per-frame control layer for the obstacle guard, PI position hold, adaptive gains and watchdogs.
Testing: a fault-injection suite throws 18 failure scenarios at the simulator (drafts, gales, shoves, reversed cameras, 0.5 s of video lag) and checks that the drone handles each one.

Challenges we ran into

LLMs are slow and drones are not. A model takes seconds to answer, and a drone can hit a wall in less than one. That's why we split the work: the LLM plans, the simulation checks the plan, and a fast supervisor reacts.
Cheap drones lie. Axes can be reversed, the trims are bad, the altitude hold is weak, and the video lags. Every drone is different. That's exactly why it has to learn its own drone instead of trusting fixed numbers.
No GPS, no rangefinder, one camera. Every position and distance comes from video and our motion model.
The drone's Wi-Fi has no internet, so the LLM needs a second connection while you fly.

Accomplishments that we're proud of

Plain English goes in and a safe, simulated flight comes out, on a toy drone with no SDK.
The AI's flight path is predicted before it flies, and unsafe plans are caught and replanned automatically.
A drone that gets better the more you fly it: drift, speeds and quirks learned on one flight carry into the next.
We reverse engineered the whole control and video protocol from scratch.
A supervisor that fixes itself: it flips reversed axes, softens a wobbling controller, and raises its push limit against wind.
An 18-scenario fault-injection test suite, plus a simulator that draws exactly where the AI is about to send the drone.

What we learned

With an LLM in the loop, safety can't be a prompt. It has to be physics. You simulate before you act, verify while you act, and learn after you land.
Splitting the work across timescales (seconds for the LLM, milliseconds for the supervisor, flight to flight for learning) is what makes an AI pilot practical.
Most of the work in robotics is handling the ways the hardware disagrees with your model, so the model has to keep learning.

What's next for VisionAiry

Voice control: speak commands instead of typing them.
Deeper learning: fit a full dynamics model to the flight logs, including momentum, lag and how the drone responds to wind.
Closed-loop replanning: compare the simulated path against what the camera actually sees mid-flight and correct on the fly.
Richer goals: "follow me," "inspect that shelf," and multi-drone missions.
Bigger drones: take the same plan → simulate → supervise → learn loop to drones with better sensors outdoors.