Inspiration
Small drones are moving into the places where the air is worst: between buildings, under bridges, behind ships. There, gusts and wake vortices hit without warning. A drone usually notices a gust only after it has been pushed off course. CFD could tell it what is coming, but a solve takes minutes. A drone has milliseconds.
So we asked: what if a drone could see the air around it, fast enough to act on it?
What it does
AeroAgent rebuilds the invisible airflow around a drone and decides how to fly it:
- Sees the air: a Fourier Neural Operator (FNO) reconstructs the full 2D flow field (velocity, pressure, vorticity) from just 32 pressure sensors in 6 ms.
- Reacts like a reflex: the predicted wind force is fed forward into the flight controller at ~50 Hz, so the drone leans into a gust before it is pushed off course.
- Thinks like a pilot: NVIDIA Nemotron, served on Nebius Token Factory, reads a 30-number summary of the flow about twice per second and returns a strict JSON intent (mode, gain, tilt limit). A deterministic validator clamps it before it reaches the controller.
Everything is evaluated in simulation, on airflow the model never saw during training.
How we built it
Three tiers, each running only as fast as it can be trusted (the same fast/slow split as NVIDIA GR00T N1):
| Tier | Rate | What runs |
|---|---|---|
| Inner loop | 240 Hz | PID attitude/position control (gym-pybullet-drones) |
| Reflex | ~50 Hz | 32 sensors → FNO → wind at the drone → feedforward |
| Supervisor | ~2 Hz, async | Nemotron 3.5 Lightning on Nebius Token Factory, in its own process |
1. Synthetic CFD data. No open dataset fits, so we wrote a D2Q9 Lattice-Boltzmann solver (flow past a cylinder, Re 100–220, randomized gust pulses) and generated 41 runs, about 12,000 snapshots.
2. The surrogate. We implemented an FNO in PyTorch (6.3 M parameters, runs on Apple MPS) and benchmarked it against NVIDIA PhysicsNeMo's FNO at a matched configuration. Each Fourier layer mixes information globally in frequency space:
$$ v_{t+1}(x) = \sigma\Big( W\, v_t(x) + \mathcal{F}^{-1}\big( R_\phi \cdot \mathcal{F}(v_t) \big)(x) \Big) $$
Inputs are 8 frames of Voronoi-encoded sensor readings. To keep it physics-informed, the loss adds an incompressibility penalty:
$$ \mathcal{L} = \frac{\lVert \hat{y} - y \rVert_2}{\lVert y \rVert_2} + \lambda \, \lVert \nabla \cdot \hat{\mathbf{u}} \rVert_2^2 $$
Relative L2 error on 4 held-out runs: u 0.067, v 0.343, p 0.201.
3. Flow-aware control. The simulator applies the true wind; the controller only sees the FNO estimate. Following Neural-Fly, the predicted drag is fed forward through the velocity channel, which avoids integrator wind-up:
$$ \hat{\mathbf{F}}^{\mathrm{wind}} = \tfrac{1}{2}\rho\, C_d A \,\lVert \hat{\mathbf{v}}^{\mathrm{rel}} \rVert\, \hat{\mathbf{v}}^{\mathrm{rel}} $$
4. The Nemotron supervisor. Nemotron never sees the grid. It gets about 30 numbers and must answer with a strict JSON schema. A validator clamps every value, rate-limits changes, discards stale answers, and latches a safe hold after three failures. A rule-based guardrail underneath acts instantly, so flight performance does not depend on LLM latency.
Challenges we ran into
- The LLM can't be in the loop. We measured Nemotron at 0.6–1 s per decision (and one model drifting from 1.1 s to 11 s over a day). The redesign into a fast reflex plus an asynchronous supervisor was a better architecture, not a compromise.
- A feedback loop we taught the model ourselves. One few-shot example made Nemotron treat normal turbulence as dangerous and retreat, which created real error. We fixed it with calibrated features, a separate hover-error metric and a 25 cm offset bound.
- LLM latency leaked into the control loop when the client ran in a thread. Moving it into its own process fixed it.
- No model to download. Pretrained neural operators map full fields to full fields. None reconstructs a field from a handful of sensors.
- Tooling friction: PyBullet doesn't build with pip on macOS, and two OpenMP runtimes clashed. We documented the fixes in the README.
Accomplishments that we're proud of
Hover error in the wake, same seeded gust, real-time closed loop (RMS / max):
| Configuration | RMS error | Max error |
|---|---|---|
| PID only | 3.10 cm | 5.65 cm |
| + FNO feedforward | 0.66 cm | 1.72 cm |
| + Nemotron 3.5 Lightning | 0.61 ± 0.02 cm | 1.64 ± 0.13 cm |
| + Nemotron with rule guardrail | 0.55 ± 0.00 cm | 1.29 ± 0.00 cm |
- −82 % hover error vs. PID on average (−77 % to −87 % across 4 unseen runs), about 93 % of what perfect wind knowledge would give.
- Robust: −65 % with a ±30 % wrong drag model, −80 % with 20 % sensor noise, −73 % with 100 ms sensor delay.
- 88 of 88 Nemotron decisions were valid JSON across 9 real-time runs.
- In one run Nemotron answered in 1.6 s and missed the gust entirely, and the guardrail still held the drone to 0.55 cm.
- Honest control experiment: compensating only the mean wind already removes 60–76 %. The FNO's value is in the gust, where it cuts the remaining error by another 57–79 %.
What we learned
- Neural operators fit robotics: the physics moves into training, and inference is one forward pass.
- LLMs belong above the control loop. They are good at judgment and bad at being a controller.
- Measure everything. Several of our assumptions about latency and prompts did not survive their first benchmark.
What's next for AeroAgent
- Deploy the FNO as a Nebius Serverless endpoint and train it as a Serverless Job.
- 3D flow fields (D3Q19 data, 3D FNO).
- Short-horizon gust forecasts so the drone can anticipate.
- Real hardware: a Crazyflie in front of a fan.
Log in or sign up for Devpost to join the conversation.