OmniControl

Inspiration

Drones are taking on real jobs: delivering blood and organs to remote clinics, cleaning up litter, helping fishers. But each job usually needs its own controller, written or trained for that one task. When a job changes, someone has to rebuild the controller. We wanted the opposite: one drone brain that can take any mission in plain English and fly it right away, with no retraining. The HackRPI theme, “In The Clouds,” made that a natural fit.

What it does

SkyHand is a zero-shot behavioral foundation model for a gripper drone carrying a dual-wavelength LiDAR.

You type a mission, like “deliver the organ cooler to the clinic gently and stay out of the no-fly zones”. An LLM, Gemini, compiles it into a safe, checked reward program, with an offline compiler as backup. Long missions become chains of steps, like grab, then carry, then release. The reward becomes a latent task vector, z, computed from a fixed set of states the model saw during training. The same frozen policy network flies the mission. Nothing is trained for the specific task.

There are three scenes:

NYC-style trash pickup: city streets and open terrain with lakes. Long-range organ delivery: between remote clinics, Zipline-style. Multi-drone net fishing: a team of drones handles a net among fish schools, whales, dolphins and turtles.

A web demo shows the compiled plan for approval, live 3D flight, the current step of a multi-step mission, and perturbations like wind gusts.

How we built it

Simulator: our own batched PyTorch simulator with Genesis, with drone dynamics, a gripper with swinging payloads, a net made of physical ropes, terrain and water, and spherical plus downward LiDAR. Thousands of worlds run in parallel on the GPU.
Learning: forward-backward representations with conditional policy regularization (FB-CPR). The model learns, without any rewards, a representation in which any reward maps to a task vector z. Scripted expert demos keep the learned behaviors natural.
Training: SAC. Old checkpoints load into new network layouts exactly, so restarts don’t throw work away. Mission layer: a small reward language with a validator, Gemini compilation with an offline fallback, and step chains with one z per step.
Evaluation: a mission success harness that scores checkpoints the same way every time, on a fixed state pool and fixed worlds.

Challenges we ran into

Reward hacking by the learner: the policy learned that ending an episode, by flying off the map, diving into water or crashing into the net, escaped negative rewards. We redesigned the world so these are soft boundaries with a penalty, not exits.
Getting z right: z inferred from the wrong states gave confident nonsense. Computing it over the training distribution, with centered rewards and a fixed reference pool, fixed it. Discount tuning: with a long horizon, the model couldn’t tell good actions from bad. Going back to the paper’s value fixed that.
Scaling: each update was bound by GPU per-kernel latency, not by data. Profiling led us to stack networks, fuse layers, add mixed precision and finally switch to a lightweight residual architecture.
Rare skills: grabbing an item happened in about 0.3% of training steps, so carrying missions barely learned. That pushed us toward start-state setting, which begins some episodes right at a handle.
Time: every restart cost an hour of progress, and we had one night.

Accomplishments that we’re proud of

A complete pipeline from English, to a reward program, to a task vector, to flight, all running live in the browser.
A frozen policy that does learned behaviors it was never trained on specifically, like hovering at a target height and heading toward named places.
A from-scratch multi-agent simulator with ropes, a net, payloads and LiDAR, fast enough for large-scale RL.
Multi-step missions that run as chains of task vectors.

What we learned

In zero-shot RL, how you infer the task vector matters as much as the model.
Any way for an episode to end that the agent can trigger becomes an exploit.
Rare events need state setting or curricula. More updates on almost no data don’t teach the skill.
Measure everything the same way, every time.

What’s next for OmniControl