STADO — Voice-Commanded Drone Fleet on Qwen Cloud

Talk to a drone fleet: Qwen Realtime hears you, picks from 38 flight tools, and commands two PX4 drones over MAVLink — behind human-armed safety gates. Field-flown. Fly it live in your browser.

Comment

Inspiration

Ground control stations are keyboard-and-mouse hell. A real drone operator at a flight line has one hand on an RC transmitter, eyes on the aircraft, and wind in the microphone — the last thing they can do is alt-tab through menus to plan a survey. We fly actual hexacopters and quadcopters at Stratofirma, and every field session made the same point: the missing interface for fleet operations is conversation. So we built STADO — a voice agent wired into a production ground control station, where "both drones take off to 30 meters and survey a 300 meter area" is a complete, executable mission order.

What it does

Hold push-to-talk and speak. Qwen3.5-Omni-Plus-Realtime hears the raw audio (no separate ASR stage), decides among 38 flight and mission tools — takeoff, land, orbit, goto, multi-drone survey planning with zone splitting and separation corridors, formation flight, target tracking, geofenced altitude overrides — and the GCS executes them over MAVLink against two PX4 drones. Qwen speaks the outcome, and speaks up proactively when a takeoff completes or a battery runs low. Qwen3.7-Plus handles vision: open-vocabulary target grounding ("track the red truck" seeds the onboard tracker), scene description, license-plate reads, and satellite-tile parcel detection for the survey planner. Qwen3.7-Max writes the end-of-flight mission report.

The underlying GCS is real and field-flown — it flies a physical hexacopter over a SIYI datalink and a quadcopter through a Jetson Orin DDS bridge. The public demo wraps that exact production code around two PX4 SITL drones so anyone can fly the fleet from a browser: https://stado.sfautonomy.com/.

One voice round-trip, end to end:

🎙 PTT ──► /ws/voice ──► Qwen Realtime ──► tool call (1 of 38)
                                              │
                        SAFETY INTERLOCKS ◄───┘
             Ready-for-Flight · max-alt ceiling · home-alt gate
                                              │ pass
                             MAVLink ──► PX4 (×2 drones)
                                              │ ack + telemetry
🔊 spoken outcome ◄── Qwen Realtime ◄── function_call_output

How we built it

  • Voice — the browser streams mic PCM16@16 kHz over a WebSocket to a FastAPI backend, which bridges it to Qwen Realtime's native WebSocket API (wss://…/api-ws/v1/realtime). Manual turn detection (turn_detection: null + buffer commit on PTT release) maps one-to-one onto push-to-talk — the correct UX for command-and-control, because an open mic near a flying drone is a hazard. Barge-in via response.cancel. Qwen's audio contract (16 kHz up / 24 kHz down) matches the browser pipeline exactly, so the frontend does zero transcoding.
  • One tool surface, three models — the 38 tool schemas are declared once and shared by the realtime session, the REST API, and the field system. Vision and reasoning calls go through Model Studio's OpenAI-compatible endpoint with the same DASHSCOPE_API_KEY. One key, one cloud, the entire model surface.
  • The demo container — two PX4 SITL instances, each in its own Gazebo world (shared physics starves the sensor watchdog and PX4 refuses to arm — learned the hard way), plus the backend and nginx, all on one Alibaba Cloud ECS instance in Singapore with a Caddy sidecar for auto-HTTPS. One-command deploy in alibaba/.
  • The trust boundary — every model-initiated action passes deterministic gates before a single MAVLink byte leaves the dispatcher. The model proposes; the gates dispose.

Challenges we ran into

  • Realtime models narrate instead of acting. Qwen would sometimes say "taking off now" without emitting the tool call. Fix: a post-response auto-nudge — if the reply describes an action but no function_call arrived, the bridge injects a system turn demanding the call. Tool-call reliability went from "usually" to "reliably".
  • A real-flight incident made us paranoid. During field testing, a flight command dispatched that the operator had not intended. The result is the Ready-for-Flight gate: a per-vehicle, human-armed software interlock. Voice takeoff is refused — politely, out loud — until the operator arms the pill in the UI. It was validated end-to-end in a live field test the next day.
  • Altitude references lie. A barometer-drift failure mode (home/EKF altitude off by meters) produced a "stuck armed, zero climb" aircraft in the field. The GCS now runs a home-alt sanity gate and a TAKEOFF-mode-first arming sequence that survives drift — the same patch fixes a PX4 SITL edge case in the demo.
  • Edge-cloud voice latency over field WiFi pushed us to keep every per-frame loop off the cloud: tracking and state estimation run on the Jetson at the edge; Qwen handles intent, orchestration, and speech.

Accomplishments we're proud of

  • Sub-500 ms voice round-trip, browser to Qwen to MAVLink and back.
  • Three Qwen models composed into one agent — realtime voice, vision grounding, mission reasoning — behind a single tool schema.
  • The safety architecture is flight-tested, not theoretical: the Ready-for-Flight gate refused an unauthorized voice takeoff in a real field test.
  • 384 backend tests, MIT-licensed, and a live demo any judge can fly right now.

What we learned

Realtime speech models drift into narration under long sessions — the engineering is in detecting and correcting it, not in the inference. And the interesting problem in edge agents isn't "put an LLM on a robot"; it's the seam: what must stay at the edge (reflexes, tracking, failsafes), what belongs in the cloud (language, orchestration, open-vocabulary perception), and the deterministic dispatcher that sits between them and trusts neither.

What's next

Hardening the Jetson DDS bridge (orbit-command relay, honest home-position acks), formation-flight polish, and pilot deployments with inspection and security operators in India and the GCC.

Built With

Share this project:

Updates