Constellate

Sketch or say a drone light show. Nemotron choreographs it, a flight checker and a wind simulation make it safe and certify it, and your phone becomes the wind for a live test flight.

Comment

Drone light shows are motion graphics made of physical things: a hundred quadrotors with thrust limits, batteries and wind to fight, flying metres apart in front of a crowd. Constellate lets you draw a shape or say "a red heart that slowly spins, then our logo, then THANK YOU in gold". NVIDIA Nemotron writes a show script, a deterministic compiler turns it into collision-free drone paths, and a flight checker and a physics simulation of the swarm in wind find what would go wrong, with the exact fix. Nemotron applies the fixes until the show is certified, a Tavily site briefing sets the limits from the venue's real rules and weather, and in live mode the phone in your hand is the wind.

What it does

  1. Brief the site. Type a venue and a date. Nemotron Nano writes four searches (altitude limit, crowd setback, nearby airspace, typical wind for the month), Tavily runs them, and Nemotron Super reads the results into numbers, each with a source link and a verbatim quote. For Marina Green, San Francisco: a 122 m ceiling ("400 feet AGL", flyusi.org), a 61 m crowd setback ("a 200 ft buffer to the audience", a local drone-show operator) and a 5.1 m/s typical wind (windy.app), so an 8 m/s design wind. The app checks every quote against its source; an unverified number may only tighten a limit, never loosen it.
  2. Design. Draw on a night-sky canvas, drop in an SVG logo, type or say what should happen, pick the fleet size. Nemotron Ultra writes a show script: formations (presets, your strokes, your logo with its colours, text), transitions (direct, a staggered ripple, or expand-then-form), spins and colour changes. Artistic choices only.
  3. Compile. The compiler assigns drones to each formation with the Hungarian algorithm and flies them along synchronized straight lines, which CAPT (Turpin, Michael & Kumar, 2014) proves collision-free when formations are spaced well enough. It writes a MotionLang document, one actor per drone, so the same validator, baker and player as our other MotionLang apps check and play it.
  4. Check. A flight checker measures the planned tracks in metres: separation (a spatial index per frame), speed, acceleration, airspace, formation density and battery time. Every problem names the step and carries the fix as a number: "d0 reaches 32.5 m/s during heart_spin; limit 8 → set heart_spin's duration from 8 to 34.5 s". The duration comes from physics, not a guess: required acceleration scales with 1/T².
  5. Simulate. Each drone is a point-mass quadrotor with a 40° tilt limit, thrust/weight 2 and drag, flown by a PID controller with acceleration feedforward through steady wind plus correlated gusts that differ from drone to drone. 84 flights (7 wind speeds × 12 gust seeds) take about 3 s. Drift, near misses and battery at the design wind become more problems with hints.
  6. Repair. Nemotron Nano (reasoning off) turns the problems into JSON-pointer edits on the script in 1–3 s; if a round doesn't help, Nemotron Ultra with reasoning takes the next one. The model can't touch the limits or the fleet size, so it can't make a show pass by loosening the safety envelope.
  7. Certify and watch. A flight certificate ("Flyable up to 8 m/s: worst separation 1.63 m, max drift 0.5 m, 90% battery left, ≥ 29.7 m from the crowd"), a wind × gust heatmap of every simulated flight, and a night-sky video of the simulated flight seen from the ground.
  8. Be the wind. Go live: the swarm flies the show in real time while a phone, paired by QR code, is the wind. Tilt it to steer a steady wind, shake it for a gust. Streaks show the wind, each drone's planned spot stays as a faint dot with a line to where the wind pushed it, and near misses are circled.

Inspiration

Our other hackathon projects are built on MotionLang, a declarative 2D motion language whose validator gives errors written for an LLM repair loop. A drone light show is the most physical motion graphic there is: the formation is a vertical picture, each drone is an actor, and the validator's job becomes safety. Show design today is slow, expensive and safety-critical, and the creative part (what should the sky show?) is tangled up with the part that must be exactly right (will two drones meet?). We wanted the model to do only the first, and code and physics to do the second, then hand their findings back to the model.

How we built it

  • Models (all on Nebius Token Factory): Nemotron 3 Ultra 550B writes show scripts (JSON mode, 2.5–4 s) and takes over repairs that stall (reasoning on); Nemotron 3 Nano 30B (reasoning off) turns checker and simulator hints into edit ops (1.2–3 s), applies user tweaks and writes the briefing searches; Nemotron 3 Super 120B reads search results into cited limits (~3.5 s for ~4k tokens). Every call's model, latency and tokens show in the app's agent log.
  • Tavily: four runtime searches per site briefing, and the results set limits the simulator enforces.
  • MotionLang: the compiler's output is a MotionLang document, validated and baked by the shared library; the browser plays it with MotionLang's own player.
  • Compiler and checker: SciPy's Hungarian solver and k-d trees, NumPy, Pillow rasterization for formation sampling (arc-length sampling for outlines, k-means plus relaxation for fills).
  • Simulation: a vectorized NumPy quadrotor swarm, run in a process pool for the sweep and stepped in real time for live mode; imageio-ffmpeg renders the night-sky MP4 from simulated positions with a ground-level camera.
  • App: FastAPI with streamed progress (NDJSON) and WebSockets for live mode, vanilla JS with no build step, a phone pad page using DeviceOrientation and DeviceMotion, served over Tailscale HTTPS so phones may use their sensors; a segno QR code pairs the phone.
  • Tests: 35 tests with a scripted fake model, plus one live test: samplers, CAPT separation, every checker code and its hint (applying the hints fixes the unsafe example), the sim and sweep, the repair loop and its protected paths, quote verification, the web API, video, and the live WebSockets.

Challenges we ran into

  • Hints that fought each other. A ring too dense for 100 drones got "make it bigger" from the density check and "make it smaller" from the airspace check, and the repair loop bounced for four rounds. Capping every size hint at what fits the airspace, and suggesting a filled shape when even that isn't enough, fixed it.
  • A physical limit the model mustn't touch. A drawn line can't hold 100 drones at safe spacing, and the first hint said "use 50 drones", which the model isn't allowed to change. A line can now be a band of drones, and the hint says so.
  • Making wind visible. The first live mode was physically right and visually boring: a good controller cancels steady wind in a second or two, and a metre of drift is a few pixels. Wind streaks, ghost dots at the planned positions and gustier, more varied wind made the effect obvious without faking the physics.
  • Serverless Jobs. The sweep was meant to run as Nebius Serverless Jobs, but those need a separately billed AI Cloud project: Token Factory's project id is rejected, and creating a job before billing fails as a bare PermissionDenied even as a dry run. We kept the job launcher (with a guard against the CLI's H100 default) and ran the sweep locally.
  • Real phone sensors. Motion sensors need HTTPS and the phone on the same network path, which meant Tailscale on the phone; the QR code also had to be scalable, not fixed-size, to scan.

Accomplishments that we're proud of

  • A clean split of responsibility: the model chooses, the compiler guarantees what it can, the checker and the simulator verify the rest, and every finding comes back as a fix the model can apply.
  • One repair round, in under two seconds, turns a show that would fly at 32 m/s and leave the airspace into one certified for 8 m/s wind.
  • Cited limits from real sources, checked against those sources, that can only make the show safer.
  • A phone in your hand as a physical input to a physical simulation, live.

What we learned

  • Hints with computed numbers turn a small, fast model into a reliable repairer; the expensive model is only needed when a fix doesn't take.
  • Safety envelopes must be enforced by the app, not requested in the prompt.
  • Physically faithful and visually legible are two separate requirements, and the second needs its own design.

What's next

  • Export to drone-show formats (waypoints with LED colours per drone) and a hardware-in-the-loop test with a few real drones.
  • 3D formations (MotionLang is 2D today; the simulator is already 3D).
  • The sweep on Nebius Serverless Jobs for thousand-drone shows, and wind fields from real forecasts instead of a single design wind.
  • The motion wand: perform a path with the phone and let Nemotron turn it into show steps.

Built With

  • deviceorientation-/-devicemotion
  • fastapi
  • imageio-ffmpeg
  • javascript
  • nano-30b)
  • nebius-token-factory
  • numpy
  • nvidia-nemotron-3-(ultra-550b
  • pillow
  • pydantic
  • pytest
  • python
  • scipy
  • segno
  • super-120b
  • tailscale
  • tavily
  • web-speech-api
  • websockets
Share this project:

Updates

Submission history