Inspiration

Most swarm demos stream every robot's position over the network. That stops scaling after a handful of robots and falls apart the moment the Wi-Fi does. We wanted to show the other approach: broadcast one tiny shared rule, let every robot work out its own movement, and keep the swarm going when packets go missing or the network disappears completely. Making it a musical performance you conduct with your hand fitted the COMPOSITION theme.

What it does

  • Conduct with your hand. Hold your hand over an Ultraleap sensor: height sets the tempo (80–150 BPM), and a fist, open hand or pinch picks the dance (pulse, orbit or figure-eight).
  • One tiny packet per beat. The laptop hub sends a single packet of about 100 bytes (beat number, tempo, dance rule and four notes). A physical mBot2 robot and its virtual siblings all compute their own positions from that same packet and play the same note.
  • Packet loss. At 60% loss the robot drifts slightly behind its dotted "target" ring, then snaps back into phase as soon as the next packet arrives, using sequence and step numbers.
  • Blackout. After 600 ms with no packets the robot switches to autonomous mode: amber LEDs, dancing on its own beat clock, avoiding obstacles with its ultrasonic sensor, and responding to people (clap to change move, cover it to spin).
  • Scale. One button grows the virtual swarm from 5 to 501 robots, all still driven by the same single packet: under 1 kB/s for the whole swarm against about 481 kB/s if every robot streamed its position. Roughly 600 times less traffic.
  • AI conductor. Gemini picks the next 8-beat phrase (notes, key and dance) from how your hand is moving and shows its one-line reason on screen. ElevenLabs announces state changes ("Packet loss detected", "Blackout. Autonomy engaged", "Swarm resynchronized"). Every beat is logged to Tiger Data (Timescale) for the history chart.
  • Run Demo. One click walks through synced → 60% loss → blackout → resync with voice callouts.

How we built it

  • Hub (Python, FastAPI): UDP broadcast to the robot, WebSocket to the dashboard, packet-loss injector and blackout switch, Ultraleap through the LeapC bindings, Gemini over REST, ElevenLabs text-to-speech, and asyncpg into a Timescale hypertable. Covered by a pytest suite.
  • Robot (MicroPython on mBot2 / CyberPi): a non-blocking loop with a 600 ms watchdog, phase resync from step numbers, an internal beat clock, ultrasonic, light and sound sensing, and buzzer notes.
  • Dashboard (single HTML page): a canvas digital twin, a Three.js lattice view and a Web Audio synth playing the same notes as the robot's buzzer.
  • Entire: we tracked our AI-assisted coding sessions with Entire and uploaded the checkpoints.

Challenges we ran into

  • Wi-Fi. The CyberPi only joins 2.4 GHz networks, and the venue network wouldn't give it an IP address even with a registered MAC. So in our live demo the physical robot runs in its blackout (autonomous) mode, and the networked half of the system is shown in the digital twin, which uses the same rules and packet logic. It is the exact failure we designed for.
  • Timing. At slow tempos a beat lasts longer than the robot's 600 ms watchdog, so the hub repeats each packet every 150 ms, and the dashboard had to count those repeats too to avoid false "packet loss" warnings.
  • Rate limits. Asking Gemini for a new phrase every 8 beats used up our free quota, so we paced the calls and keep the last real phrase in between.

Accomplishments that we're proud of

  • One small packet drives everything: the robot, the twin, the music and a 501-robot swarm.
  • The robot degrades gracefully from synced to drifting to fully autonomous and back.
  • Three people built hardware, hub and dashboard in parallel against shared data contracts.

What we learned

  • Deterministic shared rules beat streaming state, for bandwidth and for resilience.
  • Test on the real network early. Hardware and venue Wi-Fi were harder than any of the code.

What's next for Resilient Swarm Maestro

  • More physical robots on a network we control.
  • Robots that resync from each other, not only from the hub.
  • Bluetooth as a backup link.

Built With

Share this project:

Updates