ThisFlyPlaysDrums

The fly learning how to play drums from the model

Inspiration

Sam REALLY likes the drums, like... REALLYYY

What it does

It simulates the brain of a fruit fly and makes that simulation play the drums.

How we built it

The brain is the real wiring diagram of the fruit fly, Janelia's MaleCNS connectome: about 166,700 neurons and 24.7 million connections, simulated spike by spike in PyTorch with the fly-brain simulator. Music goes in through the neurons a fly actually hears with, in its antennae, as bursts of spikes for each drum. The leg motor neurons move a physics model of a fly (flybody, in MuJoCo) that holds two drumsticks over a kit we built to its size, and a note only counts if a stick really lands on a pad. Those hits play on a real Roland TD-07. Sam recorded 21 grooves on the kit to teach it, and we let only a few thousand synapses learn, so the rest of the brain is wired the way the real fly's is. A three.js viewer replays each run with the brain lighting up as it plays, and you can hand the fly new sheet music, or play something on the kit, for it to try.

Challenges we ran into

Training was the hard part. Our first three runs never did better than an untrained fly (they did worse), and Sam tracked it down to the simulator's surrogate gradient, the smooth stand-in that lets gradients pass through all-or-nothing spikes. It's so narrow that a resting neuron sends back almost nothing, and the signal has to get through two layers of spiking neurons to reach the legs. Widening it in the backward pass fixed that without changing how the fly behaves. Holding the hi-hat pedal down turned out to be 82% of the error, so now the foot just rests on it. The simulation was also just really slow, about 10 seconds to simulate every 1 second of Sam playing on the kit.

Accomplishments that we're proud of

That we were able to train the fly itself play the drums. Not a model that controls the fly but the fly itself, which is FAR more impressive and difficult.

What we learned

Teaching a fly to play drums is a LOT harder than you'd think. We learned a lot about how GPU servers work, cause we had to populate 2 of them to run the fly. We learned how this new technology, fly brain simulations, work and the many, MANY, complications that come with running it. For example, making sure that the brain was actually learning and not cheating/getting worse with every training session.

What's next for ThisFlyPlaysDrums

Continue to train the fly until it can play the drums well and understand how to better tell what to hit and where to hit. Maybe even train a few flies for a few instruments, make a band.