snake flies

Inspiration

The MaleCNS connectome release meant something that used to only exist in textbooks--the complete, synapse-by-synapse wiring of a real brain--was suddenly a public dataset anyone could download. That felt too good to leave sitting in a feather file. We didn't want to just visualize it or run an offline benchmark; we wanted to see it actually do something, live, in a way people could watch and even influence themselves. Inspired by the snake game on our hacker badges, we realized it was the perfect test: simple enough that a handful of real sensory neurons and a linear readout could plausibly drive it, but immediate enough that, when the fly dodges a threat or beelines for food, you're watching an actual biological circuit make that call in real time.

What it does

snake flies is a real-time simulation of a fruit fly's actual brain, built from the MaleCNS connectome (165,000+ neurons and their measured synaptic connections), controlling a snake-game avatar as it hunts for food and dodges collisions. Perceptual input from within the game, such as food direction or nearby danger, drives real sensory neuron pathways through a leaky integrate-and-fire model running on the fly's own wiring, and a small trained readout translates descending-neuron activity into left/straight/right moves, so behaviour emerges from actual neural circuitry rather than scripted logic. Audiences can watch multiple flies at once, lesion specific neuron types live to see behaviour degrade, compare the real connectome against a scrambled control, or scan a QR code to steer the fly's learning themselves from their phone. Nothing here is a recording or a script -- it's a real insect brain, simulated neuron by neuron and learning to survive in real time.

How we built it

We started from the original MaleCNS v1.0 connectome( released by FlyWire/Janelia) and turned it into a batched, GPU-accelerated leaky integrate-and-fire simulation in PyTorch, where every one of ~165,000 neurons tracks its own membrane voltage and fires when it crosses threshold, exactly per Shiu et al. 2024's parameters. The game's senses map onto real sensory neuron types (LC10 for food, LC4 and LPLC2 for threat), and a small trained linear readout turns the fly's own descending-neuron spikes into a move, with the connectome itself never touched, so every behaviour we see is the wiring's. On top of that we built a FastAPI WebSocket server that streams the brain's live activity to a React/Three.js dashboard, along with a lesion lab for silencing specific neuron types mid-game and an audience-facing phone controller so anyone in the room can nudge a real neural circuit's behaviour with their own two thumbs.

Challenges we ran into

With the original sensory encoding, one out of every five live-training runs would stall completely and never learn to steer, meaning a demo audience had a real one-in-five chance of watching the fly just fail on stage. Tracing it, we found that the fly's danger senses only looked one cell ahead, not enough warning to avoid dead ends, so we switched to a look-ahead encoder that senses collisions and blocked routes further out. Getting an audience's phones talking to that same live brain was its own problem, since venue WiFi commonly blocks device-to-device connections outright, which would make a QR code pointing at a local IP address simply fail to load; we solved that by routing the phone controller through a public HTTPS tunnel instead, so a scan works from any network, cellular data included. And once phones could reach the fly, we had to make sure a crowd could teach it without ever overriding what it learns on its own — so every vote from a phone is deliberately capped and divided across everyone connected, meaning one person mashing a direction can't outweigh the fly's own experience from food and collisions.

Accomplishments that we're proud of

What we're most proud of is having a real fly brain--165,000 actual neurons, wired exactly as found in nature--play the game live with nobody scripting a single move. We didn't fake a single behavior; every dodge and every hunt for food is the connectome thinking it through in real time. We're just as excited about what we built on top of that: anyone can scan a QR code, pull out their phone, and contribute toward teaching it something, nudging a living neural circuit's decisions with their own two thumbs.

What we learned

We learned that richer sensory input doesn't automatically make a model smarter: a much more detailed encoding never outperformed a simpler one in our tests, which showed that we should validate assumptions first before prematurely adding complexity. We also learned how much care live, shared, multi-user systems demand: real-time constraints and a crowd of inputs all touching the same neural network meant we had to think carefully about fairness and bounds from the start. And working with a real connectome reminded us how much about a biological brain remains a black box, even when you can simulate it neuron by neuron.

What's next for fly snake

Our simulation currently reduces everything the fly senses to just 24 discrete situations, so a natural next step is building a richer retinotopic vision encoder, so the fly is actually seeing the board instead of a simplified summary of it. Our synaptic-training experiment showed a promising direction but the result was statistically inconclusive, so we'd have to run it at a larger scale to get a real answer on whether trainable synapses beat a fixed decoder. We'd also like to push live learning to work reliably with a single fly at normal speed, not just when many brains share one readout, and open up more of the connectome for lesioning and comparison beyond the handful of pathways we currently expose.