Fly Saber

Inspiration

After the MaleCNS connectome was released, a wave of viral demos have connected the fly brain to games like Doom, Beat Saber, Minecraft, and Mario 64. These demos have two caveats:

You can only watch. The brain is a fixed performance. You can't lesion a cell type, blind an eye, change its arousal, or rewire it, and can't ask what the brain circuitry is really doing
Some don't even truly play. Popular Beat Saber demos drove its motor output by replaying movements, not by responding to what was on screen. The brain looked like the player, but it wasn't acting on what it saw.

We wanted both fixed: a fly that perceives the blocks through a compound-eye view and chooses its own moves, and a set of controls that act on that brain while it plays.

What it does

FlySaber runs a simulated fruit fly brain on a recreation of popular rhythm game Beat Saber. Blocks approach in a 4x3 grid to the beat of a song, and the fly must slash the right ones in the right direction.

It plays from vision. The fly sees only a rendering of the scene through a hexagonal compound eye. The brain gets no game state, and its output neurons choose the saber positions and cuts.
You can change the brain mid-song.
- Turn off entire cell types
- Force different neurons groups on or off.
- Create visual noise.
- Mood sliders for neuromodulators: octopamine and dopamine raise gain, and serotonin lowers it.
You can watch it think. A live 3D brain map shows neuron activity as the game runs.

How we built it

Connectome pipeline

We started from the MaleCNS v1.0 public bulk files: annotations, neurotransmitters, and synapse weights (1.1 GB).
We extracted a 19,462-neuron, 337k-edge subgraph (6.1M synapses). It covers photoreceptors, lamina, medulla, T4/T5 motion detectors, lobula columnar and projection neurons, central neurons, and 1,310 descending neurons that act as the motor output.
Edge signs come from neurotransmitter identity. Acetylcholine is excitatory, and GABA and glutamate are inhibitory. Histamine is treated as inhibitory, because photoreceptors are 100% histaminergic and otherwise the eyes would be disconnected from the brain.
We kept every second retinotopic column to fit the neuron budget. We inferred the missing photoreceptor and T4/T5 columns from each neuron's strongest synaptic partner.

Brain model

A JAX rate network runs at 30 Hz, one game frame split into 4 Euler substeps.
Per-neuron timescales and biases are trainable. The biases stand in for the input from the neurons we cut, because each kept neuron receives only about 20-54% of its real input inside the subgraph.
The mood system multiplies each neuron's output by a gain driven by the modulatory neurons, capped at about 2.7x.

Training

We used behavior cloning from a scripted expert player on normalized community song maps.
It trained on UF HiPerGator via SLURM.

Experience and engineering

FastAPI and WebSockets serve live sessions to a React, TypeScript, and three.js front end.
CI runs ruff, mypy, pytest, Playwright desktop and mobile tests, and a Docker test on every pull request.

Hosting

The backend (FastAPI plus the trained brains, in Docker) runs on a Vultr server. A Cloudflare Tunnel gives it a hostname, and a Cloudflare Worker serves the front end and proxies live traffic to it.

Accomplishments that we're proud of

It really plays from vision. On songs held out from training, the demo fly hits 32.9% of notes. Taking away its senses breaks it (as expected):

Looming detectors lesioned: 14.1%
One eye blindfolded: about 7%
Motion detectors (T4/T5) lesioned: 2.1%
Both eyes blindfolded: 0%

The brain wiring actually matters Yet there was a caveat, this is hit rate on songs, by brain:

Real wiring: 31.4%
Shuffled signs: 20.0%
Shuffled within layers: 51.8%

Unexpectedly, shuffling partners within layers actually beats the real wiring. Our explanation is that the real circuit compresses vision into signals that matter to a fly, like looming anywhere in view, and discards the where-on-screen detail this game rewards. We report it as found.

What we learned

Modeling the brain is hard, understanding some basic cognitive science is really hard
CI has the trained fly play a full session inside the Docker container, so a broken deploy fails before it ships.
The rate network runs at about 4 ms per frame on a laptop CPU, so no GPU is needed to serve the live demo.
Cloudflare Workers can't fetch a bare IP or a non-standard port, so the Vultr backend sits on port 80 behind a Cloudflare Tunnel.