Inspiration
Fruit flies pack a complete nervous system into a wiring diagram that science has already mapped — FlyWire’s adult Drosophila connectome. Most “AI brain” demos ignore that map and train a black-box net from scratch. CloudFly started from the opposite bet: treat real synapses as a fixed nonlinear feature engine, put a small trainable head on top, and make the circuit visible when it decides. HackRPI was the excuse to turn that idea into classifiers, biometrics, a town of job-flies, and a live 3D brain you can argue with.
What it does
CloudFly is a stack of fly-backed demos that share one story:
- Frozen FlyWire LIF → Torch heads for real tasks: email spam/indox, Wisconsin cancer cell features, and low-res face ID / verify (including a Raspberry Pi camera export).
engine/flyputer: ask a local Gemma agent to stimulate the connectome — smell cascades, logic gates, add/multiply, compass heading, optic-lobe images, Kenyon sparse memory, and a playable “swat the fly” escape reflex — while a 3D brain lights up and an energy ledger compares spikes to a clocked chip.- Web town (cloudfly.design): type or speak one line; it routes to a fly with a job (Puck, Pager, Sieve, Gale, Scoop, Post, Marrow, Bram, …). A ~1,200-neuron circuit fires first; Grok only phrases the voice after the label is fixed. Tool flies call real weather/news/mail APIs. Raised flies can be minted and trained with scores stored across Mongo, Tiger, and Databricks object storage.
- Generative Scenario Game: This has 4+ fruit fly models that can interact with eachother and the environment, with generative AI generating the environment. The user is the one who controls the scenario and sandbox. It is a simulation sandbox game.
How we built it
- Pulled the public FlyWire edge list + annotations (~852 MB), built 2-hop subcircuits (~1,200 neurons), and ran current-based LIF with a simple transmitter sign convention.
- Task encoders drive a fixed receptor pool; spike-count vectors feed small PyTorch MLPs (faces add an L2 embedding for verification). Wiring W stays frozen.
- Parallel Python pipelines (
fly-spam,fly-cancer,fly-faces) plus Pi export bundles so demos run without shipping the full feather. - FastAPI train/simulate APIs in
engine/, Ollama/Gemma tool-calling for the interactive brain guide. - Town UI in React / Three.js / TanStack Start on Vercel: inlined circuit weights, Fish Audio STT/TTS, Grok phrasing, Gemini for Bram, and server functions for classify/train/tools.
- Gemini + Fruit Fly fusion: Fruit fly inputs could be fused with custom Gemini prompts to achieve desired behavior (only achieved with Marrow, Bram, and the game)
- Fruit Fly model + simple Neural Network fusion: For simple deterministic tasks, we utilized a small Neural Network along with the fruit fly model (with our own training procedures) to produce highly accurate and specialized fly brain chatbots.
Challenges we ran into
- LIF saturation on face pixels — raw inputs blew out spikes; fixed with z-score, sparse projection, and current caps.
- Keeping the story honest: absolute rates are qualitative; gates and paths are motif/topology demos, not biophysics or clinical tools.
- Separating decision from chat: Grok/Gemini must not override classifier labels.
- Preventing garbage output: this was prevented by making sure the inputs were understood properly
Accomplishments that we're proud of
- End-to-end spam, cancer, and face pipelines on real FlyWire dynamics, with hold-out numbers and shareable exports (spam ~96–98% on synthetic mail; cancer ~89.5%; face verify stronger than flat ID).
- A public town where the neuron view is a live LIF readout for this sentence, not a canned clip.
- Interactive connectome demos that compose real motifs (AND/OR, compass, Giant Fiber escape) with an energy-sparsity story.
- Pi-ready face bundle with print targets (best ID recall: Camila Cabello) and a camera UI on port 8787.
- A clear product loop: speech → config → train → store → infer → speech, with the fly wiring never trained away.
- Fly models stay relatively useful and intelligent
What we learned
- A measured connectome is already a powerful feature map if inputs are encoded carefully and heads stay small.
- Biology-as-constraint forces better UX: show which cells fired, admit toy limits, and don’t let the LLM rewrite the science.
- Hackathon systems win when demos degrade gracefully — missing API keys or stores shouldn’t kill the circuit story.
What's next for CloudFly
- Tighten the orchestrator so spoken task specs become schema-valid train configs and Vultr jobs without hand-editing.
- Stronger face biometrics and more Pi-side classifiers from the same export format.
- Richer town flies and multiplayer, still with circuit-first decisions.
- Keep the public site and engine demos aligned so every claim in chat still lights the same wiring on screen.