Inspiration

Fruit flies pack a complete nervous system into a wiring diagram that science has already mapped — FlyWire’s adult Drosophila connectome. Most “AI brain” demos ignore that map and train a black-box net from scratch. CloudFly started from the opposite bet: treat real synapses as a fixed nonlinear feature engine, put a small trainable head on top, and make the circuit visible when it decides. HackRPI was the excuse to turn that idea into classifiers, biometrics, a town of job-flies, and a live 3D brain you can argue with.

What it does

CloudFly is a stack of fly-backed demos that share one story:

  • Frozen FlyWire LIF → Torch heads for real tasks: email spam/indox, Wisconsin cancer cell features, and low-res face ID / verify (including a Raspberry Pi camera export).
  • engine / flyputer: ask a local Gemma agent to stimulate the connectome — smell cascades, logic gates, add/multiply, compass heading, optic-lobe images, Kenyon sparse memory, and a playable “swat the fly” escape reflex — while a 3D brain lights up and an energy ledger compares spikes to a clocked chip.
  • Web town (cloudfly.design): type or speak one line; it routes to a fly with a job (Puck, Pager, Sieve, Gale, Scoop, Post, Marrow, Bram, …). A ~1,200-neuron circuit fires first; Grok only phrases the voice after the label is fixed. Tool flies call real weather/news/mail APIs. Raised flies can be minted and trained with scores stored across Mongo, Tiger, and Databricks object storage.
  • Generative Scenario Game: This has 4+ fruit fly models that can interact with eachother and the environment, with generative AI generating the environment. The user is the one who controls the scenario and sandbox. It is a simulation sandbox game.

How we built it

  • Pulled the public FlyWire edge list + annotations (~852 MB), built 2-hop subcircuits (~1,200 neurons), and ran current-based LIF with a simple transmitter sign convention.
  • Task encoders drive a fixed receptor pool; spike-count vectors feed small PyTorch MLPs (faces add an L2 embedding for verification). Wiring W stays frozen.
  • Parallel Python pipelines (fly-spam, fly-cancer, fly-faces) plus Pi export bundles so demos run without shipping the full feather.
  • FastAPI train/simulate APIs in engine/, Ollama/Gemma tool-calling for the interactive brain guide.
  • Town UI in React / Three.js / TanStack Start on Vercel: inlined circuit weights, Fish Audio STT/TTS, Grok phrasing, Gemini for Bram, and server functions for classify/train/tools.
  • Gemini + Fruit Fly fusion: Fruit fly inputs could be fused with custom Gemini prompts to achieve desired behavior (only achieved with Marrow, Bram, and the game)
  • Fruit Fly model + simple Neural Network fusion: For simple deterministic tasks, we utilized a small Neural Network along with the fruit fly model (with our own training procedures) to produce highly accurate and specialized fly brain chatbots.

Challenges we ran into

  • LIF saturation on face pixels — raw inputs blew out spikes; fixed with z-score, sparse projection, and current caps.
  • Keeping the story honest: absolute rates are qualitative; gates and paths are motif/topology demos, not biophysics or clinical tools.
  • Separating decision from chat: Grok/Gemini must not override classifier labels.
  • Preventing garbage output: this was prevented by making sure the inputs were understood properly

Accomplishments that we're proud of

  • End-to-end spam, cancer, and face pipelines on real FlyWire dynamics, with hold-out numbers and shareable exports (spam ~96–98% on synthetic mail; cancer ~89.5%; face verify stronger than flat ID).
  • A public town where the neuron view is a live LIF readout for this sentence, not a canned clip.
  • Interactive connectome demos that compose real motifs (AND/OR, compass, Giant Fiber escape) with an energy-sparsity story.
  • Pi-ready face bundle with print targets (best ID recall: Camila Cabello) and a camera UI on port 8787.
  • A clear product loop: speech → config → train → store → infer → speech, with the fly wiring never trained away.
  • Fly models stay relatively useful and intelligent

What we learned

  • A measured connectome is already a powerful feature map if inputs are encoded carefully and heads stay small.
  • Biology-as-constraint forces better UX: show which cells fired, admit toy limits, and don’t let the LLM rewrite the science.
  • Hackathon systems win when demos degrade gracefully — missing API keys or stores shouldn’t kill the circuit story.

What's next for CloudFly

  • Tighten the orchestrator so spoken task specs become schema-valid train configs and Vultr jobs without hand-editing.
  • Stronger face biometrics and more Pi-side classifiers from the same export format.
  • Richer town flies and multiplayer, still with circuit-first decisions.
  • Keep the public site and engine demos aligned so every claim in chat still lights the same wiring on screen.

Built With

Share this project:

Updates

Submission history