Inspiration We wanted to explore how different kinds of intelligence could work together to help a robot tidy a room. RL-BOT brings together agent-based planning, learned motion policies, and fruit fly brain connectivity on a simulated BracketBot.
What it does Our robot balances, navigates around furniture, and performs separate pick-and-place demos. We organize manipulation around three controllers:
- Fable agent control: selects robot skills to handle colored cubes.
- ACT (Action Chunking with Transformers): the planned controller for the red-ball task, with its integration scaffold ready and backend pending.
- Flybrain: a learned policy for moving a blue cube onto a rectangular target.
Flybrain uses 512 mapped neurons and 8,688 connections from FlyWire. Robot observations enter through adapters attached to 64 afferent neurons, activity propagates through central and visual-projection neurons, and 64 descending neurons feed an output adapter that produces movement commands. The replay makes nodes glow with their model activation, letting viewers explore individual neuron IDs and connections. These are artificial activations using real connectivity data.
How we built it We converted BracketBot’s model into MuJoCo, added balancing and arm control, and built a room with three task stations. Navigation combines A* planning with controlled curves, while ROS 2 and SLAM Toolbox provide mapping and localization. We built typed skills for the Fable agent, an ACT integration interface, and Flybrain training pipelines using PyTorch, imitation learning, and PPO.
Challenges and lessons Getting the physics right was as challenging as building the controllers. We repaired wheel joints and collision geometry, added contact pads to the original grippers, and handled lidar distortion caused by balancing. We learned that intelligent planning depends on reliable movement and grasping—and that biological connectivity needs careful comparison against conventional networks.
Log in or sign up for Devpost to join the conversation.