Inspiration
We want a future where one human can manage a dynamic fleet of humanoid robots for heavy industry and firefighting without writing complex code.
What it does
Our platform uses a teleoperation interface where a human physically guides a robot through a task. We capture multi-modal inputs (RGB, depth, thermal) and map them to motor commands using Behavior Cloning. Once trained, the robots execute tasks as a coordinated swarm while a supervisor monitors them via a central command dashboard.
How we built it
We powered the AI engine using Diffusion Policies to translate human demonstrations into reliable robot movements.
Challenges we ran into
Our biggest challenge was covariate shift—when a robot encountered an unseen physical state, like a shifting piece of debris, and froze. We solved this by building an interactive correction loop into the command center, allowing the supervisor to pause the swarm, take manual control, and record the recovery path.
Accomplishments that we're proud of
We successfully deployed this learning architecture across varied high-stakes environments, from asphalt repair to burning building evacuations.
imitation learning or learning from demonstration (LfD) system. This is the exact technical concept the user is describing. Core components of such a system: Hardware for Demonstration (Teleoperation/Sensors): How the human shows the robot what to do (VR, master arms, kinesthetic teaching, motion capture, visual observing). Data Capture & Processing: Recording the state (vision, joint angles, forces) and the action (what the human did). The Learning Engine (AI Model): The software that learns the mapping between what the robot "sees" and what it should "do". Mention Behavior Cloning, Inverse Reinforcement Learning, and newer models like Diffusion Policy or Action Chunking Transformers (ACT). The User Interface: Making it easy for a non-programmer to use. Start, stop, reset, correct mistakes. Sim-to-Real & Iteration: Testing it out and providing corrective feedback.
What we learned
Instead of writing code, people just "show" the robot what to do.
- The Teaching Interface (How the human shows the robot) Kinesthetic guiding (grabbing the robot arm and moving it). Teleoperation (using a VR controller, haptic feedback arm, or joystick to drive the robot remotely). Visual observation (just watching a human do it via camera, though this is harder because of the "correspondence problem" mapping human joints to robot joints).
- Data Collection (What the robot pays attention to) As the human does the job, the system records everything. Inputs: Camera feeds, depth sensors, touch/force sensors, joint positions. Outputs: The exact motor commands or joint angles the human triggered. A good system needs clean data. You have to record 50-100 demonstrations of a task to give the robot a good baseline.
- The AI Brain (How it learns) This is where the magic happens. The goal is to map the sensor inputs to the motor outputs. Behavior Cloning: The simplest method. It treats the data like a supervised learning problem (if you see X, do Y). Advanced models: Diffusion Policy and Action Chunking Transformers (ACT) are the current state-of-the-art tools for this, handling complex, messy real-world tasks much better than older methods.
- Handling Mistakes (The Correction Loop) A major challenge is "covariate shift"—if the robot makes a tiny mistake, it enters a state it never saw in training and fails completely.
Our biggest challenge was covariate shift—when a robot encountered an unseen physical state, like a shifting piece of debris, and froze. We solved this by building an interactive correction loop into the command center, allowing the supervisor to pause the swarm, take manual control, and record the recovery path.
What's next for SpaceBots
Accomplishments & What's Next We successfully deployed this learning architecture across varied high-stakes environments, from asphalt repair to burning building evacuations. Next, we plan to expand the swarm's training to autonomous mini-submarines for underwater search and rescue operations in Florida coastal environments. Robot Rescue Rangers - Physical AI Swarms Under Human Supervision with Command Center Dashboard Demo Robot Rescue Rangers - Physical AI Swarms Under Human Supervision with Command Center Dashboard Demo
AI Robotic Swarm Road Construction Crew https://youtu.be/E-va111HMSw
Humanoid Robot learns to cook by watching video-cake decoration with AI hallucination for uniqueness https://youtu.be/_klLlbqpQhE
Robot Bartender making Bitcoin secret menu drink in Wynwood Miami https://youtube.com/shorts/z02hb44Z2RA?feature=share
Miami Vice Robot Tennis 🎾 Doubles Champions https://youtube.com/shorts/Uio7D6FlKnM?feature=share
Robot Swarms - Train Railway Crossing, Aircraft Rescue, Firefighting, Nursing Homes https://youtu.be/TC0SDLJw6IA
Built With
- isaac
- nebius
- nvidia
