Watch both videos in submission.
Inspiration
Warehouse automation needs to handle more than the perfect shift. Robots stop, software crashes, and communication links fail—but the work still needs to get done.
Inspired by the Neptune North challenge and the robot duo from Iron Man, we built Dum-E and U to explore a simple question: can two robots work together and recover from failure without a central controller—or even talking to each other?
What it does
Dum-E and U are two simulated robotic arms that pack a six-parcel order into a carton simultaneously. Each robot runs its own agent and observes the workspace: parcel positions, the other arm’s position, and simulation time.
There are no messages, shared task records, or heartbeats between the agents. Instead, they coordinate through what changes in the physical environment.
If one robot’s lane stops making progress for 25 seconds of simulation time, its partner infers that it has stopped. The surviving robot finishes its own lane, then collects the remaining reachable parcels while maintaining clearance from the stopped arm.
Our demo interface lets users watch the workspace, inspect each robot’s logs, and trigger software crashes, arm faults, and recovery.
How we built it
We built the simulation in NVIDIA Isaac Sim 5.1 with Isaac Lab 2.3.2, using two UFactory UF850 arms with suction grippers and independent Python agents.
Each agent follows a local decision process based on observations of the table. During normal operation, the arms work in separate lanes. During failure recovery, the surviving arm approaches the other lane from the far end and avoids reaching within 0.22 metres of the stopped arm.
We also made completion depend on physics. A parcel only counts as packed when it has left its original mark, rests inside the carton walls, has stopped moving, and is no longer attached to suction.
This simulation complements our physical SO-101 human–robot prototype, which explores decentralised coordination through peer-to-peer UDP state exchange.
Challenges we ran into
The central challenge was distinguishing a robot that had failed from one that was simply taking time. Without messages or heartbeats, the agents had to infer failure from a lack of progress in the workspace.
Recovery also required more than assigning unfinished parcels to the surviving robot. The stopped arm remained a physical obstacle, so takeover needed an explicit clearance constraint and a different approach order.
Finally, issuing a successful movement command was not enough to prove that a parcel had been packed. We needed to verify the resulting physical state.
Accomplishments that we're proud of
In our recorded failure run, Dum-E’s software froze three seconds into the job. U inferred the failure from the table alone and completed the remaining work. The result was 6/6 parcels verified, with zero inter-robot messages.
The PR also reports 18 passing simulation unit tests and an ALL PASS result from the repository’s scenario suite. PR and recorded-run details
What we learned
We learned that a shared environment can provide useful coordination signals even when agents exchange no explicit messages.
We also learned to separate intention from evidence: a robot attempting a pick does not mean the parcel reached its destination. Checking observable outcomes made both the demo and its success criteria more meaningful.
Most importantly, failure recovery needs to be part of the design from the beginning.
What's next for Dum-E and U
We want to test more failure conditions, improve how the agents distinguish slow progress from a stopped robot, and explore larger teams and less predictable layouts.
Our next major step is to bring observation-based takeover onto physical hardware and measure completion time, recovery time, and clearance across repeated trials.
