I am a small red box. In a moment a stick is going to hit me, and it only gets one go. Before it moves, six agents push a simulated copy of me hundreds of times to find the push that lands me in the right bag. One of them gets killed halfway through. The others don't stop. Then a human says yes, and I get sorted.

Inspiration

"See it. Say it. Sorted." Anyone who has taken a UK train knows the announcement. We wanted a machine that works the same way: see the parcel, say what it plans to do, and only then sort it.

Real machines get one shot. A sorter can't undo a push, and when it gets one wrong the line stops. An RS Industria figure puts unplanned downtime for UK manufacturers at about £5,121 an hour on average, from under £500 at small firms to over £10,000 at large ones. And the UK moved a record 4.6 billion parcels last year (Ofcom).

We had been reading world-simulator research (https://jwang771.github.io/3dworldsimulator/lab), where a system imagines the outcome before acting. We asked: what if a cheap servo rig could rehearse each move in simulation before touching the real box? And what if the thing doing the rehearsing had no single point of failure?

What it does

See it. A camera captures the parcel on the bench. You confirm what it is and its colour, and a matching 3D model is placed in a digital twin of the bench.

Say it. Six independent search agents rehearse the push in a MuJoCo physics simulation. Each one tries different sweeps, then re-runs the promising ones about ten times with friction, mass, position and tilt nudged, so the chosen plan works under variation and not just once. The agents share results peer to peer with no leader. Kill one from the UI and another takes over its role in about five seconds. The viewer replays the winning rehearsal and shows the exact servo commands it wants to send.

Sort it. A human confirms, the command goes to an Arduino UNO Q, and a servo sweeps the box left into the red bag or right into the blue bag.

The engine behind it is called Gekko: like a gecko, it can lose its tail and keep running.

How we built it

  • Digital twin: Python and MuJoCo. The bench, bags, boxes and sweeper are all built from one config file, and every candidate push is a real physics rollout.
  • Swarm: separate Python processes that talk over UDP. There are four roles (Router, Planner, Checker, Watcher), any agent can take any role, and heartbeats detect when one has died.
  • Viewer: a single static page using three.js, no framework and no build step, fed over WebSockets. It replays the actual MuJoCo trajectories, not a canned animation. Models were made in Blender.
  • Hardware: an Arduino UNO Q, a USB camera and a micro servo mounted on a can, with a stick that sweeps across the bench like a windscreen wiper.
  • Process: two of us, one on the viewer and one on simulation and hardware, with Claude Code and Codex as coding assistants and automated tests (Node and pytest) on both halves.

Challenges we ran into

  • Two servos became one. Our first design used two servo motors. We couldn't work out the pin placement on the Arduino UNO Q in the time we had, so we redesigned the mechanism instead: one servo with a stick that sweeps like a windscreen wiper, left for red and right for blue. The constraint gave us a simpler machine.
  • Getting the twin to talk to the board. The simulation and viewer run in Python and a browser; the servo runs on the Arduino. Connecting the two was the hardest part of the build.
  • Phantom deaths. Agents kept declaring each other dead under load. Tuning the heartbeat timeout taught us that failure detection is a trade-off between speed and false alarms.
  • A merge that froze the UI. Joining the simulation and viewer branches dropped one function and silently disabled the Send button. Tests caught the next one.
  • Physics doesn't flatter you. Our first rehearsals pushed a centred box straight past both bags. The simulation was right and our plan was wrong.
  • Scope. We changed idea more than once in the first twelve hours and had to cut hard to finish.

Accomplishments that we're proud of

  • The whole loop exists: camera, twin, swarm, human confirmation, servo.
  • You can kill an agent live in front of a judge and the plan still arrives.
  • Everything on screen is real. The motion you see is the motion MuJoCo computed, and the human stays in charge of the real move.
  • We kept it honest: where something is pre-built (the 3D models), we say so.

What we learned

  • A plan that works once is not a plan. Testing each push under varied friction and mass changed which push won.
  • Leaderless systems are easy to draw and hard to debug. Most of the work was in what happens when messages stop.
  • Keep unit and axis conversions in exactly one place.
  • With AI coding tools, the bottleneck moves from typing to deciding and reviewing.

What's next for See It, Say It, Sort It

  • Automatic recognition and sizing of parcels from the camera, removing the manual confirm step.
  • Feeding real outcomes back into the twin so it corrects itself over time.
  • Spreading the swarm across several machines, and unplugging one mid-plan.
  • Moving from one bench to a conveyor with a queue of parcels, and from one sweep to a wider set of moves.

Built With

Share this project:

Updates

Submission history