Roblotix

Robot and Roblox
Robot Arm Controlled Dragon Game
Memory Game
Hand Assisted Obby Escape
Game Lobby

Inspiration

Robot arms and video games rarely meet. The arm does something physical, the game does something digital, and neither one changes the other. We wanted a game where a real robot arm is a teammate: the blocks it moves on a real table become platforms, bridges and targets in a Roblox world, and the game can reach back and shake, slow or flip the real arm. It was also a hackathon-sized excuse to take a robot built for labs and use it for something playful.

What it does

LeBot Games is a set of Roblox games played by two people at once: a player at the Roblox screen and a robot operator at a real SO-101 arm. Anyone else who joins is a spectator, and their view follows the player's camera, which is useful on a projector.

Robo-Obby: Smart Warehouse Escape. The player is trapped in a warehouse of hazards they can't clear alone, and the operator builds the way through with real cubes.
- Saw Wall: a closing wall, stalled by pulling two levers that need a stack of cubes to reach.
- The Void: a gap in the floor that only a cube bridge crosses.
- Laser Walk: a laser floor, safe only on the route a cube token reveals.
- Dark Zone: a sealed, dimmed room where the lights have to be restored.
- An overseer camera lets the operator look down on the whole course.
Pattern Match. A "Papa's-style" ticket shift. Order tickets arrive, the player sees the target pattern, and the operator can't see it. The player calls out moves by voice while the operator rebuilds the pattern on the real grid. A 3D replica of the table shows the live board, and a digital twin of the arm reaches over it. Random events such as vanishing tiles, tremors and an inverted arm keep it tense.
Dragon Escape. The player runs for the exit while the real arm drives the dragon chasing them. Where the arm points is where the dragon reaches, so the operator is the monster and the player is the prey.
Both directions. The game shakes the real arm too. Earthquake, fog (lag), mirror, molasses, butterfingers and freeze are all real modifiers on the arm's commands, with hard safety caps so a game script can't hurt the servos.

How we built it

Hardware and vision. An SO-101 leader and follower pair running through LeRobot, and one overhead webcam looking at a printed 4×3 grid. AprilTags locate and calibrate the sheet. Colour segmentation finds the cubes, and a vision model reads which cell each cube sits in, two seconds after the gripper opens. The model only has to look when the board can actually have changed, which is when the arm lets go. A held block is reported where it was last seen, so a platform never vanishes from under the player. Gemini helps name block colours when a session starts.

The bridge. Python is the WebSocket server, and it runs inside the arm process because only one process can hold the arm's serial port. Roblox Studio connects out to it. The same link carries arm joint angles and block state to Roblox, and carries game events back to the arm.

Matching the scale. The key design decision was making the real cube match the game. With the mat scale $s = \frac{8\ \text{studs}}{57.15\ \text{mm}}$ and a $48\ \text{mm}$ cube, one cube is

$$48 \times s \approx 6.72\ \text{studs}$$

A player can climb one cube but not a stack of two ($13.44$ studs). That single number decides how every obstacle is built: Bay 1's lever gate only opens if the operator stacks cubes.

Roblox side (all Luau). A shared core handles roles, a game registry (each game plugs in a player view and a robot view) and the connection to the bridge. In the obby, BlockSync turns real cube poses into physics crates and HazardManager builds each bay, arbitrates the objective HUD and handles resets. A giant mirrored Sky Arm delivers cubes. Pattern Match has a tick-based state machine, a pooled-block board renderer and an arm twin drawn from the real arm's forward kinematics. A solo mode and a mock bridge let the whole team work without the hardware.

Challenges we ran into

Scale had to make physical sense. A cube a few studs off broke the climb-one-but-not-two rule, so everything derives from real measurements.
Vision is noisy and costly. Frame-by-frame tracking of moving blocks was hundreds of lines and fragile. Reading the board only after a release, and holding a solved state for a second before it counts, replaced it and absorbed the jitter. Each read costs an API call, so we had to design around credits as a single point of failure and added a visible warning banner.
Studio-only constraints. Roblox only allows WebSockets from Studio, playtest edits are discarded, and cached modules kept feeding stale config to our build scripts.
Safety. Game events must never be able to damage the real arm: three layers of limits, a kill key and the power jack as a hard stop.
A servo bus that drops out. Under sustained motion the follower's bus can drop 37–220 s in, so we monitor power in the teleop loop.
Shared HUD, many hazards. A sealed dark room, a closing wall and a laser floor all want the objective text, so we built an arbitration layer.

Accomplishments that we're proud of

A real robot arm that is part of the game, not a controller for it.
One physical rule (one cube climbs, two don't) that shapes the whole design.
Two-way interaction: real cubes change the game, and game events change the real arm.
A shared core that lets a new game plug in as a player view and a robot view.
Test suites for the board logic and the end-to-end link that run with no hardware.

What we learned

Keep the physical and the virtual on one coordinate system. It is easier to build around real constraints than to patch the game later.
Design for noisy input with debouncing, hold times and forgiving tolerances, rather than trying to be perfect.
Ask a vision model only when the answer can have changed.
Build a fake version of every external dependency. Mock feeds let everyone work at once.
Safety limits belong in the lowest layer, where no game script can override them.

What's next for LeBot Games

Finish the obby's remaining bays (lava, crusher, mobs, arrow turrets, ice) at the new cube scale.
A steampunk warehouse backdrop so the whole world feels like one place.
A 1v1 arena where the player faces the operator-controlled arm.
A polished runner HUD and results flow.