Spidey & Armie

Title Screen
Spidey & Armie IRL
3D Real-Time Simulation
2D View
Spidey's Design

Inspiration

Small robots are great at looking clever and bad at being useful. A tiny quadruped can wave and dance, but ask it to cross a table and it stumbles, drifts and walks into things. It has no idea where it is.

We wanted to see how far a cheap robot could get with help from its surroundings: an overhead camera for eyes, a laptop for a brain, and a robot arm for the moments walking isn't enough. And we wanted to control it the way you'd talk to a friend.

What it does

Spidey is a self-navigating quadruped on a tabletop arena. Tell it where to go, by voice, by wasd, or by clicking the map, and it plans the route and walks it.

It knows where it is. An overhead camera tracks the robot and the arena with tags, so its position and heading are known in real time.
It plans and adapts. It routes around obstacles, recovers when it gets stuck, and corrects for drift.
It calls for backup. When there is no walkable path, such as a solid barrier across the table, a robot arm picks the robot up and carries it across.
You can just talk to it. Say "go past the blue barrier and then wave", "turn right and walk 5 steps" or "go to the top left corner". A speech model turns that into a checked, step-by-step plan and the robot carries it out.
It shows you what it sees. A live 2D and 3D map shows the robot, detected objects, the arm and the planned path. Manual driving and poses are there too, as extras.

How we built it

Robot. A Sesame quadruped with an ESP32-S2 and PlatformIO firmware. It has four gaits (forward, back, left, right) and a library of poses, all controlled over WebSocket.

Eyes. A Raspberry Pi running QNX with a Camera Module 3. A C++ tracker finds ArUco tags on the robot, the arm and the four table corners, solves the camera pose every frame, and streams the robot's position at about 30 frames a second.

Brain. A Node.js bridge that:

filters the robot's pose with a Kalman filter,
plans routes with Dijkstra's on a 2 cm grid,
drives the gaits with hysteresis, so the robot doesn't waste time switching between turning and walking,
detects when the robot is stuck and backs off,
measures the robot's real walking and turning speeds with the camera.

Arm. An SO-101 arm with a closed-form inverse-kinematics solver that we checked against forward kinematics, plus hard safety limits on every move. It works out how to grip the robot from the tag geometry. After each grip, the camera checks whether the robot moved with the gripper. If not, the arm shifts and retries.

Object detection. MobileSAM on the CPU, combined with colour and parallax cues from a moving camera, finds objects on the table. Their outlines and textures go to the map as obstacles.

Interface. A React and three.js app with 2D and 3D maps, telemetry, and a built-in simulator so we could develop and test with no hardware.

Voice. Whisper for transcription and a language model that returns structured JSON steps, both through Groq. Every step is validated before anything moves. Walks and turns are measured against the live camera position rather than a timer.

Challenges we ran into

The robot has no sense of its own movement. Its gaits are open-loop, so it drifts, veers and lags by about a third of a second. Everything had to be closed with camera feedback.
Noisy tracking. A tag's apparent size gave unreliable height measurements, so we use a filter to reduce noise across measurements.
Making the arm reach. Held upright, the arm can't reach much above 12 cm, so it tilts the gripper only as far as needed and make sure the whole lift sequence is physically possible.
Flaky hardware and networks. The servo bus dropped the odd packet, we ran into networking issues with the Pi, and Spidey had a number of electrical issues.

Accomplishments that we're proud of

A full loop across very different systems: voice recognition to machine-readable plans, computer vision for mapping and localization, and precise manipulation using a robot arm.
A robot that has built-in error recovery. It gets out of being stuck, fails cleanly instead of looping, and coordinates with the arm in difficult situations.
A hardware-free high-fidelity simulator and a large set of automated tests across the planner, navigator, voice pipeline and arm math.

What we learned

Classical approaches for control are often more than enough for situations like this, where we have very strong priors for localization/mapping.
Language models are good at understanding messy intent but shouldn't be the safety layer. Let them interpret, then verify everything in code.
Create a robust simulator and tests first to allow four people integrate and build quickly.
The weakest link is not always the one you suspect. We were facing latency issues and initially suspected compute limitations on the Raspberry Pi, but it turned out that changing our network to a mobile hotspot immediately 17x'ed our latency from 0.5fps to ~9fps.

What's next for Spidey

Make the arm hand-off smoother and more reliable, and handle more than just carrying the robot.
Support more than one robot and richer missions, like fetching or sorting.
Add speech output, so the robot can ask for clarification and confirm what it's doing.
Make it faster and more compact, with more of the processing on the robot or the Pi.