Pawnzi Scheme, starring Pawn Malone

Inspiration

Every big city park has one: the guy at the stone table with a chess clock, a stack of bills and a running commentary on everything you're doing wrong. He's the best part of the game, and you lose to him anyway.

We wanted to build that guy. Not a chess engine in a window, but someone who sits across a real wooden board, moves real pieces, slaps the clock, and has opinions about your knight. So we built Pawn Malone: a robot chess hustler with two arms, a cartoon face on a tablet, and absolutely no respect for your queen.

Then, while we were planning it, a question changed the project: if the robot already has hands, why can't it be your hands? Plenty of people who grew up playing chess can't pick up the pieces anymore, because of an injury, arthritis, a stroke or ALS. Screen chess exists, but it isn't the same as sitting across a board from your grandkid. So Pawn Malone got a second job: a voice mode where you just say your move and the arm plays it for you.

What it does

You walk up to the table. Pawn Malone asks you two quick questions, by tap or by voice: how good are you, and what do you want? A quooick puzzle, a quick game, or a theme like knight tricks, a queen sacrifice or an endgame.

The challenge. Claude picks a real, verified mate-in-2 from a slice of the Lichess puzzle database (only ones the arm can physically reach), or sets up a quick game with Stockfish matched to your level and odds on the clock. Pawn Malone introduces it in character.

The stake. Play money goes on the table. Double or nothing.

The sealed prediction. Before you move, Pawn Malone writes down what he thinks you'll play and seals it as a salted cryptographic commitment, shown as a face-down card. After you press the clock it flips, and anyone can check that the guess was locked in before you touched a piece. It's a magic trick with a proof attached.

The move. You move by hand, or by voice. The big arm plays Pawn Malone's reply, the small arm slams the clock, and the face reacts. Stockfish grades every move, and the grade drives his attitude:

Moment	Pawn Malone
You blunder	"You just hung your queen. I'll take that, thank you." plus a slow clap
You find a great move	"...okay. Didn't see that one. Respect." plus a grudging nod
He blunders	"That was a test. You passed. Barely." plus a shrug
He wins	A victory spin, then a clock slam

He roasts the moves, never the person. That's a hard rule in the line bank.

The finish. Checkmate in about two minutes, and Claude explains the solution you just played (or missed).

Two ways to play

Hustle mode: you against Pawn Malone, with the stake, the hustle and the sealed card.
Hands mode: you play both sides, or two people play each other, and the robot is everyone's hands while Pawn Malone commentates.

Voice mode

Voice mode works in both, so someone who can't move pieces can still play a real game on a real board, including with family and friends.

Say it however you want. "Knight to f3" works. So does "move my horse next to the king."
Claude proposes exactly one move. Cartesia transcribes, Claude turns loose phrasing into a move, and our own rules engine accepts it only if it is exactly one legal move. If it's ambiguous or illegal, he asks again.
You confirm. Say "yes" or tap the pad.
The arm moves it, through the same safety guard as every other move.

The browser table

The same software also runs as a robot-free chess table in the browser: Human vs CPU against our engine or Stockfish, CPU vs CPU with two virtual arms, live Stockfish analysis, hints with reasoning, takebacks, PGN and FEN, board sounds, move-quality badges, a Persona page where Pawn Malone talks, and a game journal. It works on a phone and never makes a request off the machine. We also built Pawn Malone's Puzzle Box, a souvenir page with 26 verified mate-in-2 puzzles to take home.

How we built it

Pawn Malone is 14 programs in Python, Rust and TypeScript, talking over one versioned message bus on ZeroMQ. We wrote every interface down first, in a contracts/ package (now at version 1.1.0), so the four of us could build in parallel without stepping on each other. Every path in the repo has exactly one owning program.

The safety tier comes first. A robot arm reaching across a table where a stranger's hands are is the scariest part of this project, so we treated it that way:

guard is the only program allowed to command an arm. It owns caps on speed and reach, the freeze, and the interlock. Drivers are disabled by default and no command-line flag or environment variable can turn hardware on.
Two independent hand watchers, a RealSense depth camera over the board and MediaPipe hand tracking on a side camera. Either one alone freezes both arms.
A foot pedal freeze for the operator.
The arm wakes only after 750 ms with both watchers clear and every heartbeat fresh. Freshness is judged on the receiver's own clock, never the sender's timestamp, and a stale or missing input counts as "a hand is present".
The big arm has no brakes, so nothing ever cuts its power unless it's confirmed resting in its cradle, and no error or quit key ever homes it.

We red-teamed our own arm driver. An adversarial review attacked it round after round; we fixed every finding and closed with no open P0 or P1 after five rounds.

moonshark, our own chess engine. We ported the moonfish engine to Rust on top of cozy-chess. Its rules core is the project's only source of chess legality: every move, from the camera, the voice or the engines, passes through it. In qualification it won 10 of 10 games against Stockfish 19 set to 1500 Elo, all by checkmate, five with each colour, with its slowest move at 230 ms against a 250 ms budget. (Ten games is a sanity check, not a rating. We know.)

Engines and referee. An engine host runs moonshark and Stockfish 19 over UCI, picks the opponent, referees every move with a quality grade, and streams live analysis. Stockfish runs as a separate process and is never bundled.

Perception. The overhead camera registers the board from ArUco tags on its border, reads squares, piece colours and the clock lever, and says "unknown" instead of guessing. If it's unsure, a teammate types the move on the house console.

Claude, walled off on purpose. Claude has four real jobs: picking the challenge from a verified list (never inventing a puzzle), writing live lines through a strict filter, explaining the solution from the referee's line, and understanding loose voice phrasing. It never chooses Pawn Malone's move, reads the board, decides legality, touches the clock or the money, or commands an arm. Safety and freeze lines always come from our authored line bank, and if Claude is down, everything still works from the bank.

Voice and face. Pawn Malone speaks through Cartesia (stock voice, imitating no real person) with lip timing sent to his animated face on the tablet. Speech to text for voice mode also runs on Cartesia.

Simulation first. We built a MuJoCo digital twin of the whole table: arms, cameras, pedals and injected faults. In it, both arms played full games against each other with moonshark on both sides (151 and 127 plies, both ending in checkmate, with a colour-swapped rematch), and the taught placement backend set down 31 of 31 pieces on the right square. We also set up an NVIDIA Isaac Lab task to train a pick-and-place policy on the ASUS Ascent GX10. Even a learned policy never commands a motor: motion validates every path and the guard re-checks every point as it's sent.

Tested like we meant it. Nearly 4,000 automated tests across 14 programs, with zero failures, all running against fakes and the simulator so no test ever moves an arm or makes a paid API call. We used Claude Code throughout to build and review the software roles in parallel, with every safety change reviewed by a human before it goes near the arms.

Challenges we ran into

A safety system that's right is a safety system that freezes. When we composed the whole stack in one simulated process, it sometimes stalled 48 to 103 ms against a 47 ms freshness margin. The guard did exactly what it should and froze the arms every time. That was the moment we stopped being annoyed at it and started trusting it. The real fix is splitting the safety feed into its own process.
Robots hear themselves. In voice mode, a speaker saying "yes" could confirm a move by accident. So Pawn Malone's lines never contain "yes" or "no", and on the real table a spoken "yes" is refused until we've measured the speaker's echo time. A tap always works.
An arm with no brakes. Every shutdown path had to be designed around the fact that cutting power drops the arm. We wrote a cradle rule that nothing in the code can bypass.
Licensing is a real engineering problem. The standard OpenCV wheels bundle GPL video codecs, so we built OpenCV from source without them, and audited all 247 third-party packages across 23 lockfiles.
Keeping the AI out of the loop. The tempting design is "ask the model what to play." We deliberately did the opposite: Claude talks and understands, deterministic code decides and moves.

Accomplishments that we're proud of

A robot that people actually want to play, and that roasts them for it.
Our own Rust chess engine that went 10 for 10 against Stockfish at 1500 Elo inside a quarter-second move budget.
A sealed-prediction trick that is cryptographically honest.
A safety tier with two independent hand watchers, a pedal, freshness checks and an internal red team, built before anything moved.
Voice mode: a real path back to the board for people who can't move pieces anymore.
Nearly 4,000 passing tests, and a browser version anyone can play with no robot at all.

What we learned

Simulate before you power anything. The digital twin caught bugs that would have been very expensive on a real arm, and let the whole team work before the hardware was set up.
Put the AI where it's good. Claude is excellent at picking a fitting challenge, understanding "move my horse next to the king" and explaining a combination. It doesn't need to be in charge of legality or motors, and the product is better and safer because it isn't.
Contracts are worth it. Writing every interface down first is what let four people and 14 programs come together in a weekend.
Personality is a feature. The same chess move feels completely different when someone across the table reacts to it.

What's next for Pawnzi Scheme

Finish training the Isaac pick-and-place policy on the GX10 and compare it against the taught squares.
Run voice mode with people who've lost hand mobility, and let them tell us what's wrong with it.
Measure hand-watcher freeze latency on the real arms against our 150 ms target.
Maia-2 for more human-like predictions on the sealed card.
Take Pawn Malone to a real park.