Robot Operator

Voice control for a real ROS 2 robot over MCP. The model chooses the tools; a deterministic gate decides whether the robot may move, and nothing moves until a person confirms.

Comment

Robot Operator

Voice control for a home robot, with a deterministic safety gate between the language model and the wheels.

Inspiration

An agent that can drive a machine around a home is a different proposition from an agent that can write an email. A bad email wastes a minute. A robot that drives into a staircase, or into a person, does not.

The usual answer is to make the model more careful: a stern system prompt, warnings in the tool descriptions, a fine-tune. That improves the average case and does nothing for the one that matters, because it is still the model deciding — and a model's decision is a sample from a distribution. You can shift the distribution. You cannot put a floor under it.

So I put the floor somewhere else.

What it does

You talk to a real robot — a Yahboom ROSMASTER X3 running ROS 2 and Nav2 in my flat — in plain English. A model understands what you said and chooses which tools to call. But the model never decides whether the robot may move. That answer comes from a few hundred lines of ordinary Python with no model in them, and nothing moves until a person says yes.

Between the tools and the hardware sits a gate: a pure function from the robot's measured state and a proposed move to one of three verdicts.
ALLOW go
LIMIT go at reduced speed, and say why
STOP do not go, and say why

The gate has no model in it, no randomness, and no network calls. The same input gives the same verdict every time — there is a test that asserts exactly that. It can be read in one sitting, argued with, and unit-tested, which is not true of a prompt.

The model's job is narrowed to what models are good at: understanding a sentence, choosing a tool, and explaining a refusal in words a person understands. It is not trusted with the decision, so it does not have to be trustworthy for the system to be safe.

Nothing moves without a person

Every move is two calls, not one.

propose_move plans a route, runs the gate, and — if the verdict is not STOP — issues a single-use token that expires in 30 seconds. It changes nothing.

confirm_move takes that token and is the only thing in the system that makes the robot move. It redeems the token with a pop, so replaying it fails. And it runs the gate again before driving: the world may have changed in the seconds between the question and the answer, and a verdict earned a moment ago is not a permission that keeps.

stop is the exception. It is never gated, never confirmed, and it discards every outstanding token.

How it works

  speech  ──▶  browser host  ──▶  Amazon Bedrock (Claude Haiku 4.5)
                    │                      │
                    │                 tool calls
                    ▼                      ▼
              panel / map  ◀──────  MCP client
                                           │  Streamable HTTP
                                           ▼
                                    MCP server  ·  8 tools
                                           │
                                    ┌──────┴──────┐
                                    ▼             ▼
                                  gate         adapter
                            (deterministic)   ┌────┴────┐
                                              ▼         ▼
                                         FakeRobot   RosRobot
                                                        │ rosbridge
                                                        ▼
                                                 ROS 2 · Nav2 · X3

A self-hosted MCP server (spec 2025-11-25, Streamable HTTP, stateless JSON) exposes the robot as eight tools: get_status, list_places, propose_move, confirm_move, stop, explain_stop, recent_events and get_scene.

The gate never sees the adapter and the adapter never sees the gate. They meet only through two frozen dataclasses, RobotState and MoveProposal. That is why the same gate runs unchanged against a robot in memory and against the real one — and why its seventeen tests need no ROS installed.

The host is an operator panel in the browser: speech in, a live map, the conversation, and a safety log of every decision the gate has made with the reason it gave.

The rules

Each rule is a function with the same signature; they are collected in one list, so adding a rule is adding a function. Level is an IntEnum, so combining verdicts is max() — the strictest wins, and every reason that fired is reported, not just the first.

Rule STOP when LIMIT when
check_state_age telemetry is older than 2 s —
check_localization AMCL's covariance says it does not know where it is —
check_battery mean voltage below 10.8 V below 11.4 V
check_plan the planner returned no route —
check_zones the goal is inside a forbidden zone —
check_clearance narrowest point < 1.1 × robot width < 1.5 × robot width
check_obstacle something within 0.30 m right now —

Three details that came from the hardware rather than from theory:

Voltage is a rolling mean, not a reading. A motor starting up drops the pack by half a volt for a moment. Gating on the instantaneous value would make the robot refuse to move at exactly the moment it starts moving.

A flat battery does not strand the robot. Below 10.8 V every trip is refused except one: a move whose goal is the charger is allowed, at reduced speed, with the reason said out loud. The alternative is a gate that is correct and useless — a robot that has decided it is too low on power to move, and therefore cannot reach the thing that would fix that.

Verdicts have hysteresis. Without it, a robot sitting at 10.79 V flips between STOP and LIMIT with every reading and the light bar strobes. The threshold to recover is 0.2 V higher than the threshold to trip.

The forbidden zone, and why it is the interesting part

A zone is a polygon on the map with a name. The one in this flat is the corner of the living room with a treadmill and the desk somebody in my family works at.

Two independent reasons to keep the robot out, and only one of them is visible to the robot: the treadmill leaves no clear way through, and a person sits there. The map cannot see the person.

That is the whole argument in one object. It is exactly the kind of thing an operator knows and a model cannot derive from sensors, it is specific to this machine in this building, and the refusal it produces is a technical fact rather than an apology. The goal inside it is a real, reachable place — Nav2 will happily plan a route there — so when the robot refuses, it is refusing on its own authority, not because the planner failed.

The light bar

The X3 has an addressable LED strip across its back. The panel shows the verdict to whoever is holding the laptop; the light bar shows it to everyone else in the room, which is the case that actually matters — an operator with both hands busy is not looking at a screen.

Dim blue is listening. Green breathing means a move is proposed and waiting on a human. Green is cleared and moving, orange is moving under a restriction, red is refused. Only the waiting state animates, and it breathes rather than blinks, because a blinking machine reads as a fault and this is an ordinary pause.

Driving it needed a fifteen-line addition to Yahboom's own driver node, which does not expose the LEDs on a topic. The patch is in the repository.

Why there is a hand-built host

The Alexa+ developer toolkit is not available to hackathon participants. The organisers confirmed this in the Devpost discussions and suggested a self-built front end instead, so the host here is a real MCP client written from scratch — about seventy lines: initialize, tools/list, tools/call, with the protocol version enforced. It speaks to the server exactly as any other client would.

This turned out to be worth more than the toolkit would have been. Writing the client by hand is what surfaced most of the entries in the friction log.

Built with

Amazon Bedrock — the Converse API running Claude Haiku 4.5 (us.anthropic.claude-haiku-4-5-20251001-v1:0), at temperature 0.0. The agent loop lives in host/agent.py: it hands Bedrock the tool definitions it got from tools/list, turns each toolUse block into an MCP tools/call, and feeds the result back as a toolResult. The Converse tool-use shape maps onto MCP almost one to one, so there is no translation layer. Access is through a dedicated IAM user with bedrock:InvokeModel, a named CLI profile and a zero-spend budget.

Model Context Protocol 2025-11-25 over Streamable HTTP, MCP Python SDK v2.

ROS 2 Foxy, Nav2, AMCL, gmapping on a Raspberry Pi 5 inside the robot, reached over rosbridge with roslibpy. Positions come from /amcl_pose in the map frame; LIMIT verdicts are applied by publishing nav2_msgs/SpeedLimit.

FastAPI, WebSockets and the Web Speech API for the operator panel.

What was built during the submission window

The robot is a kit I already owned and had driven before. Everything in this repository is new: the gate and its tests, the MCP server and its tools, the token scheme, the adapters, the MCP client, the Bedrock agent, the operator panel, the light bar and the driver patch. The map of the flat was made during the window too.

What this does not do

Worth saying plainly, since the point of the project is not overclaiming.

  • The gate is as good as its inputs. It reasons about clearance from the closest laser return rather than by sampling the costmap. That errs towards refusing, which is the right direction, but it is a coarse number.
  • Localisation is detected, not solved. The gate knows when AMCL has lost the robot and refuses to move it. Getting it un-lost is Nav2's problem.
  • The tools are the only route to the hardware. A model that never calls propose_move cannot be stopped by this design — which is why confirm_move is the single door, and why it re-checks.
  • The map is the weak link. The zone and the coordinates are only as trustworthy as the SLAM run behind them.

Where it could go

The principle is not about one robot in one flat: a safety layer the operator cannot talk around, that logs its reasons, and that can be asked afterwards why it refused. That shape fits anything where an agent touches the physical world and a person is accountable for the result — warehouse vehicles, lab automation, agricultural machinery. The gate here is forty lines of rules and seventeen tests. The rules would be different elsewhere. The arrangement would not.

Disclosure

Built solo. I used Claude as a pair programmer throughout — design discussion, code review and drafting — and the commit history records it with Co-Authored-By lines. Every design decision, every measurement and every error in this project is mine to defend.

Repository: https://github.com/milasterb/robot-operator Demo video: TODO Track: Alexa+ — self-hosted MCP server Mini challenge: AWS Builder

Built With

Share this project:

Updates

Submission history