Charlie: Home World Models, but actually for the home

Inspiration

We kept noticing the same failure pattern across “generalist” robot policies: one model trying to handle perception, planning, manipulation, and prediction all at once ends up mediocre at each individually, while also being expensive to run. The LLM world already solved an adjacent problem: OpenRouter routes each request to the model that fits it best instead of forcing one model to answer every kind of query.

Nobody had built that idea properly for physical AI, where a wrong model choice can make a robot arm do the wrong thing in the real world. Finding MoIRA (Modular Instruction Routing Architecture for Multi-Task Robotics, arXiv:2507.01843) gave us the architecture we needed to build Charlie.

What it does

Charlie is a routing and safety layer for physical AI. It is the product; the robotic arm is our demonstration platform.

The laptop captures camera and microphone input, keeps short-term context, and converts a spoken instruction into a structured task. A frozen MiniLM router matches each sub-task to the specialist that is actually suited to it: scene perception, grasp-pose generation, manipulation, world-model prediction, reward/risk scoring, or outcome verification. The system can evaluate up to three candidate plans over a 2.5-second simulated horizon before selecting one.

Only after the plan passes the safety and outcome checks does the laptop send a bounded motion chunk to the Raspberry Pi 4. The Pi has no cloud credentials and independently verifies the robot identity, geometry digest, request binding, timing envelope, joint limits, and gripper limits. It owns the PCA9685 driver and local emergency stop. If routing, inference, validation, or verification fails, Charlie stops before motor execution instead of silently substituting another model. The mini models, alongside many other important models, are hosted on Baseten's services.

For the demo, we built a 4-DOF desktop arm with MG996R base/shoulder servos and SG90 elbow/gripper servos. Their physical motion makes the router’s specialist decisions visible, but the architecture is designed for many robot geometries and sensor configurations.

Why it matters

Lower hardware requirements. The robot-side runtime is intentionally small: a Raspberry Pi 4 handles authentication, validation, timing, and PWM while the laptop/cloud handles the expensive perception, planning, and prediction models. A client does not need a separate high-end GPU on every robot.
Modularity. The specialists communicate through typed capability contracts. A 3-DOF arm, a 10-DOF arm, a gripper, or a different sensor stack can use the same routing layer as long as it supplies the relevant contract and robot model. The current integration profile supports a 256 MB component RAM budget, two simulation workers, and a 2.5-second prediction horizon.
Efficient scaling. With 100 clients, a generalist deployment would need to keep every capability bundled into every serving instance. Charlie loads only the specialists needed by the routed task. New capabilities can be added as independent endpoints without changing the router itself, which keeps deployment and memory costs proportional to the capability fleet rather than to every possible combination of tasks. Serverless providers that can host our model, like Baseten, can be configured to automatically rapidly scale, so the actual router cost is almost nothing.
Model future-proofing. Replacing a specialist does not require retraining or redeploying the whole system. If a better grasp, dynamics, or manipulation model becomes available, it can be registered behind the same capability contract and validated against the same safety boundary.

Technical breakdown / how we built it

The data path runs from the hardware upward, and the validated result flows back down:

Hardware and actuation. A 4-DOF arm is driven through a shared PCA9685 and an external 6 V/10 A supply. The active arm model contains the CAD-derived link geometry, actuator axes, joint limits, gripper mechanism, and mounting dimensions. The Pi keeps motion disabled at startup, initializes I2C/PCA9685 lazily, and will not enable PWM until a calibrated robot model is deployed.
Robot computer. The Raspberry Pi 4 exposes an authenticated control service. It accepts only timestamped, bounded motion chunks whose robot ID, geometry digest, plan binding, timing, joint envelope, and gripper envelope match its local configuration. The emergency stop and motor authority stay local to the Pi.
Laptop orchestration. The laptop receives the camera frame and microphone audio, uses Whisper Large V3 Turbo for speech-to-text, stores short-term personal context, and asks the scene-grounded language/vision service to identify objects, hazards, and task state. Kokoro provides the spoken response.
Routing core. Charlie implements the MoIRA routing approach with a frozen MiniLM embedding router. It compares the instruction and representative task phrases against specialist descriptions, then selects the required capability endpoints. Optional prompt-based and hybrid routing modes are available for ambiguous instructions.
Cloud specialists. Independently hosted Baseten services provide scene perception, grasp pose, waypoint manipulation, bimanual manipulation, pouring, insertion, lid opening, handover, forward dynamics, rigid/deformable/human-motion world prediction, reward/risk scoring, and outcome verification. Each specialist is accessed through a typed contract, so the router does not depend on model-specific internals.
Prediction and safety. The planner produces candidate actions, and parallel world-model rollouts score their predicted outcome, collision risk, slip risk, grasp stability, clearance, and target drift. The software-only integration profile uses at most three candidates, 0.1-second simulation steps, a 2.5-second horizon, 2 cm maximum pre-execution target drift, 20 cm/s maximum gripper speed, and explicit collision/slip thresholds.
Return path. The selected plan is converted into a bounded action chunk, sent to the Pi, revalidated there, and executed only if every check passes. Camera and robot-state feedback then return to the laptop for outcome verification and journaling. A failed check produces an error and stops the chain; it does not trigger an unreviewed fallback model.

Challenges we ran into

Different debugging worlds. The routing layer, Baseten contracts, and safety gates are deterministic software systems, while servos, power, USB enumeration, and mechanical linkages have physical failure modes. We had to keep both paths testable independently and only connect them at a narrow, validated boundary.
Power and mechanical reliability. Simultaneous servo load made supply sizing and grounding important. We also had to learn to recognize a stalled servo—sustained heat and stripped gears—before it could damage the hardware.
Safety contracts. The hardest design question was deciding what the Pi must verify without trusting the laptop. The final boundary checks identity, geometry, plan/chunk binding, timing, joint and gripper envelopes, and request uniqueness before PWM is possible.
Calibration. CAD limits are not the same as safe physical limits. Pulse endpoints, real joint angles, linkage slack, gripper pressure, and repeatability under load all need to be measured before physical execution can be enabled.

Accomplishments we’re proud of

We built a router that treats safety as a first-class architectural boundary. Charlie is designed to stop and report an error rather than substitute a different model after a routing or inference failure, especially once real motors are involved. We also connected the complete software-only path—from voice and perception through specialist routing, candidate prediction, validation, and feedback—before allowing it to control hardware.

What we learned

Physical AI is mostly systems design. A good routing decision does not matter if the power supply browns out, a servo stalls, or calibration is wrong. Grounding, power sizing, calibration, explicit failure modes, and a small trusted robot-side controller are what make the higher-level intelligence usable.

What’s next

We are finishing the arm’s calibration and pulse-endpoint confirmation, bringing up coordinated dual-arm behavior, and evaluating the router against more physical task scenarios instead of only the current software regression suite. The next step is to validate the specialist contracts across more robot geometries and sensor combinations.