Ramu & Shamu: Teach by Hand, Delegate by Voice

Inspiration

While working with robotic arms, I kept coming back to one question: why does giving a robot a simple task still require understanding so much of the machinery underneath it?

A person can ask another person to prepare a kit, change one item, or pause while something is corrected. I wanted to explore that same interaction with a small robotic workstation.

That became Ramu & Shamu: two SO-101 arms, a conversational interface, and a workflow built around teaching, execution, and human assistance.

What we are building

Ramu & Shamu is a prototype for conversational control of a dual-arm workstation.

Our first task is preparing small component kits in a controlled workspace. The intended interaction is straightforward:

“Prepare a kit with one red part and one blue part.”

Then:

“For the next kit, replace blue with green.”

The assistant should maintain the current task, identify which supported routines are needed, and show the operator what it plans to do.

The dual-arm setup gives us a practical coordination problem to explore. One arm can position or stabilize a tray while the other places components inside it. The initial demonstration will use known objects and predefined positions.

Why human intervention matters

Physical tasks do not always finish as expected. A gripper can miss an object, a part can move, or the workspace can change.

Our intended recovery workflow is to pause execution, let the operator correct the situation, and resume from a known state. A correction would only become a reusable routine after review and testing.

This is the central idea behind the project: make human assistance part of how the workstation operates.

Technical approach

The proposed architecture separates conversation from motor control.

The conversational layer interprets requests and selects supported actions. A local robot controller executes tested routines and enforces motion constraints.

The language model does not directly generate unrestricted motor commands. Task status must come from the controller, with separate verification where needed. Completing a movement does not automatically prove that an object was picked up.

For the Alexa+ track, the planned entry route is a clearly labeled simulated Alexa+ experience in a web application, connected to the physical workstation.

Current implementation

The project is being developed around a dual SO-101 setup.

Challenges

The engineering problems we are addressing include coordinating two arms, establishing repeatable pickup positions, handling interrupted tasks, and distinguishing completed motion from successful manipulation.

Another challenge is keeping language flexible while keeping execution predictable. The assistant needs to understand variations in a request without assuming the robot can perform an unsupported action.

What comes next

After validating the first task, the next steps are better outcome verification, more reliable recovery, and a library of reviewed routines that can be reused across different kit configurations.

The longer-term goal is to make small robotic workstations accessible to people who understand their work but do not want to program every movement.

Built With

• conversational
• lerobot
• python
• robotics
• so-101