AMDRobotDrawerAIModel

Friction drags robots off course on surfaces. We modelled it from real data on a low-cost arm, estimating contact effects from servo readings instead of expensive force sensors. Say "Friday, draw a smiley face," and it draws it.

Inspiration

Jarvis from Iron Man. We wanted that experience on a real robot arm: talk to it, and it does the thing.

We started from the standard robot-arm demo, an SO-101 picking up a cube and dropping it in a bin. It is a fair benchmark, but it is hard to see whether the robot got better. A drawing shows every error. A lopsided circle, a crooked smile or a dragging pen is obvious at a glance, and a person can describe it in plain words: "the eyes are too close."

So we dropped pick-and-place and asked: can you talk to a robot arm, have it draw something, tell it what was off, and have it do better next time? We named the voice agent Friday.

Force sensors on robots are expensive. We wanted a more accessible route, so we asked how much of the physics of pen-on-paper contact we could recover from the servo data a cheap arm already gives us.

What it does

You say "Friday, draw a smiley face" or "Friday, draw the number 9." An ElevenLabs voice agent understands you, and an SO-101 arm driven by an AMD Ryzen AI MAX+ station draws it on paper with a pen. Friday then asks whether it looked right. A "yes" is saved as a +1 rating and pins that version. A "no" is saved as a −1 and lets Friday propose a bounded correction: pen lift height, 9 stem length, smile depth or eye spacing.

On October 11 we ran the full loop live: a spoken request on a Mac went through ElevenLabs and SSH to the AMD controller and then to the physical arm. The smiley and the 9 both completed, and the operator called the 9 "perfect."

How we built it

1. Hardware and capture. A leader/follower SO-101 pair runs through LeRobot, with a pen mounted on the follower's gripper. We teleoperated shapes with the leader arm and logged leader commands, follower joint positions and sent targets at about 30 Hz.

2. Forward kinematics and pen-tip calibration. Joint angles become a pen-tip position through the arm's forward kinematics. The pen's offset in the gripper frame is unknown, so we touched one dot from six different arm poses. For each pose $ i $ with gripper pose $ (R_i, p_i) $, the pen tip $ t $ must land on the same fixed point $ c $:

$$ p_i + R_i\,t = c \quad\Rightarrow\quad \begin{bmatrix} R_i & -I \end{bmatrix}\begin{bmatrix} t \ c \end{bmatrix} = -p_i $$

Stacking all six poses gives a linear least-squares problem. It fit a tip offset of about 38.5 mm, close to the measured 1.5 inches.

3. The paper surface. We captured the corners of a 170 × 235 mm drawing rectangle by touching the pen tip to each mark. Drawing points are written in paper coordinates and mapped to the arm's base frame. For the contact height we fit a plane through observed contact points:

$$ z(x,y) = a\,x + b\,y + c_0 $$

4. Inverse kinematics. To drive the pen to a target point, we solve for joint angles with damped least squares. At each iteration, with position error $ e = x^* - f(q) $ and Jacobian $ J $ from central finite differences:

$$ \Delta q = W J^{\top}\left(J W J^{\top} + \lambda^2 I\right)^{-1} e $$

Here $ W $ down-weights the wrist-roll joint, and we cap the step size, clamp wrist roll near its contact value and reject any solution that exceeds a joint limit. We stop when the error is under 0.35 mm.

5. Shapes and smooth motion. Each stroke is a sequence of pen-tip waypoints. Moves use a minimum-jerk easing curve so the arm starts and stops gently:

$$ s(u) = 10u^3 - 15u^4 + 6u^5, \qquad u\in[0,1] $$

The smile is a circular arc of radius $ r $ and depth $ d $, with $ \theta = \pi(1-u) $:

$$ x = c_x + r\cos\theta, \qquad y = y_{top} - d\sin\theta $$

Between strokes the pen lifts. The 40 mm lift we chose is a modeled lift, not a measured clearance. The 9 is a teleoperated demonstration that we smoothed and reshaped into a rounder loop, with a shortened stem.

6. Modelling the servos. The follower lags its command, so we modelled it as a delayed first-order response:

$$ \dot p(t) = \frac{u(t-\delta) - p(t)}{\tau}, \qquad p_{k+1} = p_k + \left(1 - e^{-\Delta t/\tau}\right)\left(u_{k-d} - p_k\right) $$

We grid-searched the delay $ \delta $ and time constant $ \tau $ across seven recordings. On top of that we added a small ridge-regression correction for Z that switches on only after real motion starts:

$$ g_k = 1 - e^{-m_k/20\,\text{mm}}, \qquad w = \left(\Phi^{\top}\Phi + \lambda I\right)^{-1}\Phi^{\top} r, \quad \lambda = 0.1 $$

where $ m_k $ is the running maximum displacement from the start (with a 1 mm sensor deadband) and $ \Phi $ holds quadratic features of the pen position, each scaled by $ g_k $. Leaving one run out at a time, the mean Z error was 5.93 mm. This predictor is evaluated offline and does not yet correct the arm during drawing.

7. Friction and contact from data, not force sensors. To explore pen drag, we took the residual between the recorded follower path and the friction-free servo model, then projected it against the direction of travel $ u $:

$$ \text{drag} = -R\cdot u, \qquad \text{drag} \approx c_0 + c_1 v $$

The constant part $ c_0 $ behaves like Coulomb friction and the speed-dependent part $ c_1 $ like viscous drag. We treat these as hypotheses: without a force sensor or ink measurement, we do not claim a measured friction coefficient.

8. A simulated twin, built overnight. We rebuilt the arm in MuJoCo, using the recorded data to calibrate it, so we could keep working on drawings in simulation while the physical arm was unavailable.

9. The voice loop. An ElevenLabs agent calls five tools: draw, status, stop, feedback and propose. A Mac runs the microphone and speaker, and the tool calls travel over SSH to the AMD controller. The controller owns the arm's serial ports, de-duplicates requests by ID, and runs one drawing at a time. Ratings and the preferred version of each routine are stored on disk. An optional OpenAI step to turn free-form feedback into parameters is built and tested against mocked responses only. The demo uses propose directly.

Challenges we ran into

We didn't plan to model friction. Originally we only wanted to recreate the arm on a computer overnight so we wouldn't fall behind. Doing that forced us to understand the external forces acting on the robot, and it led to our breakthroughs on contact and friction.
The arm slumped. Our first plan was fully autonomous drawing. During a hover test the shoulder didn't hold its pose and the pen pressed into the paper. We stopped, recorded rough teleoperated circles first, and replayed paths derived from demonstrations.
Our geometry wasn't precise enough. The pen-tip model disagreed with itself by about 10 mm RMS on repeated touches of one dot, and the four corners failed to close by about 22 mm. We report this as a limit instead of hiding it.
Two processes fighting for one arm. Duplicate teleoperation processes held the same serial ports, so the controller now insists on a single owner.
Audio and certificates. The Bluetooth speaker wouldn't pair with the AMD PC, so we moved the voice client to a Mac and sent only tool calls over SSH. A Python certificate error then blocked the voice connection until we pointed it at a trusted CA bundle.
API credits. The OpenAI step hit credit_balance_exhausted, so we kept it as a prototype.
Time. A working loop mattered more than perfect accuracy, so we cut scope to a 9 and a smiley.

Accomplishments that we're proud of

We modelled the servos: delay, time constant and a position-dependent Z correction, fit from our own recordings.
A real physical voice → robot → feedback loop, not a mock-up.
A calibration, recording and prediction pipeline built from our own teleoperated data.
Honest scoping: the repo separates what we verified on hardware from what is simulated or planned.

What we learned

Controlling a robot arm isn't as simple as it seems. Many physical factors come into play, including servo lag, calibration error, pen contact and friction.
We learned how to build a learning environment, from recordings, smoothing and simulation, that lets the robot improve its curves.
A robot arm's joint space is not the paper's coordinate frame, and calibration error shows up immediately when a pen touches paper.
Safe voice tools need request IDs, a single owner for the serial ports, and a stop that always works.
A voice agent has to handle interruptions and half-sentences, not just clean commands.

What's next for AMDRobotDrawerAIModel

Integrate the OpenAI API for on-the-fly adjustments from spoken feedback. It is built and tested with mocked responses, but we ran out of API credits.
Measure the ink with a camera and a known paper scale, so we can report line error, stroke coverage and stray ink.
Apply the response model during playback, and add letters.
Repeat the demo many times to show it works reliably, not just once.