Guitarra

Inspiration

In 2026, more than half of Canadians and Americans remain skeptical about AI, according to studies by, notably, the Pew Research Center. And the only models generally accessible are text/image/video-generation models. Accordingly, we only have one real "shot" at making the ChatGPT moment of robotics phenomenal, accessible, and demonstrate the gentler - even showing automatons' artistic side.

A guitar-playing guitar satisfies this. A robot which listens to itself play, works out what went wrong, and tries again, was built on a $150 arm in a weekend. We wanted the closed loop from the GPT-6 Astra painting demo (propose, execute, observe, revise) applied to music, where the feedback comes from a microphone hearing a note instead of a camera looking at a canvas. We also wanted a bandmate: a robot lamp to dance, introduce the songs, and hand the guitarist the spotlight.

What it does

Guitarra is a two-robot band.

The guitar arm is an SO-101 robot arm which taps real notes on a real electric guitar. You type a musical idea into a web app. A pretrained planner (Kimi, served on Baseten) turns it into a note plan drawn from an operator-calibrated 18-position fret grid. The arm plays it. Your browser mic records the take. A pretrained audio model on Baseten grades what it heard, and the planner proposes one bounded revision, such as a timing nudge or a swapped note, which you approve before it plays again. Every attempt is stored with its audio, review, and plan. The model weights never change. The plan is what improves.

The lamp is a LeLamp, a five-axis robotic lamp on a Raspberry Pi 5, acting as frontperson. It learned nine operator-taught actions, composes them into beat-matched dances synchronized to YouTube soundtracks, and has a teaching UI for adding more.

How we built it

Hardware: two SO-101 arms (Feetech STS3215 bus servos, 12 V), a LeLamp on a Pi 5, an electric guitar, and five USB servo adapters. More on the adapters below.
Guitar arm control: a LeRobot plugin for the arm, a keyframe GUI for recording poses with fingertip XYZ, and a v3 toolset with a full 18-cell fret grid, row-hub staging, and XYZ extrapolation so the arm can reach cells it was never explicitly taught.
Rehearsal loop: a Flask web app (arm_controller/webapp.py). Prompt, Baseten note plan, real taps, WebAudio capture, Baseten audio assessment, one Kimi revision, confirmed replay. Hard bounds everywhere: 4 notes per take, 60 s capture, 3 attempts per revision chain. Revisions are limited to one category at a time (pauses in 50 ms steps, one adjacent swap, one same-pitch alternative). Proposals expire after five minutes and revalidate the calibration fingerprint.
Lamp: an SDK adapter which waits for terminal execution results instead of the dashboard's premature "OK," a camera watchdog, and a continuous five-axis choreography composer with a drift stop on the loaded elbow.
Sim: a MuJoCo rig with two SO-101s and a guitar, plus an analytic string model (press depth, fret offset, pluck speed become note, buzz, and audio) so the agent loop runs with no hardware. It includes a blind benchmark where physics constants are sealed per seed, forcing the agent to learn the rig's limits from feedback alone.
Everything the model cannot be trusted with lives in local code: trajectories, timing, torque limits, stop and fault handling. The model picks what to play. Deterministic executors own when and how.

Challenges we ran into

The bus servo adapters. Three days of "the arm won't respond." Servos lit, power confirmed at 12.2 V by the servos themselves, and the bus dead silent. We had been hot-plugging 3-pin connectors with 12 V live, which momentarily shorts VCC into DATA and kills the adapter's 3.3 V transceiver while the LED and USB keep working perfectly. Two of five boards died this way and looked identical to good ones. We diagnosed it by sending known payloads and watching the board echo its own transmission back. Nothing else was driving the line.

Connectors, not chips. Servos answering 40% of pings, drifting with arm posture, position reads failing while pings got through. Not a single servo ever tested dead. The cause was a pushed-back pin in one socket. A dead servo answers 0% of the time, never 40%.

broadcast_ping lies. LeRobot's discovery uses a broadcast which one flaky servo can garble for everyone, so lerobot-calibrate reported "missing motors" while three servos were clearly alive to unicast pings.

The gripper got hot. A sustained hold on the pick read 75 °C. We cut grip torque from 180 to 100, verified 47 °C over a check, and wrote it into the operator rules as a hard gate. Passing software tests does not clear a thermal issue.

The 0/4095 seam. A saved pose at encoder position 17. Any goal across the wrap drove the arm the long way around into its mechanical hard stop, three times, before we understood why.

The lamp's elbow. Under load it misses targets by 6 to 8 units. Twenty-one physical trials to map what each axis can actually do before choreographing around it.

Accomplishments that we're proud of

A real arm tapping real notes on a real guitar from a typed prompt, with a full-song take grading loop.
The closed loop working end to end: audio in, assessment out, revision proposed, replay confirmed, with every take persisted.
Nine taught lamp actions composed into beat-matched dances synced to video soundtracks.
The safety posture: fail-closed admission, bounded budgets, one-category revisions, and a written operator doc stating exactly what is qualified and what is not. We can point at every claim.
The sim's blind benchmark: after hardening, the best possible fixed guess wins only 26% of seeds. The agent genuinely has to search.

What we learned

On a shared serial bus, everything resembling a dead servo is a connection until proven otherwise. Intermittent means physical. Deterministic means software.
LEDs prove VCC only. The data wire is an independent conductor and fails on its own.
Never touch a 3-pin under power. This one habit accounts for most of what broke.
Better reasoning cannot rescue a bad action interface. When depth error turned out to be systematic by fret, no amount of agent cleverness could fit a single global press_mm into a narrow window. The fix was a per-fret parameter, a schema change rather than a smarter prompt.
Label the boards.

What's next for Guitarra

Bring the pluck arm back on its own adapter and supply, so one arm frets while the other picks, the two-arm scheduling the design was written for.
Qualify hover transitions between fret cells so the planner can select shorter paths it currently is not allowed to invent.
Put the lamp and guitar on one musical transport with a shared clock, so a strum never waits on a model response.
Run the same seeds against GPT-6 Astra and against us, and find out who learns the rig faster.