The Arm
first live-subject test (it was a success!)
final live-subject test (also a success!)
Simulation time
big sipper

Third Hand

Inspiration

Kat, one of our teammates, has a friend named John (name changed to protect privacy) who was born quadriplegic. John needs a caregiver and a lot of expensive tech just to get through the day. Picking up a cup, scratching an itch, moving something two feet to the left: all of it means waiting on someone else.

That loss of independence is what this project is about. For a lot of people like John, their voice is the only reliable way they have to interact with the world. But what if their voice is all they need?

Assistive robot arms like this already exist. The problem here is the price. Commercial wheelchair-mounted arms like the Kinova JACO cost $35,000 to over $40,000, and insurance often refuses to cover them. Most families can't afford that on top of caregiving costs. We wanted to see how far we could get this weekend with a low-cost, open-source version, that is also easier to control due to its unique voice implementation.

What it does

Third Hand is a voice-controlled robot arm / wheelchair combo. You say a trigger word ("Hey Arm") followed by a command, and the arm does it.

"Hey Arm, open the hand" / "close the hand" moves the gripper
"Hey Arm, extend" raises the arm into its working pose
"Hey Arm, retract" or "go home" brings it back to rest
"Hey Arm, stop" stops whatever the arm is doing, right away
"Hey Arm, dance" because we had to
"Hey Arm, take me to Walmart" drives the wheelchair to Walmart (not yet reliably implemented)

A wrist-mounted camera streams live to a dashboard, which also shows every motor's position, voltage, load and temperature in real time.

How we built it

The arm is an SO-101, a popular open-source robot arm design. We bought the motors and electronics on Amazon and 3D printed every arm part before the hackathon. The brain is a Raspberry Pi 5 we won at a previous hackathon.

The total cost for us for this project comes out to $223.54. Someone buying the pi and other tools themselves may pay closer to ~$650.

At the start of the hackathon we assembled the arm, set up/calibrated all six servos, and ran test after test until they responded predictably. From there, we built most of the control stack ourselves:

Our own motor controller and movement library on the Pi
Forward and inverse kinematics for our specific arm
A motion planner that moves all the joints together smoothly and within safe limits
Telemetry that logs every motor's position, voltage, load and faults
Voice control: Cartesia turns speech into text, and an LLM (Jev, through OpenRouter) figures out which command you meant

We built our own stack because the standard open-source setups for this arm assume several cameras watching it from different angles. That doesn't work on a wheelchair. The only place you can easily put a camera is on the arm itself. So the existing tools and trained models didn't fit, and we had to write our own logic.

Safety

An arm this close to someone's body has to be safe before it's useful, so we designed for that from the start.

Joint limits. Every joint has hard limits in both software and the servo firmware, so the arm can't swing outside its safe range.
Stops when it meets resistance. The servos constantly report load, voltage and faults. If the arm hits resistance it didn't expect, it stops and holds its position. It never tries to push through, and it never restarts a move on its own.
Voice stop. "Hey Arm, stop" halts motion immediately.
When in doubt, do nothing. If a command is unclear, negated ("don't open the hand"), or asks for two things at once, the arm does nothing.
Off by default. Listening starts off every time the system boots and has to be turned on deliberately. Only the words after "Hey Arm" are ever sent off to be understood. Nothing said before it leaves the Pi.

Challenges we ran into

Power. Our servos weren't getting the power they needed, so voltage sags were constant. When voltage drops, motors fault and moves stop partway through. We tried a bunch of fixes, including rewiring so the worst offender had its own branch instead of sharing voltage with the other servos in the chain. It helped, but the real lesson was that you need the right power supply and servos, and that's not the place to cheap out.

Getting the arm to do what we want. Most robot arm projects now use trained action models (VLAs) that learn tasks from demonstrations. We tested a public SO-101 model and dataset, but they were built for a multi-camera setup, and the converted actions would have pushed our arm past its safe limits in about 18% of frames. Training our own model would be slow, expensive, and unreliable, so we've been trying everything we can to avoid it. Some of those approaches worked better than others. After the hackathon's time constraints are done, we will be working on a proper model for this setup.

Accomplishments that we're proud of

Going from a box of parts to a voice-controlled arm in one hackathon
Writing our own control stack, kinematics and safety checks instead of relying on a setup that didn't fit our use case
Building an arm for ~$650 in parts, compared to $40,000+ for commercial options. Our receipts are in the repo.

What we learned

Physics doesn't care about your code. You can't Claude your way out of not having enough power for your servos, no matter how good your prompt is. We had to think about the physical system first: power, load, weight and where the camera sits all shape what's possible before you can even begin thinking about software.

What's next for Third Hand

The end goal is a full wheelchair system. Using only your voice, you'd control where the chair goes and use an arm mounted on it to interact with the world around you.

Next steps:

Fix the power supply so the arm runs reliably
Mount the arm on a real wheelchair (not the office chair we've been using)
Add voice navigation for the chair itself
Test it with people who would actually use it, and the occupational therapists who work with them

We see a world where people with quadriplegia, cerebral palsy and similar conditions, and their families, don't have to pay huge amounts of money just to do everyday things. That's why all of our code and findings are open source under the MIT license.