Inspiration
My aunt was paralyzed from the neck down after a car accident. Only her eyes and her mouth still move. Every drink, every pill, every time something slides out of reach, she has to wait for someone.
She is not alone: about 308,000 Americans live with a spinal cord injury, 6 in 10 new injuries cause quadriplegia, and vehicle crashes are the #1 cause. Assistive robot arms exist, but cost $35,000 or more. So we built her a hand.
What it does
Ahandformyaunt is a robot arm you control with your face:
- Your nose steers. The arm follows your nose, live.
- Your mouth grabs. Open your mouth to pick up, open it again to let go.
- Or just say it. "Grab me an orange": Amazon Nova finds the orange with a camera, and the robot works out how to pick it up.
How we built it
- Hardware: ROBOTIS OpenMANIPULATOR-X arm, NVIDIA Jetson Orin NX, a webcam, ROS 2.
- Face control: MediaPipe Face Mesh tracks the face; our own arm maths moves the arm with safety limits (slow speed, 12 cm above the table, a STOP that works offline).
- AI: Amazon Nova on Bedrock. Nova 2 Lite finds objects in the camera picture, and Nova 2 Sonic speaks.
- Quality: over 180 automated tests, run on GitHub for every change.
Challenges
- Controlling the robot properly was hard. At first it didn't follow the face reliably. Eyes alone were too jumpy, so we switched to following the nose, which is far steadier.
- Inference with Amazon's online models was too slow for real time. We keep it to one call per picture, and all real-time control runs on the robot itself.
- The robot couldn't work out what actions it needed to grab an object. Finding the orange wasn't enough: we had to turn "it's here in the picture" into arm moves, by calibrating the table camera to the arm's positions.
What we learned
This was our first robotics project. We learned to:
- assemble the robot and control its servos,
- control a robot in real time,
- do real-time object detection and connect it to robot control,
- use Amazon's newest models for object detection,
- let the robot find its own actions to grab an object,
- optimize the whole workflow so it runs in real time.
Where it stands
Head control and pick-up work on the real arm; the new mouth gesture is being tested on it now. Finding and fetching objects with Nova works on a simulated arm, and the real arm is next. This is a prototype, not yet tested with people who have quadriplegia, and not a medical device.
Ahandformyaunt: independence, one pick-up at a time.
