FollowMe: AMD Smart Phone Holder

Inspiration

Video calls assume you're sitting still. The moment you start cooking, teaching a craft, fixing a bike or doing physio exercises, you walk out of frame. Tripods don't follow you, and gimbals only rotate in place. We wanted a camera that could move and that knows who it's holding the phone for: follow you around the room, ignore everyone else, and do it all without you touching it. A robot arm holding a phone turned out to be the right body for that.

Uses

Cooking and crafts: stay in frame while you move between the counter, stove and sink.'
Live streaming: the camera follows the streamer, not whoever walks behind them.
Lectures and classrooms: follows the teacher across the front of a busy room.
Telehealth and physiotherapy: keeps the patient in view while they move through exercises.
Accessibility: for people who can't easily reach or reposition a camera themselves.
Watching Netflix: lie back on the couch or in bed and the phone stays right in front of your eyes, turning and moving with you as you shift position, so you never have to hold it up.

What it does

FollowMe mounts your phone on an SO-101 robot arm and turns it into a hands-free camera operator.

Follows you: keeps your face centred as you move around the room, holds a comfortable distance, and switches to whole-body tracking when you turn away.
Knows who you are: scan your face once, and FollowMe follows only you, even with other people in the frame.
Understands hand gestures:
- ✋ Hold up an open palm: the arm freezes in place.
- 👍 Thumbs up: it resumes following.
- 🤟 Make an L with your left hand: the phone turns to landscape.
- 🤟 Make an L with your right hand: the phone turns to portrait.
Runs locally: everything runs on an AMD Ryzen AI PC, so your video and face data never leave the room.

How we built it

The system is split into four isolated layers, so a failure in one can never make the arm misbehave:

camera → perception → gesture intent → motion planning → motor worker → SO-101

Models, training and data. We built on pretrained models and finetuned as necessary.

Job	Model
Find people	YOLOv8n (ONNX, 320 px)
Find faces and landmarks	YuNet (ONNX)
Recognize whose face	SFace (ONNX)
Hand skeleton and gestures	MediaPipe Gesture Recognizer
Face and body landmarks	MediaPipe Face and Pose Landmarker

Perception: finding you. The tracker locks onto the largest person in view, then keeps following whoever is nearest to where you just were. A jump limit stops it snapping to someone walking past, and smoothing removes detector jitter. Inference runs in its own process and Python environment with no robot code, and talks to the controller over a versioned socket protocol.

Face recognition: knowing it's you. Detecting a face is not the same as recognizing your face. We added identity on top of detection:

Enrollment (data collection) in under a minute: type your name in the live camera page and click Enroll face. FollowMe captures 20 samples of your face while you turn your head and move around, so it learns you from different angles, distances and lighting. Frames with zero or several faces are skipped automatically, so only your face is collected.
Turning a face into numbers: for each sample, YuNet finds the face and its landmarks (eyes, nose, mouth corners). The face is aligned using those landmarks, and the pretrained SFace network converts it into a 128-number "faceprint". Similar faces produce similar faceprints.
Matching: every face in the frame is compared against your saved faceprints using cosine similarity, which measures how closely two faceprints point in the same direction:

$$ \text{similarity} = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert} $$

The best match above a threshold (0.45) is labelled with your name; anything lower is "unknown".

Follow one person: in follow-face mode, only your recognized face can drive the arm. Other people in the shot are ignored.
Privacy by design: only the faceprints are stored, never photos, on the local machine and excluded from version control. Re-enrolling replaces your old samples, and a cancelled enrollment changes nothing.

Gesture intent: knowing when you mean it. Recognizing a thumbs up is easy; not firing while someone chops onions is the hard part. On top of MediaPipe's 21-point hand skeleton we added:

Hold-to-confirm: gestures must be held before they count (0.7 s for freeze and resume, 0.5 s for the L). Gaps under 0.2 s (a blurred or dropped frame) don't reset the hold.
A face-relative zone: only hands raised near your face count, so hands working at the counter are ignored.
Latch-until-release plus a 1 s cooldown, so each gesture fires exactly once.
A custom L-sign detector: MediaPipe has no "L" gesture, so we recognize it from the hand skeleton directly. The index finger must be straight, the other three fingers curled, and the thumb must stick out at 50°–130° to the index finger. MediaPipe's left/right-hand classification then picks landscape or portrait.

Motion: thinking in space, not in joint angles. A five-joint arm is hard to drive by hand: move the shoulder and the phone swings up and forward and changes angle all at once. So FollowMe never reasons about individual joints. It reasons about where the phone should be in 3D space, and lets inverse kinematics figure out how the arm gets it there.

1. See where you are. From a single ordinary camera, FollowMe estimates your position in 3D. Where your face sits in the picture tells it which direction you are, and how large your face appears tells it how far away you are: a face that looks half as wide is about twice as far.

$$ d \approx d_{\text{ref}} \cdot \frac{w_{\text{ref}}}{w_{\text{face}}} $$

These estimates are filtered over time, so the arm responds to where you're actually going rather than to a single noisy frame.

2. Turn toward you. Small corrections, like you drifting left or standing up, are handled by a fast feedback loop that turns the base and tilts the phone to pull your face back to the centre of the shot. It continuously measures the error in the picture and corrects it, like steering to stay in a lane, easing off as you settle back into frame.

3. Move the phone through space with inverse kinematics. This is the core of FollowMe. When you step back, rotating isn't enough; the phone itself has to travel toward you, which means shoulder, elbow and wrist must move together in exactly the right combination. Instead of hand-tuning that combination, we describe the goal in space and solve for the joint angles that achieve it.

Forward kinematics answers "given these joint angles, where is the phone?":

$$ T_{\text{phone}} = \mathrm{FK}(q) $$

Inverse kinematics answers the much harder reverse question: "given where I want the phone, what joint angles get it there?" There is no simple formula for that on a real arm, so we solve it numerically. Using LeRobot's placo/Pinocchio solver and a full 3D model of the SO-101, each step finds the joint configuration that best matches the target position, while also trying to keep the phone's viewing angle:

$$ q^{*} = \mathrm{IK}(p_{\text{target}}) $$

Position is weighted most, so the phone always goes to the right place, and orientation keeps the shot steady and level as the arm reaches.

Distance estimate: works out how far away you are, which decides where the phone should go.
Inverse kinematics: works out the joint angles to get it there.
Forward kinematics: checks that those angles really put the phone in the right place before the arm moves.

The solver works from the SO-101's URDF, a digital blueprint of the arm that records every link's length, every joint's axis of rotation and its angle limits, so it can compute exactly where the phone will be for any combination of joint angles.

Challenges we ran into

Integrating many models: person detection, face detection, face recognition, hand gestures and pose landmarks all had to run together on every camera frame, fast enough to keep up with a moving person. Their dependencies clashed with the robot stack, so we split vision and robot control into separate processes and environments that talk over a socket.
Hardware: with several teammates and tools on one machine, two programs would grab the same serial port or camera and crash each other.
Flaky daisy chains: intermittent cable connections made motors drop off the bus mid-run and the arm jitter. We learned to diagnose by pinging each motor ID before blaming software.

Accomplishments that we're proud of

Follow-one-person tracking: enroll your face in under a minute, and the arm ignores everyone else in the shot.
Full 3D following from a single ordinary camera: direction from face position, distance from face size, and inverse kinematics to move the phone.
A gesture system that is deliberately hard to trigger by accident, including a custom L-sign detector built from the raw hand skeleton.

What we learned

With physical robots, most bugs are cables, power and calibration. Verify the hardware state first.
Good gesture UX is mostly about not responding.

What's next for FollowMe

More gestures, including beckoning the phone closer, plus voice commands and saved viewpoints ("look at the stove").
Switching the followed person on the fly, by gesture or voice ("follow Alice"), and two-person framing.