hand.shake

Inspiration

So much of independence comes down to the hands. Carrying a bag of groceries, holding a full mug, keeping a grip on a railing. Small acts that quietly assume you can produce and sustain enough force to hold on. For people with weakened hand strength or limited motor control, whether from injury, age, neurological conditions, or disability, that assumption breaks down. The intent to grip is there. The hand just can't hold the object the way it needs to, or can't keep holding it long enough to matter.

Most responses to this are either passive or all-or-nothing: a brace that doesn't adapt, or a rigid gripper that takes the task away from the person entirely. What felt missing was something that worked with the user's own effort: not replacing their grip, but boosting it so their intention actually holds the object. Assistance that follows the person, instead of overriding them.

That's the premise we built around. The glove reads what the user is trying to hold and amplifies their grip through a tendon-actuated finger, enough force to secure the object, scaled to what the object needs so a light touch stays light and a heavier load gets a firmer hold. A camera identifies what's being picked up and sets that target, so the assistance meets the object rather than clamping down blindly. The goal is simple: let someone hold and carry the things they want to, using their own hands, with the glove making up the difference.

What it does

hand.shake is a wearable grip-assist glove that helps someone hold and carry objects using their own hands, with the glove supplying the force their grip can't.

Here's the loop it runs:

Sees the object. A Raspberry Pi Camera Module 3 on the glove captures what the user is reaching for. The frame goes to a vision model that identifies the object and decides how it should be held — a fragile item gets a gentle hold, an everyday object a medium one, a heavier load a firm one.
Sets a target force. That grip class maps to a concrete target the glove aims for — roughly 2N for gentle, 6N for medium, 12N for firm — so the assistance is matched to the object instead of clamping down at full strength on everything.
Assists the grip. A tendon-actuated finger drives toward that target force, boosting the user's own effort to secure the object. It follows the user rather than overriding them — their intention still holds the object, the glove just makes up the difference.
Holds it steady. Instead of a one-shot squeeze, the glove regulates toward the target and keeps the force there while the object is held, so the grip stays stable through the carry.
Shows what's happening. Live telemetry from the glove streams to a dashboard HUD — the detected object, the chosen grip, the target versus actual force — so you can see the system reasoning and reacting in real time.

Under the hood it's a Raspberry Pi 5 handling the camera and vision, talking wirelessly to a microcontroller on the glove that closes the force-control loop on the tendon.

How we built it

We split the system into three parts that talk to each other over the network: perception, control, and the glove itself.

Perception. The detection service takes a JPEG frame and returns a structured decision — the object, a grip class, and a target force in newtons. We ran it against free vision models through OpenRouter, with a primary model and a fallback so a single flaky response doesn't stall the pipeline. The whole detection package is written in stdlib-only Python so it can run directly on the board's Python without pulling in extra dependencies. It runs as a small HTTP server: the Pi POSTs a frame to /detect and gets back JSON with the object, grip, force target, and the model's reasoning.

The board. Vision and camera run on a Raspberry Pi 5 with the Camera Module 3 Wide (IMX708). The Pi captures frames, sends them to the detector, and turns the returned grip class into a force target for the glove.

The glove and control loop. A microcontroller on the glove drives the tendon that actuates the finger. It takes the target force coming from detection and drives the tendon toward it with a proportional follow, so the assist ramps to the object's target rather than snapping to full force. The Pi talks to the microcontroller wirelessly over WiFi, which keeps the glove untethered from the compute.

The HUD. We route live telemetry from the glove back out to a dashboard so the object, grip decision, and force are visible in real time — useful both as a demo surface and as our main debugging window into what the control loop was actually doing.

Challenges we ran into

QNX, not Linux. The target board ran QNX 8 on the Pi 5, not a normal Linux stack. That shaped a lot of decisions — most notably keeping the detection code stdlib-only, since we couldn't lean on the usual pip ecosystem. A lot of "this would be one import on Linux" moments turned into "do it with the standard library."
Cutting the cord. We started with the Pi talking to the glove's microcontroller over USB serial, then moved the link to WiFi so the glove wasn't physically tethered to the compute. Getting a reliable wireless link — creds, addressing, keeping telemetry flowing — took real iteration.
Hardware that fights back. At one point the microcontroller looked fried, so we stood up an ESP32 variant of the glove firmware as a fallback path to de-risk the demo rather than betting everything on one board surviving.
Free vision models are free. Using no-cost vision models meant working around latency and the occasional bad or empty response, which is why the detector has a fallback model and returns structured output we could validate rather than trusting raw text.
Tuning the assist. Making the tendon drive feel like assistance rather than a clamp meant tuning the proportional follow so force ramps toward the target smoothly instead of grabbing.

Accomplishments that we're proud of

A full end-to-end loop that actually runs: point the camera at an object, and the glove picks a grip, sets a target force, and drives the tendon to it.
Getting real work running on QNX on the Pi 5 under a stdlib-only constraint, instead of falling back to a comfortable Linux environment.
A wireless, untethered glove — the compute and the hand aren't wired together.
Object-aware force, not just on/off assist: the same glove holds a fragile item gently and a heavier one firmly, based on what it sees.
A live HUD that makes the whole pipeline legible in real time — you can watch it decide and react.

What we learned

Framing the problem correctly matters more than the mechanism. The build only made sense once we were clear it should amplify the user's own grip, not replace it — that decision drove the whole control approach.
Working inside QNX's constraints. Building for QNX rather than Linux, and staying stdlib-only, taught us a lot about not reaching for a dependency by reflex.
Structured outputs make LLMs usable in a control path. Forcing the vision model to return a validated object/grip/force object — with a fallback model — is what let a probabilistic component sit in front of hardware without constant breakage.
Design for hardware failure. Standing up the ESP32 fallback taught us to keep a second path ready when a single board can end the demo.
Wireless changes the ergonomics. Moving the Pi↔glove link off USB serial made the glove feel like a wearable instead of a tethered rig.

What's next for hand.shake

Closer force feedback. Tighten the loop with better force sensing so the glove holds a true target rather than following an open estimate, and stays steady under changing load.
Faster, on-board perception. Cut the round-trip latency — ideally moving detection on-device so the glove reacts instantly and works without a network.
A richer grip vocabulary. More than three grip classes, plus handling for shape and orientation, not just a single force level per object.
Form factor. Shrink and soften the hardware into something genuinely wearable and comfortable for all-day use.
Real user testing. Put it in the hands of people with limited hand strength or motor control and let their feedback drive the next iteration — the whole point is that it works for them.