Dermis: Touch for Robots

Inspiration

Both of my grandparents had strokes, years apart. Both lost the use of their right side. Recovery depended on therapy, and therapy depended on getting to a clinic, finding a therapist with time, and having someone at home to help. When the sessions ended, the exercises mostly stopped. Nobody could tell whether a squeeze was too weak, too hard, or done with the wrong fingers.

The numbers show how common this is:

Nearly 94 million people worldwide are living after a stroke, and about 12 million more have one every year. For most of them, therapy ends long before recovery does.
About 2.3 million Americans live with limb loss. Roughly 1 in 4 adults with a prosthetic arm stops using it.
88% of prosthetic users say sensory feedback matters, and grip force is the feedback they want most.

Today's prosthetic hands can't tell an egg from a mug, and today's robots can see the world but can't feel it. Both problems come down to one missing piece: skin.

What it does

Dermis is a soft, stretchy skin made of magnetized silicone over tiny magnetic sensors. When you press it, the magnetic field shifts, and Dermis turns that shift into force. A single patch senses four things:

Pressure: how hard each fingertip and the palm are pressing, in newtons.
Slip: the moment an object starts to slide, before it falls.
Movement: the hand's orientation and path, from a built-in motion sensor.
Temperature: at every pad.

For Cal Hacks we built Dermis into a glove for stroke rehabilitation at home, with an app made for the person doing the exercises, not for an engineer:

One instruction at a time. "Pinch thumb and index." "A little more." "Hold… 3… 2… 1." "Now let go slowly." The cues are short, in large type, can be read aloud, and are designed for people who may be tired, have low vision, or be recovering from a brain injury.
A meter you can read at a glance. A dot slides along a bar marked "Too light / Just right / Too hard," and a ring fills while you hold. There are no numbers unless you turn them on.
It notices compensation. Stroke survivors often recruit the wrong fingers. If the middle finger joins a pinch, the pad turns amber and the app says "Relax your middle finger."
Every rep is measured. The app detects each rep automatically and records peak force, time in the target band, steadiness, how quickly force builds, and release time.
Claude reviews every session. With one tap, Claude reads the full session data and writes two reviews. The patient gets a short, warm review in plain words: what went well and one thing to try next time. Their therapist gets a section of clinical notes on force control, compensation, fatigue across the set, and data quality. Progress becomes something you can see, even when no therapist is in the room.

How we built it

Dermis Architecture

Hardware. Four sensing pods sit on the thumb, index and middle fingertips and the palm. Each pod is an MLX90393 3-axis magnetometer under magnetized silicone, read over SPI by an ESP32 hub, with a BMI160 IMU for motion. The hub zeroes itself at boot, applies a per-pod force calibration, and streams frames over a versioned wire protocol with checksums.

Signal processing. Force is the projection of the field change onto each pod's calibrated press direction, scaled by a measured gain. That gave us under 1.3% error when we recomputed force from raw field data. Sideways field change is treated as shear, and sudden direction changes under load are flagged as slip. A complementary filter on the IMU tracks hand orientation.

Rehab engine. A state machine (rest → loading → hold → release) tracks every exercise in real time and splits the stream into reps. We tested it on a real 50-second glove capture: it found all 8 presses and correctly classified each as a single-finger press, palm press or three-finger pinch.

Apps. We built three things:

Dermis Hand Practice: the patient app. It is a single HTML page that connects to the glove over WebSocket and also includes a simulated glove, so anyone can try it.
Dermis Rehab Console: a full engineering view with a live pressure heatmap, 3D terrain, slip and shear vectors, and a session timeline.
A replay server that streams real captures at their original timing for testing.

SpacetimeDB: shared real-time backend. A TypeScript module holds sessions, live state (updated ~10×/s), a raw-sample archive, observer notes and simulator job requests. Every write goes through a reducer, and every browser updates instantly. Judges scan a QR code to watch the same live robot session on their phones, add notes or request a trial; local workers run the simulator and publish results back. It runs locally or on Maincloud, with no open ports needed.

Mana: live dashboard. We built Dermis Live in Mana to visualize the glove's incoming data streams and frames as they arrive. We also defined an open stream format (dermis.sim_stream/1) so Mana can draw the actual Dermis robot hand.

Fetch.ai: touch-intelligence agents. 14 uAgents (8 hosted on Agentverse) speak the Chat Protocol, so you can ask them questions in plain English in ASI:One (chat with @dermis-hub).

A hub routes each question to specialists (humanoid, warehouse, prosthetics, sensor health and more) and combines their answers.
All numbers come from real analysis of the recordings (force, slip vibration, sensor noise, dropped frames), not LLM guesses.
A visualizer agent builds charts and dashboards; a data-explorer agent analyzes any CSV or JSON.
Answers include interactive cards, and a certified report costs 0.1 FET via the Payment Protocol (testnet).
The hosted agents read a snapshot of 192 simulated recordings, so they stay online after the hackathon.

Claude. Each session becomes a structured data file: settings, every rep's metrics, the cues shown and when, force on every pad 10 times a second, and earlier sessions for comparison. Claude gets this file with a prompt that sets the two audiences, a fixed output structure, and firm safety rules: no diagnosis, never raise the difficulty on its own, and flag simulated data. The review streams into the app as Claude writes it.

Challenges we ran into

Physics is messy. Silicone creeps after a hard press, so the zero point drifts. Shear can look like negative pressure, and pads saturate near 1 kgf. We had to handle each case explicitly so a sideways push isn't counted as a rep and a ceiling reading isn't counted as real force.
Designing for a tired, recovering user. Our first dashboard was powerful but overwhelming. We rebuilt it around accessibility basics: one instruction at a time, big tap targets, color-blind-safe states with words alongside colors, cues that don't flicker, and encouragement instead of scores.
Making AI feedback safe. A rehab coach must never push someone into pain. We limited Claude to observations grounded in the data and kept any change to the plan with the therapist.

Accomplishments that we're proud of

A working sensing skin that measures force from magnetism, with a calibration we can check against raw data.
Rep detection that matched what actually happened on a real glove recording.
A patient app that someone recovering from a stroke could use alone.
Claude reviews that turn raw pressure data into words a patient understands and notes a therapist can act on.

What we learned

Touch is data. Every squeeze in a rehab session is also a labeled example of how a human hand controls force, which is exactly what robots lack. We also learned that the most useful interface for a patient is the one that shows the least.

What's next for Dermis

Clinical pilot: testing with stroke survivors and occupational therapists, and comparing glove measurements with standard grip assessments.
Care-team sync: a therapist sees a patient's session live, adjusts their plan remotely, and sends encouragement in real time.
Prosthetics: using Dermis on prosthetic fingertips to send grip force back to the wearer through vibration.
Robot learning: paired human-glove and robot-hand recordings, so robots can learn force control directly from people.
Smaller pods and more of them: to cover the full hand.