GLOVE

The Glove

Inspiration

We grew up with a deaf friend. She never chose to be deaf but she was the one paying for it, working harder than anyone to be included, and yet still turned away from regular classrooms. We hated the constant discomfort of watching someone not even get the chance to experience school the way the rest of us did. Our first idea was a computer-vision model that would point a camera at her hands and translate her signs. She hated it, saying that people constantly pointing a camera at her and making her feel different is exactly what she wanted to avoid.

So we built Glove Studio, a glove with sensors that speaks signs out loud, letting her conversations flow as naturally as everybody else's.

What it does

Glove Studio turns one-handed ASL signs into spoken English. The user wears a glove with flex sensors that determine the position of the fingers. The sensor data is then sent to a program that interprets it and tries to match it to a valid sign. If it sees a sign it was never taught, it says "UNKNOWN" instead of guessing, because for an accessibility device a wrong word is worse than no word. If a valid sign IS found, then the ASL grammar is converted into spoken English grammar and is converted into speech.

How we built it

We drafted a full wiring diagram, then soldered, trimmed and sewed everything onto the glove:

flex sensors on the fingers, read through a 16-bit ADC;
a repurposed drone flight controller for tilt and motion;
an Arduino UNO Q managing all data streams.

Every 20 ms the glove streams 5 parameters to a laptop.

Each flex sensor is a resistor that changes as it bends, so we read it through a voltage divider:

$$ V_{\text{out}} = 3.3\,\text{V} \cdot \frac{R}{R + R_{\text{flex}}} $$

The fixed resistor that gives the largest voltage swing between a straight finger ($R_s$) and a bent one ($R_b$) is

$$ R^{*}_{\text{fixed}} = \sqrt{R_s R_b} $$

Open-hand/fist calibration normalizes every finger so 0 means straight and 1 means bent, for any hand:

$$ \hat{f} = \frac{f - f_{\text{open}}}{f_{\text{fist}} - f_{\text{open}}} $$

The flight controller's roll $\phi$ and pitch $\theta$ become a gravity vector that tells us which way the hand is tilted:

$$ \mathbf{g} = \begin{pmatrix} -\sin\theta \ \sin\phi\cos\theta \ \cos\phi\cos\theta \end{pmatrix}, \qquad \lVert \mathbf{g} \rVert = 1 $$

The app detects where each sign starts and stops. Then two models vote on the most likely sign: a random forest and a dynamic-time-warping nearest neighbor.

$$ P(\text{sign}) = 0.65\,P_{\text{RF}} + 0.35\,P_{\text{DTW}} $$

Dynamic time warping lines up two movements even when parts of one are faster than the other, by finding the cheapest alignment between a new sign $X$ and a saved recording $Y$:

$$ D(i,j) = \lVert x_i - y_j \rVert^2 + \min{\,D(i-1,j),\; D(i,j-1),\; D(i-1,j-1)\,} $$

A sign is only accepted if the models are confident and it looks like something the glove has seen before:

$$ \max_c P(c) \ge 0.60 \quad \text{and} \quad \min_m \text{DTW}(X, Y_m) \le \tau, \qquad \tau = 1.5 \times P_{95}{d^{\text{same}}} $$

Here $\tau$ is 1.5 times the 95th percentile of the distance from each training recording to its nearest recording of the same sign.

If confidence falls below 60%, the answer is UNKNOWN. Recognized signs go to Claude Haiku, which turns ASL word order into natural English, and the ElevenLabs text-to-speech API speaks it aloud.

Challenges we ran into

Repurposing drone parts. Flight controllers aren't designed to be anything but flight controllers. We had to learn the drone's own communication protocol, and its gyro reported raw counts with no units. We measured the scale ourselves by rolling the board against gravity.
Our IMU overheated and shorted. Mid-build, the motion sensor short-circuited. We briefly fell back to using a phone as a motion sensor while we searched for another IMU.

What we learned

An ADC measures voltage, not resistance, so a flex sensor needs a fixed resistor beside it before it can be read at all.
Most of our hardware time went into wiring faults and ensuring a compact product.
ASL has its own grammar, and relies heavily on facial expression and body language in pair with signs.

What's Next

More streamlined hardware
AR Glasses that take audio input from people speaking to the user and display a small avatar signing in the frame to go with the gloves