FingerScribe

Writing on wall
Write on hand
Clean intuitive UI
Exploded assembly
Writing on thigh in bus
Built in mic, hold button to speak
USB-C rechargeable
Smart knowledge maps

Inspiration

I get headaches looking at a screen in a car or on a train. Annoyingly, that's exactly when I have my best ideas: sitting still, alone, with nothing to do but think. I'd either lose the thought or get a headache saving it.

Talking to an assistant out loud on a train isn't an option, so I wanted something quieter. What if I could just write with my finger, on my leg or inside my pocket, and have it turn up as text later? No phone in my hand, no voice, nobody noticing. That's FingerScribe.

What it does

FingerScribe is a thin black sleeve worn on the index finger. It connects to your phone over Bluetooth. You write in your normal handwriting on whatever is under your finger (a thigh, a desk, the inside of a coat pocket) and the words appear on your phone while it stays in your bag.

Beyond plain text:

It gets names, places, numbers, email addresses and phone numbers right, and adds punctuation, so you don't need to look down and correct it.
It learns your vocabulary. When you write a friend's name or a café it hasn't seen, that word is added to a dictionary kept on your phone, and next time it's read offline.
Simple drawings become actions: a box turns a line into a to-do, a star marks it, a question mark saves it for later.
With headphones in, you can ask it something by writing or holding the button and speaking, and it answers in your ear.
In the evening it turns the day's fragments into one journal entry in your own words, shows your original strokes underneath, points out things you've written about before, and answers any questions you jotted down.

How we built it

We're a team of two and built it in about three weeks. Isum did the signal processing, machine learning and bulit the app . Sofia did the mechanical and product design, prototyping and parts.

The hardware. Our first prototype was a Bosch BMI160 motion sensor wired to an Arduino. It showed a fingertip gives a usable signal, but nobody could wear it. We moved to a Seeed XIAO nRF52840 Sense, which fits a motion sensor, microphone, Bluetooth and battery charging onto a 21 × 17.5 mm board. We wanted our own PCB, but after talking to manufacturers about lead times it wasn't possible in the hackathon window. The firmware streams six-axis motion at 100 Hz over Bluetooth, and a button streams audio from the onboard mic for voice notes.

From motion to words. Our first recogniser could only read single, large, slow letters with a pause between each, which is not how anyone writes. Fine-tuning a published model built for sensor pens didn't work either: a finger bends and drags the whole hand with it in a way a pen doesn't. What did work was splitting the problem. One small network reconstructs the path the fingertip drew (within about 2 mm of the real stroke on writing it hadn't seen). A second, whole-word reader turns the motion into words. Both run on the phone in JavaScript, at 0.1–0.35 s a word, so writing works without signal.

Real handwriting. The reader has to cope with print and joined-up writing at normal speed, decide where one word ends and the next begins without pauses, and not split a word when you lift your finger to dot an i. On normal-speed print it reads 68 of 80 words correctly (85%), tested on recordings the model never trained on. That's one writer so far.

Using NVIDIA's models. A motion reader on its own gets the shapes right but not the meaning, and it only knows the 13,000 words in its dictionary. We added a language layer with one rule: the model can suggest, but the finger has the final say.

Nemotron 3.5 Lightning, on Nebius Token Factory, sees the user's whole journal (it has a 1M-token context) and suggests the words they most likely meant, including names and places, plus punctuation. It's quick and cheap enough to run after every sentence.
The phone then checks each suggestion against the actual finger movement and throws out anything the finger couldn't have written, however sensible it sounds. This stops the model from putting words into someone's journal.
Names or places that aren't already in the journal are checked with Tavily before they're accepted.
Our reader only knows the letters a–z, so for numbers, emails and drawn symbols the app renders the stroke as an image and Nemotron 3.5 Super VL reads it.
In the evening, Nemotron 3 Super acts as an agent: it decides what to look up with Tavily, then writes the day's entry.
Voice notes are transcribed by NVIDIA Parakeet, packaged to run as a Nebius AI Cloud serverless endpoint, and the training pipeline is packaged as a Nebius serverless job.

The sleeve. The motion sensor has to sit on the last segment of the finger, where the writing happens; any further back and it can't see the strokes properly. The battery goes on the segment behind it so the tip doesn't feel heavy. We tried clips and caps, then settled on a one-piece compression sleeve, the kind basketball players wear, because it reads as sportswear rather than a gadget or a medical device. It leaves the fingertip uncovered so you can still feel the surface, fits a range of hand sizes, and can be worn either way round for left-handers. We're designing it with no holes for ports or buttons, which is what will let it handle sweat and rain, and means charging has to be wireless.

Challenges we ran into

A finger is not a pen. Pen datasets transferred badly, and a reader trained only on simulated finger motion recognised none of our real words. More real writing from a real finger is what improved accuracy.
Fingertip landings last 2–5 ms, which our 100 Hz stream missed most of the time. We moved tap detection onto the sensor chip itself, running at 833 Hz.
Writing at normal speed: finding word boundaries without pauses, and not breaking words on an i-dot or a t-cross.
Trusting a language model with someone's private notes. Checking every suggestion against the motion was our answer.
Fitting a sensor, battery and button onto about 2 cm of finger without it looking like a device.

Accomplishments that we're proud of

85% of normal-speed handwritten words read correctly on the phone, before any language model is involved.
Pen-path reconstruction within about 2 mm, close to published research on ring-based sensors (1.6 mm).
A system that learns each person's names and words without ever training a model on their data.
Something that feels like using a notebook, not a lab demo.

What we learned

A language model is most useful to a sensor when the sensor can overrule it. Breaking the problem into motion → path → words worked far better than one large end-to-end model. And with a wearable, the hard problems are mostly human ones: comfort, how it looks, how you charge it, how quickly you can put it on.

What's next for FingerScribe

A custom thin PCB, an enclosed battery and wireless charging. More writers in the training data, and proper testing with left-handed users. Then a lot more testing with real people. We plan to take FingerScribe to market.

Built by Isum Nanomi-Arachchige (information engineering, software, ML) and Sofia Hametsina (mechanical, product design, hardware).