Sillow - Sensing Pillow

When your hands do the talking.

Comment

Inspiration

My grandfather had a stroke. He understood everything that was said to him. He remembered everything. He could move his hands. And he could not tell us any of it.

The tool they gave us was a laminated alphabet board. Someone holds it up, the patient looks, the helper guesses letters. It takes minutes to get out one sentence, and it costs the patient the one thing they still have: their attention.

Every assistive communication device we looked at makes the same demand. Look at this screen. Navigate this menu. Find your word. For a patient who is bed-bound, exhausted, and already overwhelmed by a room full of machines, that demand is the barrier.

Sillow removes it. The patient does not look at anything.

What it does

A pillow with touch interface on the patient's bed that can be used as an arm rest.

They draw with a finger, by feel. A soft tone marks the start of each stroke, a brighter one the end, and a short vibration confirms contact, so the patient knows they were registered without needing to look for ink. After five seconds of stillness the surface captures itself. Nothing is ever pressed.

The system ranks the drawing against everything it knows about this patient and speaks a full sentence aloud: "I would like a glass of water, please." Then it asks whether that was right. One tap on the empty surface means yes. Two taps mean no. There is no button, because the entire surface is the answer. A no retries the next guess.

A handwritten H, plus, or cross means help. It says help is on the way, and the caregiver's phone receives a full-screen emergency alert with alarm and vibration that outranks every ordinary notification.

The caregiver app shows each drawing with its image, the spoken line, and the answer, alongside a memory graph linking people, intents, drawings and days. For the first time, there is a record of what a non-speaking patient has actually been saying.

Who it serves: patients whose language is intact but whose voice is not, and who retain hand movement. Post-surgical and intubated patients, which is by far the largest group in any hospital. Post-laryngectomy. ALS. Severe dysarthria, including motor and brainstem stroke.

How we built it

Tremor smoothing, tuned against the textbook

The people this is built for have unsteady hands, so pointer input runs through a One Euro filter: a low-pass whose cutoff rises with pointer speed, smoothing slow shaky movement hard while leaving deliberate fast strokes responsive. Against a simulated 8 Hz tremor, our medium preset removes about 80% of the wobble and strong about 91%.

The interesting part is the tuning. We use a much smaller beta than standard One Euro examples, and we do it on purpose. The usual values assume fast movement is intentional and should stay sharp. But tremor is itself fast. A large beta raises the cutoff exactly when the shake is worst and waves it straight through.

The raw samples are never discarded. Every stroke carries its unfiltered points alongside the filtered ones and names the preset used. For a stroke patient the tremor may itself be the signal of interest, so we never throw it away, and any capture can be re-filtered later with different settings.

Recognition that degrades instead of failing

Five tiers, each one a fallback for the one above:

  1. Geometry, no API call at all. A clear handwritten 1 or 2 is graded on stroke shape alone.
  2. Gemini, flash-lite then flash, on a shared 15-second live budget.
  3. Grok, on a fast Gemini refusal.
  4. Local gemma3:12b via Ollama, entirely on-machine, when every cloud vendor fails.
  5. Feature templates built from this patient's own labelled repeats, validated hold-one-out.

Every candidate records which tier produced it. Ranking blends model output with a prior built from this patient's own history, so the system gets better at this specific person rather than better in general.

Every cloud vendor can be down and the patient still speaks. That is not a nice-to-have in a hospital room. Each tier is tested independently against stubs, so verifying the whole chain costs zero API credits.

Engineering we would defend in a code review

Speech synthesis runs on the laptop, not the tablet, so the API key never leaves the machine. Without a key nothing breaks: the tablet falls back to its own voice engine.

Stroke data is stored, not just images. Every point is [x, y, pressure, ms] in CSS-pixel space, so a drawing can be re-rendered at any resolution or trained on as motion rather than as a picture. Points stay grouped by stroke as well as flat, because a cross and a cup become indistinguishable the moment you lose the boundaries.

Recognition runs on a worker thread after the upload is acknowledged, so a slow API call can never stall the patient's surface or time out their upload.

Tap-answering only listens while a question is pending. Stippling a field of dots still draws dots, because nothing is asking anything. Three or more rapid taps are rejected rather than guessed at, and the question stays up.

Nothing leaves the local network. Both devices share one Wi-Fi and the server is never exposed to the internet. Patient data does not travel.

Challenges we ran into

We narrowed who this is for, and it made the product better. Stroke was our starting point. But stroke patients who lose speech most often have aphasia, and our spoken confirmation assumes intact comprehension. Rather than paper over that, we redefined the target by functional profile instead of diagnosis: language intact, voice absent, hand movement present. That reframing is what let every other design decision fall into place.

The fabric cover failed, and the failure taught us something. We wanted cloth over the touch surface. Capacitive touch reads through very thin fabric, but loose cloth drags under a moving finger, so the contact point barely travels and the stroke never registers. Dimming the display to black achieves the identical eyes-free property with none of the material risk. The goal was never fabric. The goal was that the patient does not look.

iPadOS does not implement navigator.vibrate at all. We targeted Android instead of pretending the feature works everywhere.

Smoothing costs lag, visible in our side-by-side preview as smoothed strokes ending slightly short of the raw ones. Strong is right for a pronounced tremor and sluggish otherwise, which is why the level is configurable rather than baked in.

Accomplishments that we're proud of

The confirmation tap does two jobs at once. It is the safety mechanism, because nothing is spoken on the patient's behalf until they approve it. And it is the training signal, because every approval is a labelled example. The thing that stops the system putting words in a patient's mouth is the same thing that teaches it what they meant. We did not plan that symmetry. We found it and then built around it.

There is no single point of failure between the patient and being heard. Cloud down, local model missing, templates only: it still answers.

The tremor filter is real signal processing with a measured result. We can state our parameters, explain why they contradict the standard tuning, and show the number.

The emergency path resists its own cleverness. Help is help. It never asks what kind. It never dials from the patient's side. When someone is in distress, the system does the simplest reliable thing.

We built something that produces a clinical artifact as a side effect. Nobody currently knows what a non-speaking patient was trying to say across a week. Our caregiver view does, and that record did not cost us a single extra feature.

What we learned

Feedback matters more than accuracy when the user cannot see. Without the stroke tone and the vibration, drawing blind feels broken even while recognition is working perfectly. We spent as much care on the confirmation tone as on the model chain, and it was the right allocation.

Constraining the output space beats reaching for a bigger model. Ranking against a fixed set of intents is a far easier task than open-ended recognition, and small models do it dramatically better. The most valuable thing an LLM did here was help build the prior, not run the inference.

The honest limitation is the most interesting part of the design. Working out precisely who this does not serve is what let us build something coherent for the people it does.

What's next for Sillow - Sensing Pillow

A real sensing surface. This prototype is a capacitive touchscreen with its display dimmed, which proves the interaction. The production version is a fabric pressure array inside an actual pillow: softer, washable, and the correct material for a hand that rests on it for hours.

Room integration. The system already emits structured intent. Lights, temperature, blinds, nurse call: anything that accepts a command could listen.

Higher Interaction. Optional gesture recognition for patients with higher motor capabilities in arms.

Soft pressure-sensing bed surfaces already exist, mostly for repositioning and bed-exit alerts. Screen-based AAC devices already exist. What does not exist, as far as we can find, is the combination: an eyes-free surface used for communication, where the system learns an individual patient's own drawings and fuses them with a prior built from their history.

Not a medical device. Makes no clinical decisions. Nothing is spoken on the patient's behalf until they confirm it. The person shown in the demonstration video is a team member, not a patient.

Built With

  • aac
  • accessibility
  • assistive-technology
  • gemini
  • gesture-recognition
  • healthcare
  • offline-first
  • patient-communication
  • signal-processing
Share this project:

Updates

Submission history