Sticky

Inspiration

A sticky note lets us leave instructions right where they matter. At DurHack, we wondered what would happen if the room itself could follow them. We built Sticky so that anyone with a pen could decide how an everyday object should respond and bring that idea to life in their own words. The theme was Composition, and we liked the idea that two people could each write half a program and make it work by putting their notes together.

What it does

Sticky turns handwritten notes into programs using a laptop webcam. Write “When I wave” on one note and “Say hello” on another. Put them side by side, wave, and the laptop greets you. Swap. Peel the notes away and the rule stops.

The person holding the pen decides what to make. A note can turn a bottle into a pickup alarm, start a game of rock-paper-scissors, or put a crown on every face in view. Sticky shows what it understood before the rule runs.

How we built it

Apple Vision reads the handwriting on the Mac, and MediaPipe tracks hands and faces. Our TypeScript compiler combines neighbouring notes into a program with a trigger and an ordered list of actions. Familiar instructions compile locally. For unfamiliar wording, a language model running through Ollama can propose a program, which we check against the actions Sticky can perform. The browser runs the rule each time it sees the trigger.

We used ElevenLabs to generate a response to the sticky note.

Challenges we ran into

Under the venue lights, lime paper looked almost white. A logo behind a tilted note got read as part of the instruction. And our first wave detector watched the wrist, even though people often wave with their wrist almost still. We masked out the background before reading, compared words across several camera frames, and tracked fingertip movement to recognise a natural wave.

Apple Vision initially had trouble reading unclear handwriting, therefore we performed multiple tests to ensure that it could read all types of handwriting and correctly identify what was written on the sticky note.