Footnotes

Inspiration

It started with a film. In The Music Never Stopped (2011), based on a case study by neurologist Oliver Sacks, a man with a brain injury who can't form new memories comes back to life through the songs he loved. Music reaches parts of the brain that other things can't. Sacks wrote about the same thing in people with Parkinson's: patients who could barely start walking could suddenly move, even dance, once music played.

That isn't just a movie moment. It's a well-studied technique called Rhythmic Auditory Stimulation (RAS): walking to a steady beat helps people with Parkinson's find their rhythm. A 2022 meta-analysis of 18 randomised controlled trials (774 people with Parkinson's) found RAS improved stride length and walking speed and reduced freezing of gait (Ye et al., Frontiers in Neurology, 2022).

But in practice the cue is usually a metronome: a dull click that nobody wants to listen to for long. And the "proper" kit, dedicated cueing devices and clinical gait sensors, is expensive and not something most people own.

So we asked: what if the beat were music that rewards you for walking well, and it ran on things people already have, for free? That's Footnotes: notes made by your feet.

What it does

A phone in your pocket feels every step.
20 seconds of normal walking measures your natural pace. The beat is set just above it, the way RAS works.
The beat always leads and never chases you. What changes is the music on top: walk in time and the shakers, bass, piano, melody and sparkle join in one by one; drift off and it strips back to a clear beat that's easy to find again.
A live visual score on the laptop draws the walk and the music together: every step is a dot placed by its timing (green = on the beat), and each instrument appears as its own lane when it starts playing.
A patient app designed for Parkinson's: huge, tremor-proof buttons, read-aloud on every screen, press-and-hold anywhere to stop, and each walk drawn as a little piece of music afterwards.

It costs nothing. No special hardware and no app to install: it runs in the web browser on a phone and a laptop you already own.

How we built it

Stack: TypeScript everywhere; React + Vite for the phone and laptop screens; Tone.js (Web Audio) for the music; a small Node.js WebSocket relay to pair a phone with a laptop by room code; a Cloudflare tunnel for the HTTPS that phone motion sensors require.

Step detection. The phone reads its accelerometer 50–100 times a second. We remove gravity, smooth the signal and find peaks with an adaptive threshold, then correct for the filter's delay so each step is timed to the millisecond.

Clock sync. The phone and laptop have different clocks. The laptop keeps pinging the phone and estimates the offset from the fastest round trips:

$$ \theta = t_1 - \frac{t_0 + t_2}{2} $$

Measured error: about 1 ms over the internet.

The beat. After the 20-second calibration:

$$ \text{tempo} = \text{natural pace} \times (1 + \text{offset}), \qquad \text{offset} = 5\%\ \text{by default} $$

It's scheduled on the audio hardware's own clock, so the gaps between beats vary by less than a millisecond.

How "in step" is scored: circular statistics. Each step gets a phase, where it lands relative to the nearest beat:

$$ \phi_k = \frac{t_k - t_{\text{beat}}}{T}, \qquad \phi_k \in [-0.5, 0.5) $$

Phase wraps around like a clock face (just before a beat and just after it are nearly the same moment), so an ordinary average fails. Instead we treat each step as a unit arrow and average them:

$$ R = \left| \frac{1}{N} \sum_{k=1}^{N} e^{2\pi i \phi_k} \right| $$

$R \approx 1$ means every step lands at the same point relative to the beat (locked in); $R \approx 0$ means no relationship. The score combines $R$ with how close the walker's pace is to the beat, and is smoothed so the music never flickers.

Adaptive arrangement. All six instruments play all the time, silently, and are faded in or out only at the start of a bar, so changes always sound musical. Each instrument comes in at a higher score than the one at which it leaves, so nothing flickers on and off. The music builds up one instrument per bar and strips back faster when someone struggles. Everything is synthesised in code (an original piece in D major), with no recordings and no licences.

Challenges we ran into

The beat that felt twice too fast. Slow walking, about one step a second (common in Parkinson's), looked identical to fast one-legged strides, and the app doubled the pace. We stopped guessing and counted every bump as a step.
The missing foot. A phone in one pocket feels the far leg only faintly, so every other step vanished. Now, when two steps are twice the usual gap apart, the detector looks back for the faint bump in the middle. On simulated walks, steps caught went from about 55% to 99%.
Phones going quiet. Screens switching off in pockets, old browser tabs taking over the connection, Wi-Fi blinks. We added keepalives, automatic reconnects, saving steps during dropouts, and a connection log to find out why drops happened.
Too hard for first-timers. Scoring was strict, and the two legs land at slightly different moments for a pocket phone. An Easy timing mode judges each leg separately and measures pace stride to stride: in testing, full music after about 17 seconds instead of 3 of 5 instruments after 20.
Freeze detection, done honestly. We built a freezing-of-gait detector using the published freeze index (Moore et al., 2008):

$$ \text{FI} = \frac{\text{power}{3\text{–}8\,\text{Hz}}\ (\text{trembling})}{\text{power}{0.5\text{–}3\,\text{Hz}}\ (\text{walking})} $$

We tested it on the Daphnet dataset: real recordings of 10 people with Parkinson's and 237 freezes marked by physiotherapists (Bächlin et al., 2010), with each patient's thresholds chosen only from the other nine. At the ankle it caught 86% of freeze time and left 77% of normal walking alone, comparable with the original paper's 73.1% and 81.6%. But every sensor position gave over 100 false alarms an hour, because turning around looks like trembling. From a pocket phone that isn't good enough, so we left it out of the app rather than ship something misleading.

Accomplishments that we're proud of

A beat that's rock steady and music that sounds like music, not a metronome with beeps.
Phone-to-laptop timing accurate to about a millisecond, over the internet.
A freeze detector evaluated fairly on real patient data, and the honesty to cut it.
An interface designed around tremor and low vision from the first screen.
All of it free, on hardware people already own.

What we learned

Real bodies are messy. Our simulated walker was easy; real pockets weren't.
Musical rules (bar lines, fades, not flickering) matter as much as the maths for adaptive music to feel calm.
Evaluating fairly changes the story, and makes it more believable.