StickDrift

"No Granny, it's NOT unhealthy to play video games!"

Comment

💭 Inspiration

When we started brainstorming hackathon ideas, we looked to the people in our own lives and the challenges they face. Arthritis and Parkinson’s affect people close to us, and we’ve heard about the joint pain and muscle weakness that can make everyday activities difficult. We wondered: could a simple, interactive experience help someone notice changes in their movement and prompt them to seek advice early, without making the check feel clinical or intimidating?

Here’s what got us: changes in how someone moves can show up years before a diagnosis. In the Rotterdam Study, people later diagnosed with Parkinson’s were more likely to show slowness, tremor, and trouble with everyday tasks as early as seven years before diagnosis [2]. Stroke runs on a much shorter clock: about 1.9 million neurons are lost for every minute it goes untreated [3], and 1 in 4 adults over 25 will have one [4]. Essential tremor affects about 5.8% of people aged 65 and older [5]. Diabetic nerve damage affects about half of people with type 2 diabetes after ten years, is often silent, and is a major contributor to falls [6]. Hand speed, steadiness, and sensation also change with normal ageing [7], so a useful signal may be change from a person’s own baseline.

Care staff are well placed to notice changes, and they have very little time ⏳. Before Ontario’s recent staffing push, long-term-care residents averaged 2 hours 45 minutes of direct care a day, against a provincial target of four [8]. By 2030, 1 in 6 people worldwide will be 60 or older [1].

The way a check feels matters, too. In focus groups, some clinical-trial participants said that a lack of communication and follow-up made them feel like “datapoints” rather than individuals [15]. Research also recognizes that being measured can sometimes change people’s behaviour, which may affect study results [16]. These findings don’t mean clinical assessments are unreliable; they remind us to consider both the person’s experience and the setting of the check.

So we asked: what if the check was a game? 🎮 Older adults found a gamified assessment more motivating, with no loss of measurement quality [11], though a 2025 review warns that a game has to fit its users and that validity alone is not enough [12].

PointClickCare’s “Catch It Early” challenge asked how AI could help a care team spot a health risk before an emergency. Stick Drift is our attempt at that.

🕹️ What it does

Stick Drift turns a PS5 controller into a short, repeatable movement check for seniors in long-term care, and the best part is that residents just think they're playing games.

The resident logs in, walks Grandma around a small 3D lobby, and steps onto a pad to play one of three games:

  • 🍎 Apple Picking: steer a glove with each stick to matching apples and carry them to the basket.
  • 🎣 Harbour Fishing: trace a glowing lane with a lure, one hand at a time.
  • 🐀 Rat Catching: flick a stick up to slam the hand when the target rat runs underneath.

Residents see scores and stars ⭐, never health numbers. Behind the scenes, Stick Drift measures each hand's reaction time, time to reach a target, peak speed, how far the stick is pushed, distance from the path, time on the path, and how much the hand shakes and at what rhythm. Rat Catching correctness and reaction time come straight from the game's own events.

Role-based login 🔐 keeps the two sides apart: residents get the game lobby, and care staff get the dashboard.

The care team gets:

  • 📡 A live view, from any computer, that follows the resident's hands about 0.1 s behind.
  • 🚦 A dashboard badge for each resident: No concerns, Keep an eye on, or Check soon.
  • 📈 Plain-language visit cards ("Picked 20 of 23 apples", "0.30 s quicker than last visit") and graphs of trends across visits.
  • 🧩 Pattern scores from 0 to 100 for resemblance to three movement patterns: Parkinson's, Stroke, Essential tremor. A score of 50 raises "Keep an eye on" and 70 raises "Check soon", and each concern lists everyday explanations (tiredness, medication, vision, setup) and what to check first.
  • 🤖 An AI assistant (Gemini) that answers questions using only the resident's recorded numbers.
  • 📄 A one-click clinician report (PDF) with charts, a visit table and a review sign-off.

⚠️ Stick Drift does not diagnose anything. It flags a change worth checking, and a nurse decides what happens next.

🛠️ How we built it

  • 🎮 Games: a three.js lobby and three browser games that read the controller through the Gamepad API. The lobby and its levels were modelled in Blender. A separate measurement layer watches each game's state without changing it, so we can tune a game without breaking its measurements. Our first training data came from an earlier pygame garden game that recorded the controller at about 500 Hz.
  • 🌐 Web app: React 19, TypeScript and Vite, with role-based auth and a dashboard that graphs each resident's visits.
  • 🗄️ Backend: FastAPI and psycopg 3, with Tiger Data (PostgreSQL + TimescaleDB) as the only thing the resident's laptop and the caregiver's laptop share.
  • ⚡ Live streaming: the lobby posts samples about ten times a second. The API stores them and sends a Postgres NOTIFY in the same transaction; each caregiver's API LISTENs and pushes updates over a WebSocket through a 150 ms jitter buffer. If notifications stop, the API reads the database every 0.15 s, and if the WebSocket fails, the browser polls.
  • 🧠 Pattern model: a PyTorch temporal convolutional network built from dilated 1-D convolutions [14]. It cuts each hand's movement into stretches and scores the three patterns. We trained it on synthetic recordings and on teammates acting out each pattern.
  • 🛡️ AI guardrails: Gemini only sees a facts block built from the database. Any number in its answer that isn't in the facts gets flagged, condition names outside the model's three patterns are blocked, and a rule-based answer takes over if Gemini fails. ReportLab draws the PDF.
  • 👥 Team:
    • Ayan built the dashboard functionality, real-time game capture and role-based auth system.
    • Hreem built the game lobby and levels in Blender, improved the UI and ported in TigerData
    • Bryan trained the TCN models and refine the data for richer modelling
    • Zain trained the models on gameplay and got us a bunch of sample data/researched to back idea

🧗 Challenges we ran into

😴 Four people and very little sleep. Beyond that, everyone hit their own wall:

Ayan

Real-time capture was the boss fight. Our two laptops share only a database, Postgres notifications have a size limit, and teammates were often running older API code that sent none, so I built fallbacks (database reads, then browser polling) to keep the live view alive. Walking around the lobby isn't a health signal, so I save only in-game movement, and I save each game the moment it ends so a resident who plays one game and leaves still has a record.

Hreem

Learning to properly use Blender within a short amount of time. Have worked with Blender once before but not at this pace and scale. Turning Blender models into a lobby that runs smoothly in a browser, then keeping the UI clear enough for older eyes and unsteady hands was a challenge. Every level had to feel like a game while staying readable for seniors. Additionally, was my first time working with TigerData so learning how it differed from other databases was a challenge

Bryan

We had no real patient data, so the model learned from synthetic and acted recordings, and we present its output only as resemblance. I also went back and forth on the pattern labels, trying to make them clear enough for a caregiver to tell apart without reading as a diagnosis. And with four people pushing at once, a model checkpoint and its code ended up on different schema versions.

Zain

Recording training data meant playing the games as each movement pattern, convincingly, hundreds of times. The catch is that a model trained on one person's gameplay can end up learning how that person plays, which is why we kept retraining on more sessions. As well as deep diving into why certain movements may indicate certain impairments certainly was a challenge to find resources.

As a team

  • The sticks steer an in-game hand by velocity, so distance from target has to compare the on-screen hand with its target, while speed and shake come from the stick itself.
  • We ran out of Gemini's free-tier quota during testing. The rule-based fallback kept the assistant and the report working. 🙌
  • Merge conflicts, mismatched versions and a lot of "wait, which branch are you on?"

🏆 Accomplishments that we're proud of

Ayan

A live view that follows a resident's hands about 0.1 s behind, a dashboard with graphs of every visit, and a role-based login that keeps residents and care staff in their own worlds.

Hreem

A lobby and levels built in Blender that feel like a real game, plus a UI polished enough that the whole project looks like a product + getting TigerData in there

Bryan

A temporal convolutional network that scores a hand's movement against three patterns, with scores that stay in "resemblance" territory and never claim a diagnosis.

Zain

Gave the model its first realistic data by training on his own gameplay, which got us from an idea to a working pattern score quickly.

As a team

  • A complete loop in 36 hours: a resident plays, a caregiver watches live on another laptop, and the visit becomes saved history with trends, pattern scores, an assistant and a clinician PDF.
  • A detailed per-hand movement profile from an off-the-shelf game controller.
  • An assistant that only uses recorded numbers, flags anything it can't back up, and has a fallback when the AI is down.
  • A caregiver site in plain language, with light and dark themes, a glossary of measurements, and a layout that works on a phone.

💡 What we learned

  • We had to word every label, threshold and sentence so that a measurement never reads as a diagnosis.
  • Reaction time changes with age and varies a lot between people [13], so one reading tells you little. Change from a resident's own history is what matters.
  • Postgres LISTEN/NOTIFY, with fallbacks, was enough to carry our whole live view.
  • Synthetic data got our model started, but only real, labelled recordings can show whether it works.
  • Anything that depends on a network, a model or an outside API needs a plan for when it fails.

🔮 What's next for StickDrift

  • A consented pilot with residents and staff in a long-term care home, to test whether it stays enjoyable and whether the measurements hold steady week to week.
  • Clinical validation of the pattern model on properly labelled recordings, evaluated person by person.
  • Personal baselines for lobby visits, so concerns come from change over several visits.
  • A care workflow where staff mark each prompt as acknowledged, needs assessment, or dismissed with a reason, and the system learns from those outcomes.
  • Integration with care platforms such as PointClickCare after a privacy review. We have not connected to any EHR during the hackathon.
  • Controller stands and placements for residents with reduced dexterity.

Want to see a controller spot a change before anyone else does? 🎮🧠
Stick Drift is just getting started.

Play it here: https://stickdrift.health/

Sources

  1. World Health Organization. Ageing and health fact sheet. https://www.who.int/news-room/fact-sheets/detail/ageing-and-health
  2. Darweesh SK, Verlinden VJ, Stricker BH, Hofman A, Koudstaal PJ, Ikram MA. Trajectories of prediagnostic functioning in Parkinson's disease. Brain. 2017;140(Pt 2):429-441. https://doi.org/10.1093/brain/aww291
  3. Saver JL. Time is brain—quantified. Stroke. 2006;37(1):263-266. https://pubmed.ncbi.nlm.nih.gov/16339467/
  4. World Stroke Organization. Global Stroke Fact Sheet. https://www.world-stroke.org/world-stroke-day-campaign/about-stroke
  5. Louis ED, McCreary M. How common is the most common adult movement disorder? Update on the worldwide prevalence of essential tremor. Tremor Other Hyperkinet Mov. 2021;11:28. https://pmc.ncbi.nlm.nih.gov/articles/PMC8269764/
  6. Pop-Busui R, et al. Diabetic neuropathy: a position statement by the American Diabetes Association. Diabetes Care. 2017;40(1):136-154. https://pmc.ncbi.nlm.nih.gov/articles/PMC6977405/
  7. Ranganathan VK, Siemionow V, Sahgal V, Yue GH. Effects of aging on hand function. J Am Geriatr Soc. 2001;49(11):1478-1484.
  8. Government of Ontario, Ministry of Long-Term Care. Published plans and annual reports 2022-2023 (Fixing Long-Term Care Act, 2021 direct-care targets). https://www.ontario.ca/page/published-plans-and-annual-reports-2022-2023-ministry-long-term-care
  9. Giancardo L, et al. Computer keyboard interaction as an indicator of early Parkinson's disease. Sci Rep. 2016;6:34468. https://doi.org/10.1038/srep34468
  10. Bot BM, et al. The mPower study, Parkinson disease mobile data collected using ResearchKit. Sci Data. 2016;3:160011. https://doi.org/10.1038/sdata.2016.11
  11. Zhao, et al. Balance assessment using gamified digital technology in community-dwelling older adults: mixed methods validation study and randomized controlled trial. JMIR Serious Games. 2026;14:e80747. https://doi.org/10.2196/80747
  12. Mantell, et al. Evaluating the user experience and usability of game-based cognitive assessments for older people: systematic review. JMIR Aging. 2025;8:e65252. https://doi.org/10.2196/65252
  13. Der G, Deary IJ. Age and sex differences in reaction time in adulthood: results from the United Kingdom Health and Lifestyle Survey. Psychol Aging. 2006;21(1):62-73.
  14. Bai S, Kolter JZ, Koltun V. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:1803.01271. 2018. https://arxiv.org/abs/1803.01271
  15. Boyd P, Sternke EA, Tite DJ, Morgan K. “There was No Opportunity to Express Good or Bad”: Perspectives From Patient Focus Groups on Patient Experience in Clinical Trials. Journal of Patient Experience. 2024;11. https://doi.org/10.1177/23743735241237684
  16. French DP, et al. Reducing bias in trials from reactions to measurement: the MERIT study including developmental work and expert workshop. Health Technology Assessment. 2021;25(55). https://doi.org/10.3310/hta25550

Built With

Share this project:

Updates

Submission history