Motivation
Children with cerebral palsy (CP) get better hand function through repetition: hundreds of the same movement, practiced at home, week after week. But home programs are hard to keep up. Studies report that many families struggle to stick with them, and the exercises feel like homework. We wanted to turn therapist-recommended movements into games children want to repeat, using affordable hardware that processes camera footage locally without uploading or storing video.
Unlike conventional motion-controlled games, WonderMoves adapts gameplay to each child's comfortable movement range rather than asking the child to meet a fixed difficulty level. By combining affordable sensors, on-device computer vision, and progress tracking, we make repetitive movement practice more engaging without requiring expensive rehabilitation equipment.
Features
WonderMoves is a set of three short games for children aged 6–12, each built around one single movement:
- Fossil Brush – brush in preset patterns to uncover a dinosaur fossil. Pip the mascot shows the movement first, then it's the child's turn. It runs on the camera, and tracks the child's index finger to perform motions.
- Treasure Reach – reach forward and come back. A Modulino Distance sensor on the table measures each reach, and every completed reach opens a chest to reveal a treasure.
- Bubble Pop – bubbles rise up from the ocean floor for a set period of time, and the child can pop them by moving their hands across the screen. Motion tracking is done using the Brio 105 webcam.
Parent Dashboard: Caregivers can track their child's practice over time through a dedicated dashboard showing completed movements, play duration, and activity trends. This helps families monitor consistency and engagement between therapy sessions.
AI Voiceover: Pip guides children through each game with friendly AI-generated voice narration, celebrating completed actions and offering gentle encouragement. This makes the experience more engaging and accessible for younger children who may struggle with written instructions.
Every design choice follows the research:
- Completion, not speed. A slow reach counts exactly like a fast one.
- No fail states. There are no lives and no penalty for mistakes.
- Never beyond the approved range. A grown-up calibrates the child's comfortable reach and sets a maximum that the game never exceeds.
- Gentle breaks. After 10 minutes of play (adjustable), Pip suggests a stretch and a drink.
Stack
- Hardware: Arduino UNO Q (Qualcomm QRB2210 + STM32U585), Modulino Distance sensor, and a Logitech Brio camera.
- On-device vision: pose tracking with OpenCV DNN and a MediaPipe pose ONNX model running on the Qualcomm CPU (~119 ms per frame). Video is processed on the board and never saved or sent.
- Games: HTML5 canvas games in one shell, shared by iframes, with a board bridge so the same game works with sensors, camera, touch or mouse.
- Data & Analytics: session data is processed through Databricks to support the parent dashboard and visualize practice trends over time.
Challenges
- A sensor that only saw itself. The distance sensor reported a constant ~27 mm. It turned out to be a protective film and crosstalk from its own cover window. We peeled the film, wrote a filter for those readings, and added a caregiver hint for it.
- The CPU couldn't run MediaPipe's native library (ARMv8.0, no LSE atomics), so we switched to OpenCV DNN with an ONNX pose model.
- Merging four people's work across games, a shell UI, the board and a data dashboard, while keeping every game inside the clinical rules.
Accomplishments
- Ran distance sensing and camera-based pose tracking together on a $59 Arduino UNO Q, with local video processing that requires no cloud connection.
- Integrated camera-based tracking and distance sensor measurements to support more consistent reach detection.
- Integrated AI-generated voice guidance through Pip to make gameplay more accessible and engaging for younger children.
- Every game rule traced back to published evidence, written up for our mentors.
Takeaways
Designing for children with CP is mostly about what you leave out: speed rewards, fail screens, penalties. And embedded AI on real hardware is mostly about debugging the physical world: films, cables, clocks and lighting.
Next Steps
Additional games for lower body control. A pilot with an occupational therapist: a small group of children with unilateral CP using one game at home for several weeks, measured with the Assisting Hand Assessment (AHA) and the Canadian Occupational Performance Measure (COPM), with research-ethics approval first.
References
Vezér, M., Gresits, O., Engh, M. A., Szabó, B., Molnár, Z., Hegyi, P., & Terebessy, T. (2024). Effectiveness of video-game-based therapy to improve hand function in children with cerebral palsy: A systematic review and meta-analysis. Journal of Clinical Medicine, 13(24), 7524. https://doi.org/10.3390/jcm13247524
Abd-Elfattah, H. M., Galal, D. O. S. M., Abdelmageed, S. M., Aly, S. M., Ameen, F. H., Sayed, A. O., & Abd-Elmonem, A. M. (2024). Effect of touch screen tablet use on fine motor functions in children with hemiparetic cerebral palsy: A randomized controlled trial. NeuroRehabilitation, 55(1), 137–146. https://doi.org/10.3233/NRE-240134
Sah, A. K., Balaji, G. K., & Agrahara, S. (2019). Effects of task-oriented activities based on neurodevelopmental therapy principles on trunk control, balance, and gross motor function in children with spastic diplegic cerebral palsy: A single-blinded randomized clinical trial. Journal of Pediatric Neurosciences, 14(3), 120–126. https://doi.org/10.4103/jpn.JPN_35_19
Malick, W. H., Butt, R., Awan, W. A., Ashfaq, M., & Mahmood, Q. (2022). Effects of augmented reality interventions on the function of upper extremity and balance in children with spastic hemiplegic cerebral palsy: A randomized clinical trial. Frontiers in Neurology, 13, 895055. https://doi.org/10.3389/fneur.2022.895055
Jackman, M., Lannin, N., Galea, C., Sakzewski, L., Miller, L., & Novak, I. (2020). What is the threshold dose of upper limb training for children with cerebral palsy to improve function? A systematic review. Australian Occupational Therapy Journal, 67(3), 269–280. https://doi.org/10.1111/1440-1630.12666
