Inspiration
After a stroke, patients are often sent home with a list of rehabilitation exercises. These movements are important, but repeating them alone can quickly become tiring and discouraging. Without a therapist present, it is also difficult to see whether progress is being made.
We realised that the exercise itself was not the only problem. The bigger challenge was making patients want to continue practising.
That inspired us to build Beat By Beat, a spatial rhythm game for Apple Vision Pro that makes upper-limb rehabilitation more engaging, personalised and measurable.
What it does
Beat By Beat places musical targets within each patient’s own measured reach.
During calibration, the patient reaches in six directions with each arm. The app uses these measurements to create separate three-dimensional workspaces for the left and right arms. Gameplay stays within 85% of the calibrated area for additional comfort and safety.
This means the same song can work for someone with 20 cm of reach and someone with 60 cm. Instead of expecting the patient to adapt to the game, the game adapts to the patient.
Patients can practise several movement types:
- Reach by moving their hand towards a target
- Grip by arriving with an open hand before closing it
- Pour by guiding their hand along a curved path
- Hand orientation through cup and overhand grasp positions
These movements are inspired by everyday activities such as lifting a mug, pouring from a jug and picking up an object from a table. Because the objects are virtual, patients can practise without worrying about dropping, spilling or breaking anything.
Targets appear in time with the music and provide several seconds of visual warning. Every completed movement earns points, while misses remain silent. There are no harsh failure sounds, penalties or streaks to break. We wanted the experience to encourage patients to continue moving, even on a difficult day.
How we built it
We developed Beat By Beat for visionOS using Apple’s spatial-computing frameworks.
Hand tracking allows the app to follow the patient’s hands and determine whether they have reached, gripped or followed the required movement path. The calibration system records each arm independently because stroke-related weakness is often asymmetrical.
Songs use beat maps containing actual timestamps from the audio. This allows targets to align with the beats patients hear, even when a song’s tempo changes slightly.
We also built an automatic session-recording system. It captures raw outcomes such as:
- Which targets were presented
- Whether each target was reached
- Which arm was assigned
- Where the target appeared
- How long the patient took to reach it
- When tracking was interrupted
These outcomes are then converted into useful statistics, including success rate, average reach time, movement consistency, directional performance and reachable workspace. A heatmap helps show where within the workspace a patient is succeeding or struggling.
For testing without a headset, we created a simulated hand controller so that the full experience, including calibration, could be tested in the visionOS Simulator.
Challenges we faced
The largest challenge was that this was our first time developing for spatial computing. It was also my first hackathon, so we had to learn a completely new platform while building a working prototype within four days.
Calibration was one of the hardest systems to design. Our first approach placed fixed points around the patient and asked them to touch each one. We realised that this measured whether the patient could reach our targets rather than measuring the patient’s actual ability.
We redesigned it so that the app only asks the patient to move in a direction. Wherever their arm naturally stops becomes the measurement. This creates a workspace based on the patient rather than an arbitrary set of points.
Hand tracking introduced another challenge. If the headset loses sight of the hand near the edge of its tracking range, a short measurement could be mistaken for limited patient movement. To address this, we record tracking limitations separately from physical reach.
Grip detection was also more complicated than simply detecting a closed hand. A patient could approach the target with an already closed fist and incorrectly score. We therefore treated gripping as a movement sequence. The app must first see an open hand at the target before the closing action can complete it.
Another challenge was balancing fun with rehabilitation. Targets needed to follow the music, but they could not appear so quickly that patients were forced to rush. We separated spatial difficulty from speed so that patients are never asked to move farther and faster at the same time.
What we learned
We learned that accessibility cannot be added only after the main experience has been built. It has to shape every interaction from the beginning.
Small design decisions mattered greatly. Recentring the workspace automatically reduced the effort required to begin a session. Silent misses prevented the game from punishing patients. Measuring each arm separately prevented the stronger arm from hiding the limitations of the weaker one.
We also learned how different spatial interfaces are from traditional applications. Position, depth, hand visibility, fatigue and physical comfort all become part of the user experience.
Most importantly, we learned that rehabilitation does not have to feel clinical to remain meaningful. Fun is not separate from the solution. If enjoyment helps someone practise more consistently, it becomes part of the rehabilitation experience itself.
Accomplishments that we are proud of
In four days, we created a working visionOS prototype featuring:
- Hand-tracked spatial gameplay
- Individual reach calibration for both arms
- Personalised three-dimensional workspaces
- Beat-mapped rhythm targets
- Reach, grip and pouring movements
- Multiple mobility levels
- Automatic session recording
- Directional statistics and reach heatmaps
- Simulator support without a headset
We were incredibly proud to receive the Most Fun award at Spatial Hack 2026. A stroke rehabilitation project winning an award for fun may sound unusual, but it represents exactly what we hoped to achieve.
What’s next for Beat By Beat
Beat By Beat is currently a prototype and is not a replacement for professional therapy or a clinical assessment.
Our next step would be to work with occupational therapists, physiotherapists and stroke survivors to validate the movement design and improve its safety and usability.
We would also like to add secure report exporting so that patients can share their session history with healthcare professionals. Future development could include more everyday movement simulations, additional music, therapist-configurable programmes and studies examining whether confidence gained through virtual practice transfers to real household tasks.
Our goal is simple: make rehabilitation practice something patients are more willing to return to, one movement and one beat at a time.
Log in or sign up for Devpost to join the conversation.