Inspiration

SignWave began in a research lab and grew up to be a game.

While working with the Johns Hopkins University Applied Physics Laboratory, the Uniformed Services University and Walter Reed National Military Medical Center on prosthetic-limb research, we went looking for the hardest unsolved problems in movement recovery. The answer kept coming back to the hand.

Our hands do almost everything for us, and they're surprisingly easy to hurt. U.S. adults made 34.4 million emergency visits for hand problems over four years (Colen et al., 2018). A nationwide study ranked hand and wrist injuries as the most expensive injury type, ahead of hip fractures and brain injuries (de Putter et al., 2012).

The fix is often simple: gentle hand exercises, done daily. But people stick with home exercise only about half the time, and sticking with it strongly predicts recovery (Lyngcoln et al., 2005). Unfortunately, paper printouts are not compelling interfaces for these exercises.

We come from video games, where rhythm games get people to repeat precise movements thousands of times, for fun. So we turned the exercise sheet into a song.

What it does

SignWave turns hand and wrist exercise into a hands-only rhythm game for Meta Quest, built for Meta VR Glasses.

Put it on. Pick a song. Glide along. 3D hands float toward you on the beat, each showing a shape: a grip, a fist, a flat palm, a finger stretch. Match it with your own hand before it arrives. Every move comes from real hand therapy, and together they build flexibility, mobility and strength in any hand.

New for this competition: dynamic calibration. SignWave now reads your hands as you calibrate, learns how you move today, and tunes the game to be challenging for you, based on what it sees rather than through a preset. You land just past comfortable, where you get results fastest.

Your first five minutes: headset on, pick a song with pinch or your eyes, and play while SignWave fits itself to you. No controller, no wall of text.

  • Hands only: your hands play the game; pinch or gaze to navigate the menus.
  • Sit back: a gentle seated mode or an energizing standing mode.
  • Your way: one hand or both, left or right.
  • Any amount of time: sessions from 30 seconds to 3 minutes.

Why people come back: it's fun. Music, movement and a strong virtual version of your own hands add up to real flow. We've had people play so much that we've had to ask them to take breaks!

How we built it

SignWave is built in Unity for Meta Quest and holds 90 fps on Quest 3. Meta's hand tracking is both the controller and the measuring tool: nothing to hold, just gaze or pinch to navigate, seated play and short sessions, exactly what Meta VR Glasses are built for.

SignWave reads every finger joint angle in real time. Each shape is a set of target angles; we score candidates as a weighted sum of joint-angle errors, and the closest match wins:

$$ G = \sum_{j} W_j \left| \, b_j \, \theta_{\text{actual},j} - \theta_{\text{target},j} \, \right|, \qquad G_{\text{matched}} = \arg\min_{G} \, G $$

Challenges we ran into

Designing for the edges. Many of the people we first built for live with PTSD or strong reactions to sudden sounds and visuals. We designed for them first, weighing how every sound, visual and change of pace would land.

Every hand is different. In the clinic, a therapist sets each player's range. At home, SignWave had to learn to do it itself. That challenge became our newest feature.

Accomplishments that we're proud of

  • Clinical roots, consumer-ready. SignWave has completed two research studies with Johns Hopkins APL and Walter Reed, with papers on the way. What participants and therapists asked for, especially personal calibration and shorter sessions, shaped this wellness edition.
  • Adaptive calibration, which brings to anyone's living room a personalized fit.
  • Accessible from the start: no controllers or text walls, one-handed play, either hand, seated modes and the option of eye-tracked menus navigation.
  • A game people choose to play. People recovering from serious injuries have used SignWave to repeat hard, sometimes painful movements for small gains. If it motivates them, it will motivate anyone.

What we learned

  • Sticking with it is a design problem. People come back to what's easy to start and satisfying to finish, the same reason they return to a favorite run or fun workout.
  • The beat is a teacher. Rhythm tells your body when to move, not just what to do.
  • Design for the edges. Every choice we made for players with the most limited movement or the strongest sensitivities made SignWave better for everyone.

    What's next for SignWave

  • Launch on the Meta VR Store, targeted for next quarter.

  • Meta VR Glasses, a natural fit for seated, hands-first play.

  • Progress you can see between sessions, week over week.

  • Publishing our research, and continuing toward a medical version for clinical recovery.

Somos XR's mission is to evoke wonder and conquer despair: an agency delta, the gap between what you can do today and what you could do. We start with people who have the most to gain, but everyone has a story like that, big or small. SignWave helps you reach your goals, with your hands and beyond. somosxr.com

Share this project:

Updates

Submission history