RECOVR

RECOVR is a tool and experience that allows amputees to reclaim control of a virtual arm. Paired with immersive storytelling we allow them to rediscover the joy, wonder, and magic of touch.

Comment 1

Inspiration

The idea for this project came when a friend of ours got into a motorcycle accident and lost all control of his left arm. What really shocked us was not only the physical pain he had to endure but also the emotional. The pain of losing control, dignity, and connection. And he isn't the only one who suffers from those pains. Today, two million Americans are living with a lost limb. So, we began to ask ourselves how we could create a tool and experience that could tackle this emotional burden.

Objectives (what it is)

Our goal is threefold. The primary objective is to give disabled users the ability to control a virtual limb. Paired with immersive storytelling, we allow them to rediscover the magic, joy, and wonder of touch so as to ease the emotional pain due to a lost limb. Furthermore, in this immersive environment, we can transform the long and painful rehabilitation procedure into a joyful adventure. Lastly, for a broader impact, we collect muscle sensor data, preparing for the customization of future realistic prosthetic limbs. RECOVR is designed to accompany users during their recovery journey of a prosthetic limb with immersive joyful experiences.

Our users

We design RECOVR for users starting or currently under prosthetic limb rehabilitation procedures.

How we built it

Utilizing the strengths of our team members, we built a prototype to prove the concept using EMG sensors to track users' muscle movement, which controls a virtual arm to interact with different objects in an immersive environment created using Unity Game Engine.

  • We separated the two EMG musical signal patterns (fist and open) using signal filters
  • We handcrafted the assets and animation using Blender
  • We designed immersive interaction (flower blooming and blowing leaves) using Unity

Challenges we ran into

  • We attempted to create better skin rendering using Galaxy shader; however, the results we obtained look flat (rather than 3D). We did not have enough time to figure the solution out. At last, we used standard materials for the skin.
  • We also encountered communication issues using the ESP32. We were able to resolve it by implementing a one-way communication using System.IO instead.
  • The FFT filter worked well to separate the muscle EMG signal patterns (fist and open); however, since we only had one sensor, it was challenging to identify the right spot to put the sensor. On the last day, we encountered hardware failure, and we did not have a replacement sensor. To provide a similar experience to users, we had to implement a workaround to classify users' first and open actions -- we opted for a simple manual mouse click.

Accomplishments that we're proud of

  • We executed a project and made our idea come true in 2.5 days.
  • None of us have a business background, but we managed to enter the later round(s) of the pitch competition.
  • We met great people, and we are proud to be part of the Reality Hack.

What we learned

  • With passion, we can achieve a lot even in a very short time period.
  • With mutual understanding, we can maximize our team's productivity. (Thanks to the team analysis worksheet.)
  • Last but not least, with good food, we are confident that we can do even better!

Our team

What's next for RECOVR

  • Manufacture a better mount for multiple muscle sensors, which is adjustable and can be easily mounted onto the user's arms.
  • Instead of separating two simple gestures (fist and open), multiple sensors will allow us to separate finer gestures such as finger movements.
  • Design more immersive interactions and storylines in the virtual environment, which allows us to collect meaningful muscle signal data for future prosthetic arm customization.
  • Support multiplayer mode, such that users can interact with others to regain their lost touch in the virtual world.

Acknowledgement

We would like to thank all the mentors for their hard work and support. This project would not have been possible without you.

We'd also like to express our special thanks to our coach Rafiq Sabur, our friend Matt, and our medical expert consultants Dr. Greg Osgood (MD) and Prof. Dr. Nassir Navab (Ph.D.) for all the inspiration, advice, and support for the startup track pitch competition.

Built With

Share this project:

Updates