Paths To Recovery

Title Screen
Fight Screen
Gesture Recognition

👋 Paths To Recovery!

For ShellHacks 2025, we decided to tackle something personal. Several of us have watched family members struggle through physical therapy, and we knew there had to be a better way to make recovery less of a grind. Anyone who's been through rehab knows how tedious those repetitive exercises can get. Week after week of the same movements with little to show for it.

That's where our game comes in. We built six completely different story paths, each with its own ending, custom cutscenes, and soundtrack. The idea is simple: instead of just doing exercises, you're progressing through an adventure where your real movements control the story. Finish one path? There are five more waiting, each offering a fresh experience that keeps you coming back to do the work your body needs.

🧙‍♂️ What it does?

Picture this: you're a knight standing guard at your castle gates when a mysterious figure emerges from the fog. Do you try to reason with them, or do you draw your sword? The cool part is you don't need a controller or keyboard. Just lift your hand in front of your webcam and watch the cursor follow your movements on screen.

Your computer's camera becomes the bridge between your physical therapy exercises and the game world. Every gesture you make drives the story forward, turning those repetitive movements into meaningful choices that shape your adventure. The path you choose is entirely yours to decide. Will you be able to unlock all the endings?

🤔 How we built it?

We developed a Unity game that interfaces with an OpenCV server over TCP to classify hand gestures, aimed at rehabilitating physical therapy patients by gamifying their exercises. We developed a custom gesture recognition system with Python and MediaPipe. We also created original game assets, sprites, and built the entire Unity game architecture using C#.

♟️ Challenges we ran into

Unity Collaboration and Version Control: Managing Unity project collaboration proved challenging due to binary file conflicts in Git. Unity's asset files and scene data don't merge well in version control systems, requiring us to implement careful coordination strategies and file locking protocols to prevent overwriting each other's work.
Gesture Recognition in 3D Space: Developing reliable gesture classification required overcoming the inherent noise and jitter in 3D hand tracking data. Small variations in MediaPipe's landmark detection could cause significant angle fluctuations, particularly in neutral hand positions, which caused us to implement filtering algorithms and deadband zones to achieve stable gesture recognition.
Styling consistency: Achieving uniform pixel art aesthetics across all game assets required strict adherence to consistent pixel densities and color palettes. Mixing different pixelation levels or art styles resulted in visually jarring inconsistencies that broke the game's cohesive visual identity.
Integrating 2D sprites with 3D lighting systems: Implementing proper lighting effects on 2D sprites within Unity's 3D environment presented unique technical challenges, as 2D objects don't naturally interact with 3D lighting systems. This required custom shader work and creative workarounds to achieve the desired visual effects while maintaining performance.

🏆 Accomplishments that we're proud of

Developed an optimal gesture classification system that overcomes 3D tracking noise.
Successfully bridged Python-based OpenCV/MediaPipe gesture recognition and Unity game engine by implementing TCP communication system.
Created a cohesive visual experience through consistent pixel art styling and accompanying audio design that improved the therapeutic gaming environment without sacrificing visual accessibility.
Created an accessible control system that can be used by players with different physical therapy requirements, guaranteeing that the game is still fun and useful despite the need for rehabilitation or a lack of technical expertise.
Fostered collaboration and teamwork under tight deadlines, leading to a cohesive final product. ## 🧠 What we learned
Utilizing generative AI can exponentially speed up development with quick turn arounds on assets and sprites.
Several team members picked up Unity from scratch during the hackathon and managed to make real contributions to the game development process.
Used Universal Render Pipeline to create 3D shadows on 2D sprites
Learned how to use MediaPipe for gesture classification and utilize vector math for positional analysis.

🤞What's next for Paths To Recovery

We have so many more ideas for the future of Paths To Recovery! Our project mostly focuses on hand gestures, but we can easily expand to other common movements needed to aid recovery. Our story can also be expanded with more characters and memorable story elements. We are so proud of what we've been able accomplish with Paths To Recovery and are looking forward to our next set of challenges.