Inspiration

Strokes are currently the second leading cause of death worldwide, causing 44% of deaths worldwide, and of the very few who survive, the path to recovery is brutal. This project aims to help those who have had their fine motor skills impacted by a stroke, or other related conditions. By presenting these therapeutic exercises as a way to control a game, we can engage the patient, and ensure a path towards their recovery.

What it does

Guiding Hands is an interactive top down game projected on a board, that uses specific fine motor exercises as a way to engage post-stroke individuals with their rehabilitation of their fine motor skills and improve consistency in their physical therapy.

How we built it

Hardware Acceleration: We utilized a Jetson Nano provided by Nvidia, using the Google Media pipeline model with Linux native hardware acceleration to improve model efficiency. The Nano also runs the game on a locally hosted web app to make the project more accessible.

Gesture Tracking: In conjunction with the Jetson Nano, we were able to use cameras in the project to track the gestures made by the patient's hands to ensure they are properly doing the exercises intended, as well as to let these gestures control the game.

Game Projection: With a projector mounted on top of the main board, we were able to project the game onto the surface of the board, which provides immersion to the patient.

Programmable Robot: One of our goals is bring the game beyond the screen, and unto the physical world, using a programmable mBot combined with an onboard esp32. The Jetson Nano would send executables to the esp32, then the mBot, based on the position of the mBot as seen from a camera on the robot.

Challenges we ran into

There was a challenge when it came to running the Google Media pipeline model on the Nano, which didn't run as fast as we wanted to at first. We resolved this issue by implementing a form of hardware acceleration onto the Nano, being able to run the model over 900% faster.

We also had difficulty in the scalability of the hardware when it came to using the esp32 and the MBot boards, where we encountered many problems, such as the lack of appropriate components, and difficulties in communication between the boards and the Nano.

Accomplishments that we're proud of

We are proud of the engineering process the team took when it came to resolving issues like the hardware acceleration issue for the Google Media pipeline, being able to work through the problem, and resolve it quickly. But we are most proud of the fact that we could actually create a system that helps with a real world problem like post-stroke therapy.

What we learned

Through this project, we were able to learn about the uses of hardware acceleration in models, hardware integration and wiring, serial communication with microcontrollers, as well as the issues that come with hardware compatibility.

What's next for Guiding Hands

We hope to be able to be able to more seamlessly introduce the iteration of our design with a sleeker finish, a full robotic integration, and more user-friendly interface.

Built With

3js
cpp
css
fastapi
html
javascript
python
react

Submitted to

Knight Hacks IX

Created by

I contributed to the project's concept, frontend development, backend API integration, camera calibration, and robotics implementation. I helped design an interactive rehabilitation game that uses hand gestures to make physical therapy exercises more engaging.

On the backend, I worked with Python, FastAPI, and the ElevenLabs API to support event-driven voice dialogue for our main character, Lenny. I also worked on the user interface and camera feed calibration to improve player interaction and gesture tracking.

On the hardware side, I developed movement-control logic for an mBot rover and explored integrating it with an ESP32 and NVIDIA Jetson Nano to create a physical companion that could respond to game events. Although we encountered communication challenges that prevented full integration, I helped establish the foundation for connecting our game to real-world robotics.

Marc Alexis
I worked as the main Creative Lead for the project, contributing to the story and overall creative direction of the project. Besides creating the story and concept art of Guiding Hands, I helped assemble the hardware, debug and test the gesture tracking side of the software, as well as the mBot and ESP32.

Diego Gonzales
I worked on the gpu acceleration which lead to the over 900% of increase in the media pipe gesture tracking learning model as well as minor help in implementation of other minor features.

Leonel Hernandez
I primarily contributed to the game's visual design and development, focusing on 3D asset creation, 2D environment design, and world-building. I developed immersive fantasy environments, designed and integrated 3D visual elements, and helped establish a cohesive art direction to enhance the overall player experience.

Ethan Lamb