Juve | Devpost

pose detection in action!

Why we made this

Juve is designed with the concept of "rejuvinate" in mind. We aim to help elderly individuals to recover from physical therapy by harnessing the universal appeal of games in a fulfilling way. By offering engaging and interactive methods for users to practice muscle movements and achieve better health results, we seek to craft an accessible platform where rehabilitation feels less like a chore and more like a delightful adventure.

What Juve does

After creating an account and logging in, the user enters a page that displays their recovery statistics (games played, time taken, movements tracked, etc). They proceed by entering "game mode", which incorporates our main technology of leveraging video feed data from the user's web camera and displaying it in a video using OpenCV. We track the data of 32 x and y-coordinates of joints in a feedback loop that updates at a rate of 25 frames per second. The relevant joints being trained (such as the right palm in our current model) are tracked specifically and incorporated into the game logic. During this hackathon, we aimed for a game similar to Fruit Ninja, in which collisions between "fruits" and the user's hand are recorded and used to increment points. We improve upon the existing Fruit Ninja game by adding specifications allowing the user to change fruit size, speed, and direction for specific movement training.

How we built it

Python3 backend, HTML/CSS/Javascript Frontend (with a substantial React/Node frontend effort), OpenCV pose detection, Google Cloud Dataflow data storage and retrieval, SocketIO for data streaming & Flask app routes for user inputs.

Challenges we ran into

While we developed a React / Node frontend, we were ultimately unable to reconcile this frontend into our project due to the challenges of connecting the video stream to the web. In addition, Google Cloud was a new technology to us and authentication barriers would often prevent team members from running files locally. Finally, the backend technology of displaying a video stream with joint overlays was a surprisingly arduous task (at one point, our video feed contained a dizzying stream of continuous flashes).

Accomplishments that we're proud of

Many of the tools and ideas we implemented were brand new for us, including Google Cloud, OpenCV, and the entire concept of adding a video stream to a static web page. Everyone in our team came out having learned something new.

What's next for Juve

In the future, we would like to design more motion games (Flappy Bird, Just Dance) and fine-tune the movement detection options we make through coordinate analysis, such as differentiating between a hand wave and a hand jab by looking at the delta differences in x and y coordinates over a period of time.