What it does

Cooking Therapy allows for hand therapy patients to go through exercises that improve hand mobility and allow them to work towards rehabilitation. Users can set up a routine that encourages practicing activities daily, can work on specific self-led exercises, and can use the game mode to work through a set of hand therapy workouts that mimic cooking a recipe.

In general, the application accesses the user's camera, utilizing the TensorFlow model as a source of comparison to detect the user's hand motions. After detecting a hand motion (such as a claw stretch) that matches the task at hand, the interface changes depending on the mode the user is in.

For the game mode, each successive hand motion will result in a cooking action (i.e. finger stretch --> picking up ingredients), where the interface will change to show this animation. For self-led individual practice, the tracker will increase by one per hand exercise practiced.

How we built it

We built Cooking Therapy as a web app using HTML, CSS, and Javascript. We implemented a TensorFlow.js model to create and classify our own finger poses.

Challenges we ran into

We initially wanted to implement a Leap Motion controller to track hand movements but were unable to due to hardware constraints.

Accomplishments that we're proud of

  • The animations we created in-house for the game mode as well as
  • Being able to work through our hardware issues and switch over to creating a web application late into the hackathon

What we learned

  • While we weren't able to use the Leap Motion, we learned the initial steps of how to use software, such as Unity, that would allow developers like us to create programs for the device.
  • We learned about how pose detection works on devices without depth sensors
  • We learned a lot about hand therapy exercises and how specific ones promote increases in a patient's strength and motion.

What's next for Cooking Therapy

In the future, we plan on updating the application to include: user statistics, a functioning settings page that would allow the user to make the app more accessible to any visual disabilities they might have (low-vision, blind, colorblindness, etc.), the ability to set up a routine that can send out reminders for the user to come back and practice, and lastly, more recipe options and game modes that users can choose from.

For the settings, this can include the ability to set personalized routines and daily goals for repetition, for which users can work towards these goals through any of the modes: game, self-led, or routine. Any repetitions will count towards these daily goals. In the future, to save this information, we plan to store information on a cloud database to allow users to continuously come back to the website.

This ties into the statistics page, which will show the trajectory of the user's progress (i.e. taking less time to complete cooking a meal from the menu, meaning easier range of motion in fingers) and the user's use of the website throughout time, such as total repetitions and favorite exercises.

Built With

Share this project: