Rekovery

Inspiration

Each one of us can think of countless times where pain and injury has held us back from being our best selves. Just a few weeks ago, two of us got ourselves concussions and we were reminded of how when things are going awry with your health, all you wish for is to get back to normal. That's where physiotherapy saves the day! By going to a practitioner and getting treatment, complete with a set of exercises to perform frequently, you'll notice just how drastically your condition can improve.

However, bringing the exercises home is one of the hardest parts of getting the most out of physio! It can be difficult to remember which exercises to perform for which problems and how to do each exercise the right way. To make this a lot easier on ourselves, we decided to build Rekovery!

What it does

Rekovery focuses on empathizing with you and understanding all there is to know about the pain or injury you are experiencing, so it can tailor a treatment plan to you. Our friendly assistant will start by asking you a set of quick questions about the nature of your pain, so it can figure out what exactly needs to be addressed. Once it has all this information, it will put together an exercise for you in a set of simple movements, targeting all the muscles/parts of your body that need some attention. Once it has this exercise ready, our avatar will explain and demonstrate how to do it, so you can follow along. Once you're all done, you'll feel a lot more confident and empowered to perform these exercises and give your body the help it needs.

How we built it

In order to dynamically generate treatment plans and provide an interactive experience for the user, this project required us to put together a project with a lot of moving parts.

Frontend:

Javascript + React.js (our language and framework of choice for the User Interface)

OpenAI Whisper (takes in speech from user describing pain, transcribes into text)

Our special Avatar!

Avatar models were created in Blender
Three.JS + Drei (used to render the 3D Avatar)
Mixamo (provided us with animations for our 3D avatar - demonstrating exercises)

Mediapipe (performing pose estimation to determine correct performance of exercises)

Backend:

Python + Flask (language and framework of choice for building a simple REST API)

Gemini 2.0 Flash (processing user's pain information and generating treatment plan)

ElevenLabs (taking treatment plan and converting to audio for avatar to speak)

Challenges we ran into

A major obstacle that challenged us all throughout the process of creating Rekovery was the ensuring that the flow of events in the user interface was very synchronized and felt natural. This challenge began even in the backend, where we had to focus on ensuring that the treatment plan generated by the LLM was broken down into steps that could be performed logically and seamlessly in a simple sequence, so that it feels right for the user. Then, when this generated plan was translated to the frontend, getting the avatar to perform the exercise step-by-step, with a reasonable cadence and in a way that looks realistic proved to be difficult. Finally, using Mediapipe at the same time as everything else in our UI, to pay attention to how the user was performing the exercise, was quite challenging to run alongside the avatar. We wanted to make sure that this aspect of the UI worked harmoniously with the avatar, but at times we found that it was making the overall user experience very noisy, or actually causing errors that prevented the avatar from being able to progress to its next steps.

For each problem, we came up with the following solutions:

LLM treatment plan: we made sure to provide the LLM with a list of common exercise actions that it was allowed to use to put together a physio exercise, and used one-shot prompt engineering to give it an example of expected output
Avatar exercise demonstration: we collected a set of animations from Mixamo and ensured that they correspond to actions that could be generated by the LLM, and ensured that they were performed slowly.
Mediapipe: we used pose-estimation to prevent the program from moving onto the next step of the exercise until it was determined that the user performed the exercise correctly. We had to perform a lot of trial and error to figure out optimal values (e.g., angles of tilt of the neck) for what it meant for an exercise to be performed correctly.

Accomplishments that we're proud of

All three of us have accomplished things we would not have imagined coming into this hackathon! First and foremost, we are really proud that we were able to get an end-to-end flow of Rekovery that worked seamlessly. We proved to ourselves that it was possible to make a multimodal, interactive healthcare app despite the challenges that seemed like they would make it too difficult to do in a weekend. Most of all, we also built something that means a lot to us, so we're extremely proud of our efforts this weekend.

What we learned

As for personal learnings, Alan and Kevin got to learn new frontend-related skills, particularly related to the creation of the Avatar (e.g., Three.JS, Mixamo, etc.) and Matt got the chance to learn new prompt engineering techniques and gain more practice validating and manipulating LLM outputs to make agentic workflows work seamlessly. Beyond this, we feel that building Rekovery helped us get a glimpse the power of AI in the field of healthcare, which we are excited to explore more!

What's next for Rekovery

Rekovery would definitely have a long road ahead of it before becoming a robustly usable tool. However, we envision it being something that could be developed alongside physiotherapists as an assistant that patients could bring home for reminders and continued care outside of the clinic. This would help them make the most of physio, both recovering a lot quicker and staying healthier for longer!