MotionMend VR

Used the Fetch AI API to send and receive data from Firebase, store information in context, and communicate between agents to share data.
Utilized the Gemini LLM to gauge the progress and areas of improvement of the user and add more interactivity alongside encouragement.

Problem

The current landscape of post-stroke rehabilitation faces significant challenges. Physical Therapy still relies on qualitative assessments, which lead to inconsistent progress tracking and poor analytics. Traditional therapy methods often lack engagement and customization, which are crucial for effective recovery. Globally, the variance in access to skilled therapists further complicates the delivery of timely and efficient care. With stroke being a leading cause of serious long-term disability, enhancing the efficacy of rehabilitation is imperative.

In simple words, therapy is qualitative, expensive, unavailable in remote areas, and lacks engagement and customization.

The American Heart Association notes that the intensity, frequency, and duration of practice are critical for recovery, yet many patients struggle to achieve these therapy goals.

Our Solution

Our Philosophy is based on Repetitive Task Training (RTT), allowing users to relearn lost skills and progressively improve on the same tasks.

Our VR therapy app leverages AI agents to dynamically personalized gamified experiences for the users. By simulating a virtual environment where patients can engage in targeted exercises, the app encourages more frequent and intensive therapy sessions, thus adhering to the best practices outlined by the American Academy of Physical Medicine and Rehabilitation and the American Society of Neurorehabilitation.

Key Features of the VR Therapy App:

Customized Exercise Programs: Tailors exercises to individual patient needs, progressing in difficulty based on real-time performance and recovery rate based on Fetch AI's AI agent.
Engaging Virtual Scenarios: Increases patient motivation through gamified elements and immersive environments made in Unity that make repetitive practice more enjoyable.
Real-Time Feedback: Provides instant feedback on performance, helping patients correct their form and technique to maximize the efficacy of each session.
Tacking Experience Quality: Using virtual checkpoints, the app makes sure users perform the correct movements.
Data Tracking and Analytics: Monitors patient progress over time, giving therapists detailed reports to fine-tune rehabilitation programs.
Output SOAP Notes after each session: SOAP stands for Subjective, Objective, Assessment, and Plan. SOAP notes are a documentation method used by healthcare providers to write notes in a patient's chart.
Gamifiying therapy: Users gain points in a game after they complete levels!

How we designed the games

Each game targets different muscles, which allows us to analyze the patient's progress on every muscle!

Game 1: Forearm Flexors, a virtual gardening experience targets muscles along the inner side of the forearm responsible for wrist and finger flexion. In biological terms, the Flexor carpi radialis, Flexor carpi ulnaris, and Flexor digitorum superficialis.

Game 2: Bicep Builder, a virtual rowing experience, targets the muscle in the front of the upper arm, crucial for bending the elbow. In biological terms Biceps Brachii.

How we built it

We used Fetch AI's agents to process the user's progress based on real-time data from the game. The agent sends real-time data to a Firebase database that is connected to our VR Unity application. The Unity app pulls data from Firebase and changes the game's specs, making it easier or harder dynamically.

In the background, the Gemini Pro model processes all the data, including voice from the user and an in-game log. The Gemini Pro model relays an encouraging statement and guides the use through the therapy. Gemini also makes notes in the background keeping track of what the user says and their performance. Gemini produces a SOAP report at the end for a doctor or therapist to review.

The Unity app uses 3D figures and parameters from the Fetch AI model to customize the game for the user!

Challenges we ran into

Implementing VR games since we were short-staffed and couldn't find enough Unity devs.
Constant back and forth with API endpoints since we transmitted a lot of data between the Oculus and Python server. ## Accomplishments that we're proud of Building a VR-AI pipeline in 36 hours! We had never used databases with Unity before, so we had to figure it out! Turns out Unity and real-time database support is also very poor. We used Rest API to link to Firebase in real-time.

What we learned

How to use Firebase in Unity.
A LOT OF Fetch! Wrote 600 lines of Fetch code!
We learned how to handle latency between multiple systems (VR Oculus, AI Agents, Firebase database.
We learned how to run a chat log on Gemini to give constant responses to the VR Oculus. We also made sure Gemini used data points from the game to tailor its responses.

Citation

[1]: Winstein CJ, et. al, “Guidelines for Adult Stroke Rehabilitation and Recovery: A Guideline for Healthcare Professionals from the American Heart Association/American Stroke Association.” Stroke, U.S. National Library of Medicine, pubmed.ncbi.nlm.nih.gov/27145936/. Accessed 20 Apr. 2024. [2]: French, Beverley, et al. “Repetitive Task Training for Improving Functional Ability after Stroke.” The Cochrane Database of Systematic Reviews, U.S. National Library of Medicine, 14 Nov. 2016, www.ncbi.nlm.nih.gov/pmc/articles/PMC6464929/. [3]: Podder, Vivek. Soap Notes, U.S. National Library of Medicine, 28 Aug. 2023, www.ncbi.nlm.nih.gov/books/NBK482263/.