Mendly

• 

  Patient dashboard

• 

  Practitioner Exercise Selections

• 

  Practitioner Planning

• 

  Practitioner Alerts

• 

  Practitioner Dashboard

• 

  Patient Plan

• 

  Patient Goals

Inspiration

After one of our teammates grandparents suffered from a stroke, and found that stroke recovery could be better, we decided to make a change in the way stroke victims receive treatment.

What it does

Mendly uses AI to recommend cognitive and physical exercises for stroke patients that are tailored and finalized by practitioners. Then patients undergo these exercises and a detailed report and future plans are created depending on their performance.

How we built it

We built a Next.js application that utilizes other various third-party APIs to support its numerous features. Here is the list below:

• Next.js - Provides a monolithic web application framework where frontend and backend work can happen in one place
• MediaPipe - Tracks patients' movements to work on their exercises and report them to the AI pipeline for analysis
• Backboard - Provides the AI features utilized by the application such as memory, data storage, etc.
• Gemini - LLM model for recommending training sets, plans, and summarizing reports
• ElevenLabs - Provides the voice-based accessibility features in training exercises
• TigerData - Stores accounts and user data, as well as patient training results and safety alerts

Challenges we ran into

Our primary issue was dealing with inaccurate results from the MediaPipe engine; it often would not recognize when certain actions were performed correctly and therefore stop progress. But further training and fine tuning of the data has allowed us to largely overcome this obstacle.

Deployment across our devices was another issue that we solved quickly via Docker Compose, which has largely eliminated the issue with different deployment environments.

Coming up with a disability friendly UI/UX was another constraint we worked with by researching friendly designs that are usable across different generations while maintaining an eye-catching design.

Accomplishments that we're proud of

Perfecting our CV engine to a point of near-complete accuracy has been our largest effort as it was the most make-or-break aspect of the project.

We also enjoyed how we researched the wide spectrum of stroke symptoms and built a platform that aims to cover as many symptoms as possible in an AI-driven and adaptive manner.

What we learned

We learned how to effectively use AI coding tools to build our project by coming up with clear, comprehensive software engineering diagrams and plans to build the project with as little friction as possible.

We also learned about the complexities of CV, we learned how can the website detect body landmarks by using MediaPipe and how to make the AI's result become reliable by calculating movement metrics like joint angles, range of motion, and valid repetitions. Moreover, we also learned that we shouldn't send the raw video and raw metrics directly to the AI, instead we can make it into structured performance metrics and it can be used by Backboard and Gemini when drafting the patient's next adaptive exercise set.

What's next for Mendly

One of our first priorities is getting a HIPAA-compliant Gemini model to ensure a production-ready environment.

We also want to extend ElevenLabs's functionality to work across the whole website to increase accessibility beyond exercises.

Lastly, we want to expand our exercise sets beyond what we currently have to reach out to as many patients as possible. Because everyone deserves to have a chance at recovery.

Built With

• backboard.io
• elevenlabs
• gemini
• mediapipe
• next.js
• react
• tigerdata