Hercules

• 

  Screen Design

Inspiration

All three of are international students who converged on this idea based on a shared experience: our grandmothers, who dreamed of coming to see us graduate, could not make the long trip due to loss of mobility. We have also seen our parents, who are around 60 years old, unable to enjoy the things they used to with us when we were kids: playing basketball, backpacking, skiing, leaving those cherished moments as memories. We realised that even though children born today are expected to live maybe 100+ years, their quality of life in their later years depends on how we take care of our bodies today. We did extensive research into how earlier life experiences affect mobility and realised that this is an uphill battle that starts today. We thus developed an accessible and intuitive approach to tracking and understanding factors that go into your present and future mobility.

What it does

Hercules can see and understand your pain using only your iPhone camera and/or a voice note describing your pain, therefore simplifying understanding pains in the body to even those who aren't health literate. He allows you to log any fleeting, acute, or chronic pain, all while seamlessly tracking your activity using wearable data. He will create a long-term log correlating the two, helping develop an exercise routine based on the user's specific needs and limitations, ensuring that they say safe but active enough to promote long-term mobility health.

How we built it

We first did in-depth research on what affects mobility health in the future and what we can do to alleviate effects of it deteriorating early on. We mapped out user journeys and decided that our target users would be everyone, with an emphasis on those who still have the power to prevent future mobility issues Our product is a react-native app built in JavaScript/Node.js, and utilizes the OpenAI API, namely Whisper for speech-to-text of user voice input, GPT-3.5 Turbo for semantic understanding of that text, GPT-4 Vision for image recognition and reasoning, and OpenAI TTS for text-to-speech for Hercules' voice. We used Figma to mock up the app design while we built the bare-bones user flow, then added the styling. Additionally, although we did not have time to implement this into our app, we worked on developing a "Mobility Score" that uses wearable data to evaluate an individual's mobility, which was inspired by the Oura Ring's "Sleep Score" and "Activity Score" measurements and equations.

Challenges we ran into

We ended up having to design and implement a lot of screens, which had to be done after implementing the image recognition and AI speech software. We also realised that there simply isn't enough data that measures changes in one's mobility long-term, and how acute pains could manifest into more serious conditions with old age. This actually made us more excited about building the product however, as it could allow us to accumulate such data, especially for those historically underrepresented in medical data such as women and minorities. Technically, the main challenge we ran into was that OpenAI's GPT-4 vision model was not recognizing body parts in some cases from the pictures we sent it. We eventually realized that it was due to OpenAI's API automatically cropping our image if it wasn't in the correct aspect ratio, and were able to fix it by implement cropping client-side.

Accomplishments that we're proud of

We were able to design and implement 80% of our screens that would be required for a fully functional product. We also managed to make Hercules a very helpful and personable AI agent that makes the user feel at ease. We are most proud that we have created a tool for our generation to enjoy the later stages in life more than our older family members ever did.

What we learned

We understand mobility physiology far better, as well as designing not only for older generations but the older generations of the future. We also learned that the biggest causes of the health span - lifespan gap are lack of health literacy and access to personalised care, both of which could now be solved given new technologies such as AI agents, which is what we envision for our product.

What's next for Hercules

We would love to develop personalised fitness programs for those who have very specific pains and injuries but not the means to go to a doctor or physical therapist. We would likely offer these programs at a very low cost and keep them accessible in terms of ease of following them. We also want to introduce educational aspects of monitoring pain and habit-building to ensure users are maintaining the actions needed to preserve their mobility until later in life. Finally, a key indicator of mobility health is nutrition, so we want to incorporate a nutrition journal that's as easy to use as the pain log, using just a picture for Hercules to tell the user about their eating habits as it relates to their long-term mobility health.

Built With

• computer-vision
• expo.io
• llms
• node.js
• openai
• speech-to-text
• text-to-speech
• wearables