Inspiration
Have you had times when you feel a sudden sting feeling in your heart and felt worried about it? You went online and checked, and couldn't find an answer. Then finally you decided to go to the doctor, only to forget exactly where you felt the pain and speficically how it felt.
For people who are cannot afford frequent health checks from doctors, the time gap becomes longer and the memory becomes worse, which then hinders their chance of getting accurate treatment in time. When diseases don't get treated in time, they grow into more severe cases.
Even Harder
For people who are not proficient in English, communicating how their pain feels is also a barrier. They might only know the word "hurt", (which I think that's the level I can best achieve (cry)), and doctors may get impatient. According to a study done in 2022 published on the National Institute of Health, "Between 2012 and 2018, 9754 English proficient and 1878 LEP patients were hospitalized for traumatic injury. In multivariable models adjusted for demographic and injury characteristics, LEP patients had 2.4 fewer pain assessments per day compared to English proficient patients (7.21 versus 9.61, P = 0.001)".
That's why we build Iris, to make it easier for people to document, express and heal their pains.
What it does
Record your pain: Draw your pain on a 3D body with layers to select from for. With specially designed assistive visuals, you can frictionlessly document your pain with structure (location, pain types & severity).
Wearable Sensor Data: To complement symptom tracking, Iris incorporates sensor data from an ESP32-based wearable. It tracks your health data (heartrate, body temp, impact and fall, etc.) and notifies you of any anomalies. Then you get prompted to type in any symptoms you have, which then helps the sensor learn and give more specific notifications.
(Due to time constraints, we're having difficulty transmitting the sensor data to our database because our sensors were damaged while transferring from the breadboard to the 3D-printed container, so for now we are using sample data on our mobile app for the sensor)
Express freely: Our built in language translation tool takes voice and text entry and translates them into English. Not only that, it captures any symptom-related expressions and organizes and highlights them.
Medical data-driven analysis model: All of you data, both self-reported and sensor-collected, is analyzed using this biomedical knowledge graph: https://monarch-app.monarchinitiative.org/ No AI is involved in the process to minimalize hallucination.
Condensed report export: With one click, you can export your health data and symptoms over time to pdf for clinicians to view. The disease indication and data graphs are in the front and original data are added in the bottom, saving the clinician's time.
How we built it
Mobile app development:We designed and prototyped our mobile app with Figma, using agents and editing ourselves. We then exported code with Claude with Figma MCP.
3D body model integration: For the interactive anatomy experience, we prepared the Z-Anatomy 3D model in Blender and exported it as .glb assets, which we render using Three.js.
Health data tracking:We utilized the API of an online open-source medical knowledge map analyzing real medical data. https://monarch-app.monarchinitiative.org/
Wearable hardware: We used ESP32-S3 microcontroller with multiple sensors. We 3D-printed a container we designed on Fusion. We created a prototype on breadboard, tested, soldered and transferred it to the wearable container.
Challenges we ran into
Preparing a large 3D anatomy model: Z-Anatomy is a detailed anatomy atlas that was resource-intensive to prepare and load into our app. We used Blender to prepare and export the model into .glb assets, but getting the 3D viewer to initialize reliably remained a challenge. At one point, the flat body illustration stayed on screen because the 3D view crashed before it could replace it.
The model also doesn't have a female version, so we couldn't allow users to mark or draw cramps or PMS-related symptoms as we visioned.
**Exporting code from design and prototype on Figma: **Setting up VS Code to work with Figma Dev Mode took troubleshooting. We solved the issue by reading Figma Learn docs and going to the Figma workshop.
Configuring Expo: Getting Expo Go and Expo CLI authentication working across devices required additional setup. At first, we didn't know that they are two separate accounts. We also encountered Node.js and browser-related issues that interrupted our development workflow. We solved it by consulting Chat-gpt lol and relentlessly trying.
Limited hardware supplies: We ran into shortages of sensors needed for our wearable, especially the heartbeat sensor was out of stock completely. We built an alternative with photoresistors and LED lights (put fingers between it).
Wiring: We first followed the wrong documentation for our ESP32 model and connected the wrong pinouts. Then other teammates double checked and spotted the issue.
Transferring the hardware prototype: We tried our best to solder and wire everything but due to some bad connections, some of our sensors couldn't pick up data.
Accomplishments that we're proud of
Building a working product with all the core features we wanted. Learning so many new tools throughout the hackathon! We had limited mobile app development experiences so it took a long time to figure out how to use Expo Go.
What we learned
Pivoting when something doesn't work out in limited time and resources. Though frustrated, we managed to build an alternative for the heartrate sensor. Also our models don't have female versions, so we decided to delete the reproductive system and make it a general human model.
What's next for Iris
Fix the sensor connections so the data can be sent to the mobile app.
Develop a machine learning model that enables learning and more personalized feedback, or even prediction in advance.
Adding multilingual support: Support more languages for the app UI to support users with other native languages. We've tried adding this feature but kept encountering Node.js errors so we ended up not adding it.
Expanding anatomical coverage: Develop more detailed tracking for reproductive and pelvic health, helping users document symptoms in areas that may be difficult to discuss or track with general-purpose health apps.
Built With
- esp32
- figma
- react-native
- supabase
- three.js
