Inspiration

We were inspired by Google FitBit's AI assistant giving personalized workout information and thought that this could be developed into a website that helps to simplify the navigation of medical care. This process is unique to each person, so we thought of using AI to personalize it, especially for specific disabilities.

What it does

Takes photos of medical forms/documents (not including insurance, billing, etc.) and translates them into laypeople speak. This can be done through manual input or speaking input (speech to text assistance). When analyzing the documents, it catches and removes personal information like emails, phone numbers, and user IDs. The simplified version of the test will be displayed in a side-by-side format with the original. The user can click on certain sections of the simplified document and the corresponding translation will be highlighted in the original version. The user can also ask clarifying questions about the document by speaking or text input. Spoken questions will be answered aloud, while typed answers will have a “read aloud” option. Any answers that cannot be found in the document will return “Not in this document”.

How we built it

TypeScript and React with accessible components (React Aria). The same code runs as a website, an installable offline app, a Windows/Linux desktop app (Tauri), and an Android app (Capacitor).

Challenges we ran into

OCR found only about 43% of key phrases on phone-style photos at first. We built an evaluation harness and tested cleanup steps one at a time. Removing shadows helped dim photos a lot but hurt clean pages, so we made it conditional. Picking results by OCR confidence backfired, so we dropped it.

What's next for Accessible Medical Care Navigator

We would like to add support for different languages, as there are many who many not be proficient in English or are more comfortable reading in another language. Additionally, we would like to add walkthroughs, where an AI assistant can walk one through making an appointment or filling out a form. This would be helpful for those unfamiliar with the process or for those with cognitive impairment. It would provide things like checklists that will clarify the process/remind users.

Video Link:https://drive.google.com/file/d/1zABICu95d4EZOw4NpW8YGB2xQGPSZQnb/view?usp=sharing