ClarityAI

Step 1 on onboarding form
Homepage
Medical Explanation Page
Step 2 on onboarding form
Step 3 on onboarding form

Inspiration: As children of immigrant parents, we’ve witnessed how language barriers and complex medical jargon create cognitive overload and mistrust during stressful hospital visits. We wanted to build a tool that bridges the gap between clinical healthcare and cultural understanding, ensuring families can comprehend treatment plans and side effects without having to rely on unverified internet forums.

What it does: Our app translates a doctor's live speech into simplified, culturally relevant terms in the patient's preferred language. Upon downloading, users complete a short onboarding form regarding their cultural and linguistic preferences. During a consultation, the app provides real-time, culturally sound translation. It also saves session histories and includes a dedicated medical search feature where patients can look up treatments and medication side effects, empowering them and rebuilding trust in their healthcare providers.

How we built it: Frontend & Design: We designed the UI/UX using Figma (Make & Prototype). The app is built with React Native and hosted via Expo.

Backend: Developed using TypeScript.

AI Architecture: We utilized the ElevenLabs API for seamless Speech-to-Text and Text-to-Speech. The Gemini API handles the live multi-lingual translation and jargon simplification. For rigorous clinical accuracy regarding medicines and treatment plans, we integrated the OpenBioLLM-70B open-source model via Backboard.ai.

Challenges we faced: The Frontend Framework Gap: Converting our Figma designs into our Expo environment was a major bottleneck. Since Figma naturally exports to React JS, manually refactoring the component structures and styling into React Native caused extensive compatibility errors, costing us hours of critical hackathon time.

LLM Tokenization Limits: We hit a massive roadblock when feeding non-Latin characters (like Mandarin and Hindi) into the OpenBioLLM model, as it couldn't process those specific tokens. We resolved this by engineering a middleware pipeline, leveraging the Gemini API to pre-translate the text before routing it into the biomedical model.

What we learned: Complex API Pipelining: We learned how to effectively orchestrate and synchronize multiple LLMs and audio APIs within a React Native environment without breaking the user experience.

Engineering Empathy: Most importantly, we learned that accessible healthcare requires far more than literal translation. Building our "Cultural Nuance Layer" taught us how to move beyond basic NLP and design AI with cognitive empathy and sociological awareness.

Our use of interdisciplinary fields: Our project is a critical fusion of computer science, linguistics, biology, and pharmaceutical sciences. This interdisciplinary approach allows us to address medical inequity by creating a product that not only delivers medically accurate information (biology and pharmaceutical sciences) but also ensures it is delivered with cultural nuance (linguistics and computer science). It is a clear example of building a great product that is both culturally sensitive and medically reliable.

Our impact on social justice: Our project democratizes critical medical knowledge, especially for families navigating complex healthcare systems, often in low-income areas. It provides the support and assurance needed by translating clinical jargon into comprehensible, culturally sensitive terms. Every family deserves to understand the care their loved one is receiving and be empowered to ask essential questions, fostering trust with their healthcare providers. Our app ensures anyone, regardless of background, can actively participate in their family member's treatment.

Check google drive for video!