MedClarify AI

💡 Inspiration

Medical jargon is confusing, and misinterpreting prescription labels can be dangerous. Patients often leave the pharmacy with bottles covered in tiny text, complex chemical names, and vague instructions. We built MedClarify AI to act as a digital patient advocate. We wanted to create a tool that instantly translates intimidating medical labels into empathetic, easy-to-understand daily schedules, prioritizing patient safety above all else.

🚀 What it does

MedClarify AI is a secure, client-side web application. Users input their Gemini API key, snap a photo of their prescription bottle or handwritten note, and our AI engine processes the image to extract:

• The exact medication name.
• Plain-English, jargon-free instructions.
• A clear, chronological Daily Schedule (e.g., 8:00 AM: Take 1 pill).
• Warning labels and contraindications.

Crucially, it features strict medical safety guardrails. If a label is blurry or illegible, the app safely defaults to advising the patient to consult a doctor, rather than guessing.

⚙️ How we built it (Track Alignments)

1. Best App Deployed on Google Cloud

We engineered a completely serverless architecture deployed natively on Google Cloud Infrastructure via Firebase Hosting. By utilizing a "Bring Your Own Key" (BYOK) architecture, we kept the application 100% statically hosted. This ensures infinite scalability, near-instant load times, and top-tier security without exposing backend variables or relying on vulnerable server middlemen.

2. Best Use of Gemini API

We bypassed standard conversational chat interfaces and integrated gemini-2.5-flash as a deterministic data-extraction engine. By routing base64 image data directly to the multimodal vision endpoint and configuring the SDK with responseMimeType: "application/json", we forced the AI to act as a strict parser. It returns a guaranteed TypeScript interface that flawlessly populates our React components.

3. Best Usage of Codex

We utilized the Codex framework to enforce vital health-tech safety guardrails:

• AGENTS.md: Defined an empathetic, plain-English patient advocate persona.
• SKILL.md: Created a strict "Forced Schema Evaluation" fallback. We engineered the prompt to force the AI to mathematically calculate its OCR reading confidence before writing any instructions, locking it out of hallucinating if the text is illegible.

🚧 Challenges we ran into

1. Architectural Boundaries & CORS: We initially attempted to use the brand-new @google/genai SDK. However, we quickly realized the Interactions endpoint enforces strict CORS policies requiring a dedicated backend. To keep the app completely serverless, free, and accessible on Google Cloud static hosting, we made the strategic engineering decision to pivot to the stable @google/generative-ai SDK (Gemini 2.5 Flash), which safely supports client-side calls.

2. The LLM "Overconfidence Bias": During stress-testing with dense, handwritten, non-English (Bengali) prescriptions, we discovered a massive flaw in standard zero-shot prompting: LLMs suffer from "Overconfidence Bias." The model wanted to be so helpful that it hallucinated standard medical advice to fill in the blanks of text it couldn't read. We overcame this by inventing a Forced Schema Evaluation. We restructured our JSON output so the AI had to output a boolean is_safe_to_translate variable first. This structurally locked the AI into a safe fallback response if it couldn't read 90% of the label.

🏆 Accomplishments that we're proud of

• Shipped a fully functional, highly polished AI computer vision application entirely on a serverless free-tier stack.
• Successfully tamed LLM hallucination in a high-stakes (medical) context using advanced JSON schema prompting.
• Built an incredibly clean, accessible UI using React, Vite, and Tailwind CSS.

🧠 What we learned

Prompt engineering is not just about writing good text instructions; it is about structural enforcement. We learned that forcing an LLM to evaluate its own confidence as the very first key in a JSON object fundamentally changes how it processes the rest of the generation, practically eliminating hallucination.

🔮 What's next for MedClarify AI

To make this ready for real-world clinical use, we would migrate to a Multi-Agent Pipeline: Agent 1 would strictly perform OCR transcription with zero formatting, and Agent 2 (with a strict temperature of 0) would route that raw text into the JSON schedule, entirely removing the overconfidence bias of a single-pass model.

Built With

• firebase
• gemini-api
• prompt-engineering
• react
• tailwincss
• typescript
• vite