Inspiration
Healthcare systems worldwide are strained by rising patient loads, limited specialists, and delayed diagnoses, especially in resource-constrained regions. Doctors often juggle fragmented data—images, lab reports, and notes—making it easy to miss subtle but critical patterns. With Gemini 3’s multimodal capabilities, we saw an opportunity to build an AI “second pair of eyes” that can analyze medical images and clinical context together, not in isolation. Our goal was to create a tool that augments clinicians, reduces diagnostic uncertainty, and brings near-specialist support closer to smaller hospitals and clinics. We were inspired by how recent medical AI prototypes can already read scans and suggest differentials, and wanted to turn that power into a usable, hackathon-ready product.
What it does
MedDiag AI is an AI-powered diagnostic assistant that ingests medical images (like X-rays, CT scans, or MRIs) along with key patient details and generates a ranked list of possible diagnoses with explanations. It highlights salient image regions, correlates them with symptoms and history, and produces a structured report that clinicians can quickly scan. The system is designed as a decision-support tool: it does not replace doctors, but offers a fast, transparent “second opinion” to help them prioritize risks and plan next steps. MedDiag AI can also suggest follow-up tests, flag red-flag findings, and summarize its reasoning in simple language suitable for patient communication or teaching.
How we built it
We built MedDiag AI on top of the Gemini 3 family, leveraging its native multimodal support for both images and text. First, we designed a streamlined interface where clinicians can upload scans and enter minimal clinical context (age, symptoms, key lab values). On the backend, this data is packaged into a structured prompt for Gemini 3, which is tuned to output differential diagnoses, confidence-style scores, and an explainable reasoning chain. We then post-process the model output into a clean, sectioned report and display it in a responsive web UI for fast review during clinical workflows. Throughout, we followed hackathon best practices for the Gemini 3 ecosystem—using its APIs for reasoning and multimodal fusion, and focusing on a single high-impact workflow rather than many shallow features.
Challenges we ran into
One of the biggest challenges was designing prompts that encourage clinically safe, cautious behavior instead of overconfident guesses, given the high-stakes nature of medical diagnosis. Balancing detail with latency was also tricky: richer context improves reasoning, but too much text or too many images can slow responses and clutter the UI. Handling diverse imaging types (chest X-rays vs. MRIs vs. CT) with a single pipeline required careful prompt engineering and formatting of instructions. We also had to clearly communicate limitations and add strong disclaimers so the tool is not mistaken for a certified medical device.
Accomplishments that we're proud of
We’re proud that MedDiag AI can take a raw scan plus a few clinical notes and produce an organized, explanation-rich differential diagnosis within seconds. The system surfaces its reasoning in a way that is understandable for junior doctors and students, turning each case into a mini learning module. Even within hackathon constraints, we were able to implement a usable, end-to-end workflow—from upload to AI analysis to final report—rather than just a model demo. Most importantly, mentors and testers immediately saw its potential for triage support and education in settings where specialist oversight is scarce.
What we learned
We learned how powerful multimodal models like Gemini 3 can be when you treat them as clinical reasoning partners instead of just text generators. Good outcomes depended less on fancy UI and more on careful problem framing, prompt design, and clear safety boundaries. We also saw that healthcare AI products live or die by workflow fit: doctors need fast, focused outputs they can trust, not long essays. Finally, exploring related healthcare projects showed us the importance of explainability, disclaimers, and continuous validation with real-world edge cases
What's next for MedDiag AI
Next, we plan to expand support for more imaging modalities and disease areas, starting with high-impact use cases like chest, neuro, and musculoskeletal imaging. We want to incorporate longitudinal analysis so clinicians can compare scans over time and track progression or treatment response. Another direction is integrating guidelines and literature retrieval, so the tool can attach concise, up-to-date references to each suggested diagnosis. Longer term, we envision pilots with teaching hospitals, using MedDiag AI as both a decision-support system and an interactive tutor for medical students and residents.
Log in or sign up for Devpost to join the conversation.