Medlm

Inspiration

Most times during hospital visits, I realize that many people don't really understand what the doctors say. They keep nodding, but deep down they want to know more about their condition. Often, due to time constraints or feeling intimidated, they suppress asking these crucial questions. I have been in similar situations myself, and that is what inspired me to build MedLM.

MedLM is an AI-assisted health insight tool that helps patients understand their health better, empowering them to make informed decisions. MedLM ingests and analyzes patient records to visualize major timelines of their health history and detect invisible trends (including vitals) from their medical records. It also provides a smart chat interface for users to ask follow-up questions all powered by Google Gemini 3.

What it does

MedLM serves as a personal "Medical Intelligence System."

• Unified Timeline: It ingests fragmented records (PDFs, images, text) and unifies them into a chronological health history.
• Trend Detection: It spots long-term trends in vital signs (like rising blood pressure or falling iron levels) that might be missed in isolated visits.
• Medical Chat: Users can chat with their records. "What did the cardiologist say in 2023?" The system reads the actual files to answer accurately.
• Simplification: It translates complex medical terms into plain English.

How I built it

I built MedLM using a modern, scalable, and type-safe stack:

• The Brain (AI): I moved beyond simple prompting to DSPy, a framework for programming LM interactions. I used Google Gemini 3 Flash for its speed and massive context window.
  • I implemented ReAct Agents that can use tools—specifically, a ReadMedicalRecord tool that allows the AI to fetch and read full file contents on demand, ensuring high-fidelity answers (I plan to also compliment this with context caching).
• Backend: Python FastAPI and PostgreSQL handle the core logic.
• Async Pipeline: Celery and Redis manage heavy extraction tasks. I built a custom Server-Sent Events (SSE) pipeline to stream real-time analysis updates to the client, creating a "storytelling" experience while the AI works.
• Search: I used Qdrant for vector search to find relevant context across thousands of pages.
• Frontend: React (Vite), TanStack Router, and Framer Motion create a fluid, premium dashboard feel.

Challenges I ran into

• Hallucinations vs. Trust: In healthcare, accuracy is paramount. An LLM vaguely summarizing a file wasn't enough. I solved this by enforcing DSPy Signatures that require the model to explicitly cite the verbatim text from the record for every claim it makes.
• The "Needle in the Haystack": Finding a specific test result in hundreds of pages. I implemented a hybrid search strategy, combining vector embeddings with keyword search, and gave the application the specific ability to "open" and read documents when necessary.
• Latency: Deep analysis takes time. Instead of boring loading spinners, I engineered a real-time status stream that updates the user on exactly what the AI is doing ("Extracting text...", "Analyzing trends..."), turning wait time into an engaging experience.

Accomplishments that I'm proud of

• The "Reader" Agent: Successfully allowing the chat bot to recognize when it needs more context and autonomously fetching/reading a specific medical record to answer a user's question.
• Visualzing the Invisible: Taking raw, unstructured PDF text and converting it into structured JSON data that powers beautiful, interactive charts for vital signs.
• A "Human" UI: Building an interface that feels empathetic and clear, not sterile and clinical.

What I learned

• Structure is Freedom: By constraining the AI with strict schemas (DSPy), I actually enabled it to be more capable and reliable.
• Context is King: The huge context window of Gemini allowed me to dump entire medical histories into the prompt for trend analysis, something impossible with smaller models.
• User Expectations: Patients don't just want data; they want stories. Explaining "why" a trend matters is just as important as showing the graph.

What's next for Medlm

• Live Integration: Connecting directly to hospital portals (Stanford Health, Mayo Clinic) via FHIR API integration (already planned and prototyped).
• Mobile App: Bringing MedLM to iOS and Android for health insights on the go.
• Caregiver Mode: allowing secure sharing of simplified insights with family members.

Built With

• bash
• dspy
• fastapi
• gemini
• python
• tailwind
• tanstack
• typescript