MediMemo

Your patients' history, always remembered.MedMemo uses AI to ingest, organize, and recall medical records instantly. Upload documents, record sessions, and ask questions — all in one place.

Comment

🧠 Inspiration

Healthcare data is fragmented, hard to interpret, and often overwhelming for both patients and clinicians. We were inspired to build MediMemo to simplify medical understanding by transforming complex records into clear, actionable insights. Our goal was to create a system that not only retrieves information but also remembers, connects, and explains it over time.

🚀 What it does

MediMemo is an AI-powered medical assistant that enables users to interact with healthcare data in a conversational way.

  • Upload and analyze medical documents (PDFs, reports, notes)
  • Perform intelligent search using hybrid retrieval (vector + keyword)
  • Provide contextual answers with precise citations
  • Maintain longitudinal memory of patient history
  • Support multimodal inputs (text, images, future audio/video)
  • Deliver real-time, streaming responses through an intuitive UI

⚙️ How we built it

We designed MediMemo as a modular, scalable system:

  • Frontend: Next.js + assistant-ui for real-time conversational UI
  • Backend: FastAPI handling ingestion, querying, and orchestration

  • RAG Pipeline:

    • Chunking (500–800 tokens with overlap)
    • Hybrid retrieval (BM25 + vector search)
    • Reranking for relevance

  • Vector DB: Weaviate / Qdrant for semantic search

  • Storage: MinIO for documents and multimodal assets

  • LLM Layer: Self-hosted / API-based LLM for reasoning and response generation

  • Streaming: SSE for real-time responses

  • Deployment: Docker + Azure Container Apps (Terraform for IaC)

⚠️ Challenges we ran into

  • Multimodal complexity: Handling text, images, and structured medical data consistently
  • Latency vs accuracy trade-offs: Balancing fast responses with deep retrieval
  • Chunking strategy: Avoiding context loss while keeping retrieval efficient
  • Frontend-backend sync: Managing streaming responses and state updates reliably
  • Environment configuration: Fixing hardcoded localhost dependencies during cloud deployment

🏆 Accomplishments that we’re proud of

  • Built a fully working end-to-end RAG system with real-time responses
  • Achieved ~30% improvement in relevance using hybrid retrieval + reranking
  • Designed a clean, production-ready architecture (Docker + cloud deployment)
  • Implemented persistent memory across interactions
  • Created a professional, intuitive UI for complex medical workflows

📚 What we learned

  • Importance of hybrid retrieval over pure vector search
  • How to design LLM systems with grounding and citations
  • Trade-offs between latency, cost, and accuracy
  • Real-world challenges of deploying multi-service AI systems
  • Value of modular architecture for scaling AI applications

🔮 What’s next for MediMemo

  • Add voice interaction and real-time consultation features
  • Integrate clinical decision support systems
  • Enhance multimodal understanding (video + medical imaging)
  • Implement fine-tuned domain-specific models
  • Expand into a collaborative platform for clinicians and patients
  • Strengthen privacy, security, and compliance (HIPAA-ready)

Built With

  • augment
  • railtrack
  • tool-ui
Share this project:

Updates