Inspiration Medical trainees are constantly busy with hospital work and running around . They are also overwhelmed with a plethora of resources. We wanted to simplify learning by giving them a personal tutor that centralises their curriculum by using the curriculum set by the department and adapts to how want to learn.
What it does Medical Tutor AI ingests medical books and journals to teach trainees through conversation, generate questions from MCQs to case scenarios, and support voice interaction. Responses adapt to the trainee's level.
How we built it FastAPI backend with a React frontend. We used a hybrid RAG pipeline combining BM25 keyword search and vector similarity search with multi query expansion over ChromaDB. Document processing with PyMuPDF, embeddings and chat through OpenAI, and a voice pipeline using MediaRecorder with Whisper for transcription and OpenAI TTS for speech output.
Challenges we ran into Getting retrieval quality right was the biggest challenge. Pure vector search missed parts of medical classifications spread across pages, so we had to build hybrid retrieval from scratch. Voice mode was also tricky since the Web Speech API was unreliable, so we scrapped it entirely and built a custom pipeline with Whisper.
Accomplishments that we're proud of The hybrid retrieval actually works. Asking about a five type surgical classification now returns all five types instead of three. The voice mode runs smoothly with no lag. And the whole system runs on less than a dollar of API credits for a 4000 page textbook.
What we learned RAG is only as good as your retrieval. Chunk size, overlap, and search strategy matter more than the LLM you pick. We also learned that combining keyword and semantic search is essential for medical terminology where exact terms matter.
What's next for Medical Tutor AI Stratified access levels so medical students, junior residents, and senior residents get fundamentally different depth and complexity in responses. We also want to add spaced repetition tracking, performance analytics, and support for image based questions from surgical atlases.
Log in or sign up for Devpost to join the conversation.