MedVisit

EMR Dashboard
Video Analysis
Doctor Visit Summary
Medication Suggestions
Timestamp Indexing
SOAP

Inspiration

Hospital visits are often overwhelming and confusing. Patients frequently forget important recommendations shared during appointments and struggle to interpret complex medical reports. Language barriers can further limit understanding, leaving many patients unsure about their care plan. At the same time, clinicians spend valuable time creating SOAP notes instead of focusing fully on the patient. MedVisit was inspired by the need to make clinical conversations clearer, more accessible, and more efficient for everyone involved.

What it does

MedVisit is a comprehensive healthcare consultation platform that bridges the gap between doctors and patients through intelligent AI-powered analysis and accessible reporting:

For Doctors:

Upload consultation videos for automatic analysis
Receive AI-generated Electronic Medical Records (EMR) with structured SOAP notes (Subjective, Objective, Assessment, Plan)
Review and approve AI-extracted medication suggestions before finalizing
Save time on manual documentation through automated clinical note generation

For Patients:

Access a personalized, easy-to-understand health dashboard with:
- Vital signs and health metrics displayed with visual status indicators
- Active diagnoses and medical codes explained clearly
- Current medications with dosages and frequencies
- Lab results with normal ranges for reference
- Allergies prominently displayed for safety
- Personalized care plan with timestamped video navigation
Upload After Visit Summary documents to auto-populate their health profile
Support for multiple document formats (PDF, TXT, DOCX) and multiple languages

How we built it

Architecture

We built MedVisit using a modern two-service architecture:

Backend (FastAPI):

Modular router-based architecture for scalability:
- TwelveLabs integration for video transcribing
- ElevenLabs for natural, multilingual audio summaries
- SOAP note generation and clinical summary extraction
- Chapter-wise Consultation segmentation
- After Visit Summary document processing with medication review
Background task processing for long-running video indexing

Document Processing Pipeline

For robust document parsing without external APIs:

PDF/DOCX/TXT Extraction - PyPDF2 and python-docx libraries
Pattern-Based Parsing - Regex-driven extraction of:
- Patient demographics (name, DOB, age, MRN, insurance)
- Vital signs with automatic status classification (normal/warning/critical)
- ICD-10 diagnosis codes with active/chronic/resolved status
- Medication information (dosage, frequency, status)
- Lab results with dates and normal ranges
- Allergies with severity levels
- Care plan actionable items

Data Flow

Doctor uploads consultation video
TwelveLabs multimodal AI analyzes the recording
Backend extracts medications, generates SOAP notes, creates summaries
Doctor reviews, approves, or overrides AI-suggested medications
Patient accesses dashboard with:
- AI-extracted clinical insights
- Structured medication plan
- Timestamped video navigation for quick reference review

Frontend

Built using React and Next.js
TanStack Query for efficient server state management with smart caching
Tailwind CSS with shadcn/ui for a consistent, professional healthcare UI
Context-aware routing (Doctor, Patient, Dashboard views)
Drag-and-drop document upload interface for documents and videos

Challenges we ran into

Finding the right LLM for generating a user-friendly and doctor-approved EMR report
TwelveLabs API usage limit
Finding the right solution for document parsing (Gemini, OpenAI, or PyPDF)
- Deprecation of the original google.generativeai package to google.genai with breaking API changes
- Encountered OpenAI rate limits and quota issues requiring pivot to local processing
- Solved by implementing comprehensive regex-based document parsing that works completely offline
Needed to handle multiple document formats and variable formatting styles
- Implemented flexible regex patterns that handle missing data fields

Accomplishments that we're proud of

One of the biggest acomplishments we are proud of is our use of TwelveLabs. We were able to use it to automatically analyze doctor-patient consultation videos and extract clinically relevant information. TwelveLabs processes the video's visual and audio content to generate structured SOAP notes (Subjective, Objective, Assessment, Plan), chapter summaries with timestamps, and searchable transcripts—transforming hours of unstructured video footage into organized, actionable medical documentation that both doctors and patients can easily review and act upon.

What we learned

One of our biggest takeaways was realizing that ideation is just as critical as execution. We spent significant time pivoting and refining our concept—debating what features to include, how the doctor and patient views should interact, and what AI capabilities to leverage. Each tweak to our vision meant reworking code, restructuring our architecture, and sometimes starting over on components we'd already built. This taught us that investing more time upfront to solidify the idea—understanding the problem deeply, defining clear requirements, and committing to a focused scope—would have saved us countless hours of rework during development. In a hackathon setting where time is your most precious resource, a well-defined idea from the start is worth just as much as clean code.

What's next for MedVisit

Looking ahead, MedVisit has significant potential for growth. Our next steps focus on improving the AI model's accuracy and validation capabilities—fine-tuning Llama on medical datasets to better recognize drug names, dosages, and clinical terminology, and implementing confidence scoring so doctors can quickly identify which AI-extracted information needs closer review. We also plan to implement full HIPAA compliance with encrypted data storage, secure authentication, and audit logging to ensure patient information is protected and the platform is ready for real-world clinical deployment. By making the AI smarter and more reliable, we reduce the cognitive load on physicians—transforming their role from manually extracting information to simply validating and approving accurate data.