• 

  Architecture diagram

MedAssist AI - Hackathon Submission

Inspiration

Growing up, I witnessed rural clinics struggle when internet connectivity failed just when patients needed medical guidance most. With 3.7 billion people living in areas with unreliable internet access, and over 600 million Hindi speakers underserved by existing medical AI tools, I was inspired to create a solution that would never leave anyone behind regardless of their connectivity status or language barriers.

What it does

MedAssist AI is a bilingual (Hindi/English) voice-enabled medical assistant that seamlessly operates both online and offline. When connected, it leverages Claude 3.5 via AWS Bedrock with RAG powered responses using medical datasets. When offline, it automatically switches to local BioGPT inference with edge computing. The system accepts voice/text input, processes medical documents via OCR, and provides audio responses, ensuring healthcare accessibility regardless of network conditions.

How I built it

Online Mode:

• AWS Bedrock (Claude 3.5) for intelligent responses
• Pinecone vector database with PubMedQA and MedQuad datasets for RAG
• OpenAI Whisper API for speech-to-text
• Amazon Polly for multilingual text-to-speech
• AWS Textract for medical document OCR

Offline Mode:

• BioGPT (Q5_K_M GGUF) via llama.cpp for local inference
• faster-whisper-base.en for local STT
• Glow-TTS for offline voice synthesis
• easyOCR for document processing

Architecture: I built a Flask backend with React frontend, implementing automatic network detection for seamless mode switching, and session-based conversation history.

Challenges I ran into

• Model optimization: Quantizing BioGPT to run efficiently on edge devices while maintaining medical accuracy
• Seamless switching: Creating sub-second network detection and failover logic between cloud and local processing
• Multilingual complexity: Handling Hindi language support in online mode while maintaining English-only capability offline
• Memory constraints: Balancing model size with performance on resource-limited devices
• Medical accuracy: Ensuring reliable health information across both high-powered cloud models and smaller local models

Accomplishments that I'm proud of

• Built the world's first truly offline-capable medical AI assistant
• Achieved <10 second response times online and <20 seconds offline
• Successfully integrated multiple AWS services (Bedrock, Polly, Textract) with local inference
• Created automatic network-aware routing that switches modes transparently
• Developed a voice-first interface serving 600+ million Hindi speakers
• Implemented RAG with medical datasets for evidence-based responses
• Built a privacy-conscious system where offline data never leaves the device

What I learned

• Edge computing is crucial for healthcare accessibility in underserved areas
• The importance of graceful degradation in AI systems serving critical needs
• Quantized models can maintain surprising accuracy while being deployment-friendly
• Voice interfaces dramatically improve accessibility for elderly and illiterate populations
• Building for offline-first scenarios requires rethinking traditional cloud-native architectures
• Medical AI must balance sophistication with reliability and speed

What's next for MedAssist AI

• IoT Integration: Connect with medical devices (blood pressure monitors, pulse oximeters) for contextual health data
• 5G Optimization: Leverage 5G networks for real-time telemedicine consultations with specialists
• Expanded Languages: Add support for Bengali, Tamil, and other regional languages
• Clinical Validation: Partner with healthcare providers for real-world testing and accuracy validation
• Mobile App: Develop iOS/Android apps for wider accessibility
• Symptom Tracking: Add longitudinal health monitoring and trend analysis
• Emergency Mode: Implement critical condition detection with automatic emergency contact alerts
• Offline Vector Search: Add local RAG capabilities for offline mode using compressed medical knowledge bases

Shared Medical RAG (Cloud)

• Push anonymized symptom-treatment-outcome data to a global vector database
• Doctors across hospitals can search similar cases in real-time
• Enables nationwide learning and second-opinion support

Regional Language Expansion (Offline)

• Add Marathi, Tamil, Kannada, Bengali support, even when offline
• Helps patients & local doctors communicate better in their native languages
• Enables healthcare accessibility in linguistically diverse regions

Smart Offline Health Memory

• Store past records locally (anonymized, no personal identifiers) to assist in repeat visits
• Fully usable in disconnected rural hospitals and health camps
• Provides continuity of care without requiring internet connectivity

Built With

• amazon-web-services
• bedrock
• biogpt
• claude
• css
• easyocr
• edge
• faster-whisper
• flask
• glow-tts
• html
• javascript
• llama.cpp
• local-llm
• openai
• pinecone
• polly
• python
• react
• textract