MediAI

Your pocket clinical co-pilot, powered by multi-agent AI.

Comment

Inspiration

Modern healthcare systems face severe bottlenecks: long emergency room wait times, geographical isolation, language barriers, and a lack of immediate, reliable triage tools. Often, patients with critical conditions delay seeking care, while those with minor issues overcrowd clinics.

We built MediAI to act as a 24/7 pocket clinical co-pilot. By combining multi-agent clinical AI orchestration, local semantic medical search, visual wound analysis, and voice-assisted accessibility into a hybrid mobile-web app, we aim to democratize access to immediate, grounded, and life-saving clinical intelligence.

What it does

MediAI is a comprehensive, production-ready healthcare accessibility platform that offers:

  • Intelligent Symptom Checker: A multi-agent orchestrator that extracts clinical concepts, scores urgency, and translates regional dialects/languages.
  • Vision Lab (Wound Analyzer): Zero-shot multimodal image analysis that scans for vascular redness and purulent discharge to detect localized skin infection risks.
  • Voice-First Diagnostics: Fully integrated Whisper-powered voice input for hands-free clinical descriptions, ideal for visually or physically impaired users.
  • Local Medical RAG Engine: A localized knowledge retrieval database containing official WHO and CDC guidelines, ensuring all recommendations are highly grounded and completely hallucination-free.
  • Automatic SOS & Doctor Match: Connects patients to nearest clinics using geocoding APIs and dispatches live emergency alerts (symptoms and GPS coordinates) directly to emergency responders via Telegram.
  • Medication Tracker: Pill scheduling and interactive digital logs with persistent Cloud database syncing.

How we built it

We designed a clean, modular, and robust client-server architecture:

  • Frontend (Mobile & Web): Built using React 19, Vite 8, Tailwind CSS, and Framer Motion for a premium, responsive glassmorphism UI. We used CapacitorJS to compile the web application into a native Android app, with custom configurations for system audio recording, cleartext local endpoints, and responsive mobile viewport scaling.
  • Backend Server: Engineered with FastAPI (Python), utilizing high-performance asynchronous endpoints and CORS middleware.
  • Machine Learning Orchestration:
    • Clinical Entity Extraction: Specialized BioClinicalBERT tokenizers for clinical entity recognition.
    • Clinical Triage Triage: PubMedBERT text classifiers to evaluate emergency levels and patient risk.
    • Vector Search & RAG: Local ChromaDB Persistent Client coupled with BAAI/bge-large-en-v1.5 embeddings for highly accurate, contextual, and hallucination-free retrieval.
    • Reasoning LLM: Mistral-7B via API for medical reasoning, diagnostics synthesis, and multilingual translation.
    • Multimodal Vision: CLIP (ViT-B/32) paired with custom RGB pixel-density scanning to assess infected open wounds and rashes.
    • Audio Transcription: Whisper Large v3 serverless inference.
  • Database & Storage: Fully integrated with Supabase PostgreSQL for robust authentication, secure doctor-consultation logging, medication CRUD operations, and Supabase Object Storage bucket mapping for clinical images.
  • Integrations: Geoapify API for geolocation mapping and Telegram Bot API for dispatching live emergency telemetry payloads.

Challenges we ran into

  1. Capacitor Native Bridge Networking: When compiling the application for Android, we encountered connection failures due to hardcoded localhost (127.0.0.1) URLs which are unreachable from the Android Emulator network space. We solved this by designing a dynamic environment router mapping (VITE_API_URL) and configuring the Android manifest to support secure cleartext traffic.
  2. DNS Subdomain Deprecations: During testing, the legacy Hugging Face serverless endpoint (api-inference.huggingface.co) was decommissioned, breaking the entire model inference pipeline. We dynamically bypassed this by upgrading our entire Python backend to use the active router.huggingface.co/hf-inference routing gateway.
  3. Low-Latency Voice Fallback: Capturing raw mic audio from mobile webviews and processing it through cloud APIs often introduces latency. We developed a dual-path voice engine that dynamically falls back to native Web Speech API if the backend transcription endpoint encounters bandwidth or network constraints.

Accomplishments that we're proud of

  • Developing a 100% functional multi-agent clinical orchestrator that handles regional languages (such as Hinglish) and provides localized, safe, and highly accurate recommendations in under 4 seconds.
  • Reconciling advanced cloud-based neural networks with native Android physical hardware features (Microphone permissions, storage cameras, and network channels).
  • Writing a production-ready RAG architecture that automatically initializes and rebuilds its local index from scratch on server startup, guaranteeing highly reliable offline capabilities.

What we learned

  • We gained deep insights into clinical natural language processing (NLP) and the nuances of deploying domain-specific neural networks.
  • We mastered the CapacitorJS compilation pipeline, learning how to successfully manage Android permissions, Gradle configurations, and Android Studio asset generation.
  • We learned the critical importance of designing fallback architectures (e.g., using local RGB image analysis when deep learning vision models are offline) to maintain safety in mission-critical applications like healthcare.

What's next for MediAI

  • On-Device Edge ML: Porting lightweight clinical models directly into the mobile bundle using ONNX Runtime Web or TensorFlow Lite to enable 100% offline triage when cell service is completely unavailable.
  • Wearable Integration: Syncing real-time heart rate, blood oxygen levels, and biometric data from Apple HealthKit and Android Health Connect to automatically alert the backend during cardiac anomalies.
  • FDA-Approved Guideline Integration: Expanding the RAG database to ingest broader clinical libraries, including PubMed articles, pharmaceutical databases, and direct regional hospital directory lists.

Built With

Share this project:

Updates

Submission history