Vitalis

Bridging the Global Medical Consultation Gap with Multimodal AI

Comment

Inspiration

Vitalis was inspired by a structural imbalance in modern healthcare systems: patients wait, doctors rush. In many hospitals, consultation time is consumed by repetitive, low-complexity questions, while patients experience long wait times, travel burdens, and anxiety before receiving even preliminary guidance. At the same time, hospitals already possess rich structured medical records (FHIR), yet these records are rarely leveraged in real-time patient-facing tools.

Existing chatbot systems fall short because medical consultation is inherently multimodal and context-dependent. Patients describe symptoms imperfectly. Skin conditions require visual inspection. Elderly patients prefer speaking over typing. Medical advice must incorporate longitudinal patient history.

Vitalis was designed as an AI-powered pre-consultation layer — not a replacement for physicians, but an intelligent triage and support system that integrates reasoning, dermatology image analysis, medical speech recognition, and patient-specific record retrieval into one cohesive workflow.

What it does

Vitalis is an AI medical consultation agent that provides:

  • Context-aware medical Q&A powered by a clinically aligned LLM (MedGemma).
  • Dermatology image classification using Derm Foundation embeddings + Logistic Regression (8 pediatric skin conditions, 88.7% test accuracy).
  • Medical speech-to-text transcription for elderly or accessibility-focused users.
  • Retrieval-Augmented Generation (RAG) over patient-specific FHIR records for grounded historical queries.
  • Emergency symptom detection and escalation before response generation.
  • Strict privacy isolation — only authenticated hospital patients can access their own records.

It supports three query types:

  1. General medical advice
  2. Personal medical history lookup
  3. Multimodal consultation (text + image and/or audio)

Key projected impact:

  • 30–40% physician time reallocated to complex cases
  • History retrieval reduced from 5–10 minutes → under 2 seconds
  • Up to 8 staff-hours saved per department per day
  • 24/7 automated triage with emergency escalation

How we built it

Vitalis is architected as a layered, production-ready system.

At the backend, FastAPI provides an async REST interface with JWT authentication, rate limiting (20 req/min, 100 req/hr), SHA256 file integrity validation, and security middleware. MongoDB (Motor async driver) stores conversations and sessions. Uploaded files are validated by MIME detection and size constraints (10MB images, 50MB audio). The backend architecture follows a layered microservice pattern with API gateway, router, service, and external service layers :contentReference[oaicite:0]{index=0}.

AI orchestration is handled by LangGraph. The structured workflow includes:

Input Router → Speech/Image Processing → MedGemma Reasoning → RAG Tool → Safety Check → Response :contentReference[oaicite:1]{index=1}

For multimodal intelligence, we integrate three Google HAI-DEF models:

  • Derm Foundation as a frozen 6,144-dimensional feature extractor feeding a Logistic Regression classifier.
  • MedASR for domain-specialized medical speech recognition.
  • MedGemma-4B-IT (quantized Q4_K_S, ~2.4GB) running locally via Ollama for reasoning.

The dermatology model was trained on 8,000 images (1,000 per class, 70/15/15 split). We deliberately chose an embedding-then-classify strategy instead of fine-tuning the full model to minimize deployment costs while preserving clinically learned representations.

For RAG, FHIR bundles are parsed into narrative clinical episodes, embedded using nomic-embed-text, and stored in per-patient FAISS indices. A year-aware pre-filter narrows retrieval scope before cosine similarity ranking — enabling queries like “What was my blood pressure in 2010?” with grounded outputs.

A MedicalGuardrails layer runs on every response, enforcing emergency detection, blocking definitive diagnoses, preventing cross-patient leakage, and appending disclaimers.

The system architecture and consultation goals are detailed in the project overview :contentReference[oaicite:2]{index=2}.

Challenges we ran into

One of the core challenges was aligning multimodal signals into a coherent reasoning pipeline. Image classification outputs probabilistic predictions, speech transcription introduces noise, and patient history retrieval can return partially relevant records. Designing a deterministic LangGraph state machine to handle these without hallucination required explicit routing logic and multiple validation layers.

Emergency detection posed another challenge. If implemented after LLM generation, unsafe responses could slip through. We redesigned the flow so that emergency keyword detection occurs before MedGemma reasoning, ensuring immediate escalation.

Latency optimization was also critical. MedGemma cold starts can take 5–10 seconds. We mitigated this with model warm-up strategies and async orchestration. Ensuring RAG queries remained under one second required pre-indexed FAISS vectors and year-aware filtering.

Finally, privacy constraints shaped major design decisions. Running MedGemma locally via Ollama ensured no patient data leaves hospital infrastructure — but required careful memory and hardware optimization.

Accomplishments that we're proud of

We are particularly proud of building a clinically aligned multimodal system that is:

  • Grounded: Patient-history queries are retrieved, not hallucinated.
  • Private: On-premise LLM inference avoids external API transmission.
  • Efficient: 88.7% test accuracy in dermatology classification with CPU-friendly inference (<4 seconds).
  • Safe: Emergency detection across 10+ symptom categories with escalation before generation.
  • Structured: Deterministic LangGraph state graph with explicit safety gates.

We also successfully maintained production-level backend rigor — JWT session validation, layered security headers, rate limiting, file hashing, and background cleanup scheduling — rather than treating this as a prototype-only AI demo :contentReference[oaicite:3]{index=3}.

What we learned

We learned that medical AI is fundamentally a systems engineering problem, not just a modeling problem. Model quality alone is insufficient without orchestration, guardrails, retrieval grounding, and strict privacy boundaries.

We also learned that specialized domain models significantly outperform general-purpose substitutes in clinical contexts. General ASR models misrecognize drug names. General vision models lack dermatology-specific feature representations. General LLMs hallucinate medical terminology more frequently.

Architecturally, we learned that embedding-then-classify strategies can offer an excellent trade-off between performance and deployability for hospitals with limited hardware.

Most importantly, we learned that safety must be proactive and architectural — not reactive and post-hoc.

What's next for Vitalis

The next phase of Vitalis includes:

  • Expanding beyond dermatology to cardiology and pulmonology models.
  • Structured vital-sign integration for real-time triage scoring.
  • Federated fine-tuning pipelines for hospital-specific adaptation.
  • Multi-language clinical consultation support.
  • EHR system API integration for real-time hospital deployment.
  • Risk stratification dashboards for physicians.
  • Clinical validation studies with real-world pilot hospitals.

Our long-term vision is for Vitalis to become a hospital-native AI consultation layer that safely scales physician expertise — not by replacing doctors, but by amplifying their time and focus where it matters most.

Built With

  • aiofiles
  • audio
  • faiss
  • fastapi
  • fhir-(fast-healthcare-interoperability-resources)
  • google-derm-foundation
  • google-medasr
  • google-medgemma-4b-it
  • gunicorn
  • jwt-(python-jose)
  • langgraph
  • mediarecorder-api
  • mongodb
  • motor-(async-mongodb-driver)
  • nomic-embed-text
  • numpy
  • ollama
  • pandas
  • pydantic
  • python
  • python-magic
  • react
  • scikit-learn
  • typescript
  • uvicorn
  • vite
  • web
Share this project:

Updates

Submission history