MSCS - MultiScope Clinical Support

MSCS implements multimodal AI agent to make instruction videos interactive and streamline prescription with a CPIC-confirmed pharmacogenomic safety layer that lowers epistasis-related ADR risk.

Comment

Inspiration

Two recurring pain points sparked MSCS:

  • After-visit confusion: Patients leave with great intentions but hazy memories. Staff drown in callback loops for simple questions (“When do I remove the bandage?”).
  • Genetics-related medication risk: Clinicians know pharmacogenomics matters (e.g., warfarin, some statins), but guidance is fragmented or trapped in static PDFs at the moment decisions are made.

--> We asked: What if a clinic’s own voice scaled beyond the room—and safety checks were transparent, guideline-anchored, and right where prescribing happens? MSCS is our answer: Video Copilot for interactive, cited patient guidance + a CPIC-confirmed pharmacogenomic support layer that boosts prescribing confidence and reduces epistasis-related adverse events.

What it does

  • Video Copilot: Turns doctor instruction videos into interactive, timestamp-cited Q&A inside the clinic portal. Patients ask in plain language (“When do I remove the bandage?”) and get answers grounded in the clinic’s own content—no external searching.

  • CPIC-confirmed prescribing support: A built-in pharmacogenomic layer that runs a “genetic compatibility check” for high-impact meds (e.g., warfarin, select statins). It presents dose-adjust / consider alternative / avoid with clear CPIC citations, accounting for phenoconversion (inhibitors/inducers) to reduce epistasis-related risk.

How we built it

AI-Powered Interactive Video Pipeline: Upload → auto-transcribe (word-level timestamps) → auto-chapter → vector index → retrieval-augmented generation constrained to clinic sources → answer cards with timestamp jump-links, captions, transcripts, and english support → Deployed on AWS EC2.

Pipeline (PGx support): VCF ingest → variant normalization → star-allele/activity score calling → phenotype derivation → phenoconversion adjustment from live med list → CPIC rules engine → guideline-cited action card → Dockerized and deployed as an API endpoint via Render.

Architecture: React/TypeScript UI; Dockerized Python FastAPI for future horizontal scaling of core features; Data pipeline with Polars and BS4 data processing from credible sources.

Challenges we ran into

  • Hallucinations in Q&A: Solved with strict retrieval constraints and must-cite policy (timestamped clinic sources).
  • Variant normalization & star-alleles: Resolved by versioned normalization with explainable genotype → phenotype lineage.
  • Algorithm for risk scoring: Come up with weighted score for each factor then retrain the dataset with Xg-boost to deliver a lightweight but high-accuary model complemented with LLM model (Llama 3.3) second-layer support.

Accomplishments that we're proud of

  • Clinic-specific, cited answers that keep patients in-portal and reduce callbacks.
  • A CPIC-confirmed pharmacogenomic layer that boosts prescriber confidence and surfaces why (genes, phenotype, citation) in one glance.
  • Accessibility by default: captions, transcripts, multilingual toggles aligned with best practices.

What we learned

  • Grounded QA > generic LLMs: Retrieval-augmented generation (RAG) constrained to clinic-authored sources cut hallucinations dramatically.
  • Auto-chaptering needs multimodal cues: Text-only topic segmentation missed “show-and-tell” transitions. Adding acoustic energy + slide-change heuristics improved boundary by 12% on our labeled set.

What's next for MSCS - MultiScope Clinical Supportas

  • Pilot studies: Quantify reductions in callbacks, faster question resolution, and safer prescribing on a focused drug panel.
  • Broader PGx coverage: Add more CPIC gene–drug pairs and strengthen DDGI scoring.
  • Multilingual support: Support more languages to increase customer outreach
  • More features: Remote AI assistant with rehab exercises --> generate logs and convert into human friendly language using LLM and log into system --> Summarizing for future references Format these in latex

Built With

Share this project:

Updates