Inspiration
Every day in diagnostic centres across India, critical lab reports sit on desks unnoticed. A patient's haemoglobin could be dangerously low. Their blood sugar could be life-threateningly high. But the printed report just waits in a pile while the lab staff attends to the next patient.
The referring doctor never finds out until it's too late.
We built VitalAlert because this is not a rare edge case — it happens hundreds of times every day in small and mid-sized diagnostic centres across India. There is no automated system watching for danger. No one calling the doctor. No safety net.
We wanted to fix that. Completely automatically.
What it does
VitalAlert is an AI-powered diagnostic report analysis and critical alert system for diagnostic centres.
Here is exactly what it does:
Lab staff uploads any diagnostic report as an image or PDF — blood test, X-ray, MRI, ECG, ultrasound, biopsy, or any other report type.
NVIDIA NIM Vision Model reads every image and extracts all test values, findings, and measurements with high precision.
Mistral Large 3 analyzes the extracted values and writes a simple, clear health summary in plain English that any doctor can understand in seconds.
VitalAlert checks every value against medical critical thresholds automatically.
If a dangerous value is detected — the referring doctor gets an instant WhatsApp alert with the patient details, critical findings, AI summary, and suggested action — in under 60 seconds.
If AI confidence is low — manual alert buttons are shown to lab staff so a human can review and trigger the alert themselves.
The clinic owner gets a real-time dashboard showing all reports, critical alerts, acknowledgement status, and top referring doctors — all in one place.
How we built it
Frontend: Pure HTML, CSS, and JavaScript. Three panels — lab staff panel for uploading reports, clinic owner dashboard for analytics, and doctor portal for acknowledging alerts. Fully mobile responsive since doctors open alerts on their phones.
Backend: Python and FastAPI with fully async architecture using Motor for MongoDB operations. Every service runs asynchronously so the app never blocks.
AI Pipeline: Two NVIDIA NIM models working together. First nvidia/llama-3.1-nemotron-nano-vl-8b-v1 reads the report images visually and extracts all values as structured JSON. Then mistralai/mistral-large-3-675b-instruct-2512 takes those values and writes a simple English health summary for the doctor.
PDF Support: PyMuPDF converts every page of a PDF report into a high resolution image which is then sent to the vision model. A 5 page PDF becomes 5 images all analyzed together in one API call.
Database: MongoDB Atlas stores patients, doctors, reports, extracted values, AI analysis, and alert records. MongoDB Change Streams watch for new reports and trigger the alert pipeline automatically the moment a result is saved.
Alerts: Twilio WhatsApp API sends critical alerts to doctors instantly. Alert records are saved with full audit trail — who was notified, when, and whether they acknowledged.
Google Cloud: Google Cloud Agent Builder orchestrates the entire multi-step agent workflow — from file upload to AI analysis to alert dispatch.
Challenges we ran into
Multi-page PDF handling — converting each page to a clear image at the right resolution so the vision model could read handwritten and printed values accurately took significant tuning.
Structuring vision model output — medical reports come in hundreds of different formats. Getting the vision model to consistently return clean structured JSON regardless of report layout required careful prompt engineering.
Confidence scoring — deciding when to auto alert vs when to flag for human review was a critical design decision. Too sensitive and doctors get spammed. Too conservative and dangerous values get missed.
MongoDB Change Streams — keeping the change stream alive and auto-reconnecting on connection drops without crashing the main app required careful async handling.
WhatsApp sandbox limitations — every recipient needs to opt into the Twilio sandbox first which adds friction in a real clinical demo environment.
Accomplishments that we're proud of
Built a fully working end-to-end pipeline — upload report → AI reads it → critical value detected → doctor gets WhatsApp — in under 60 seconds.
Two AI models working together — one sees, one explains. This combination gives both accurate extraction and human-friendly summaries.
Supports every diagnostic report type — blood tests, X-rays, MRIs, ECGs, ultrasounds, biopsies — all in one system.
PDF support with multi-page handling — most real clinic reports come as PDFs. We handle them natively.
Real-world problem — we validated this with actual diagnostic centre owners in Kolkata who confirmed this exact problem exists and costs lives.
Clean, mobile-first doctor portal — doctors can acknowledge alerts, call patients, and send messages all from their phone in seconds.
What we learned
Vision models need very specific prompting to return consistent structured output from medical documents.
Two specialized models beat one general model — separating extraction from explanation gave dramatically better results.
Real-world healthcare problems are simpler than they look — the core of VitalAlert is just: see the value, compare to threshold, send a message. The complexity is in making it reliable.
MongoDB Change Streams are extremely powerful for real-time event-driven architectures — perfect for medical alert systems.
Designing for doctors means designing for people who are busy, stressed, and reading on a small phone screen. Every word in the alert message matters.
What's next for VitalAlert
Production WhatsApp number via Interakt or Wati — Indian WhatsApp Business API providers so no sandbox opt-in needed.
Voice alerts — if doctor does not respond to WhatsApp in 30 minutes, an AI voice call is triggered automatically via Omnidim.
Patient history and trend tracking — detect when values are worsening over multiple visits even if each individual value is not yet critical.
Multi-branch support — one dashboard for clinic owners managing multiple diagnostic centre locations.
Integration with existing lab software — direct API connection to popular Indian lab management systems so reports flow in automatically without manual upload.
ABDM integration — connect with India's Ayushman Bharat Digital Mission health records for seamless patient history.
Expand to 20+ Indian languages — WhatsApp alerts in Hindi, Bengali, Tamil, Telugu so doctors across India can use it natively.
Log in or sign up for Devpost to join the conversation.