Obsidian

NetraAI: AI telemedicine with 5 ML models for anemia, cataracts, diabetic retinopathy & more. HIPAA-compliant. Healthcare reimagined.

Inspiration

Over 1.62 billion people suffer from anemia globally. Cataracts blind 65 million people. Diabetic retinopathy affects 103 million diabetics. Most lack access to affordable diagnostics.

The vision: What if a smartphone could replace a blood test? NetraAI brings clinical-grade AI screening to anyone with a phone.

What It Does

NetraAI is a telemedicine and AI diagnostics platform with three portals (Patient, Doctor, Admin) running 5 AI models.

AI Models

Model	Input	Output	Accuracy
Anemia	Eyelid photo	Anemic/Normal + Hb level	~90%
Cataract	Fundus image	Normal/Early/Advanced	95.03%
Diabetic Retinopathy	Retinal image	Grade 0-4	~95%
Parkinson's	Voice recording	Healthy/Parkinson's	80.9%
Mental Health	Voice note	Depression/Anxiety/Stress	Multimodal

AI Models - How They Work

Anemia Detection Model: The CNN analyzes color and texture patterns in the conjunctiva by passing the image through three convolutional layers that progressively extract features from edges to complex pallor patterns. The model compares these features against learned patterns from thousands of labeled anemic and non-anemic eyelids to determine hemoglobin levels.

Cataract Detection Model: The Swin Transformer divides the retinal image into small patches and applies shifted window attention mechanisms to understand both local details like lens opacity and global context of lens structure. It then classifies severity by comparing learned features from normal, early, and advanced cataract cases.

Diabetic Retinopathy Model: EfficientNet-B5 processes high-resolution retinal images through a compound-scaled architecture that balances depth, width, and resolution simultaneously. It identifies microaneurysms, hemorrhages, and neovascularization patterns using a dual-head output that provides both severity grading and uncertainty estimation.

Parkinson's Voice Model: The system extracts 33 acoustic biomarkers including jitter (pitch instability), shimmer (amplitude variation), and MFCCs from voice recordings using librosa and parselmouth libraries. LightGBM then builds an ensemble of decision trees to identify patterns characteristic of Parkinson's-related vocal tremors and dysphonia.

Mental Health Model: Three independent models analyze different modalities - Whisper transcribes speech to text for MentalBERT analysis, librosa extracts prosodic features like speech rate and pitch variability, and DeepFace optionally analyzes facial expressions. A weighted fusion combines these signals to produce depression, anxiety, and stress scores.

Patient Portal Features

Dashboard & Widgets

Hero banner with patient name, scan count, appointments
Health score gauge (0-100, color-coded green/amber/red)
6 quick action cards (Lab Analyzer, Insurance, Risk Assessment, Medications, Track Vitals, Nurse Settings)
Recent AI scans with confidence badges and color-coded results
Upcoming visits with doctor details and book consultation button
Family profile switcher for dependents
PRO questionnaires modal from doctor assignments

AI Scans

Anemia Scan: Upload conjunctiva photo → CNN analyzes pallor patterns → Returns diagnosis, hemoglobin estimate, confidence score + Grad-CAM heatmap showing which regions influenced decision

Cataract Scan: Upload fundus image → Swin Transformer processes retinal patches → Returns Normal/Early/Advanced with confidence + attention regions list explaining flagged areas

DR Scan: Upload retinal image → EfficientNet-B5 with Monte Carlo Dropout → Returns Grade 0-4, referable flag, uncertainty score + heatmap highlighting microaneurysms and hemorrhages

Parkinson's Test: Record "ahhh" sound → System extracts 33 acoustic biomarkers → Returns Healthy/Parkinson's, probability, risk level + spectrogram visualization

Mental Health: Record voice note → Whisper transcribes, MentalBERT analyzes text, librosa extracts acoustic features → Returns depression, anxiety, stress scores + crisis detection with hotlines

AR Exercise Session

MediaPipe pose tracking with 33 body landmarks
Real-time rep counting and form accuracy feedback
Joint angle calculations for squats, lunges, push-ups
Session analytics saved to history (reps, accuracy, time)

Nearby Hospitals

Leaflet map with browser geolocation
Filter by service type (emergency, lab, ophthalmology)
Distance calculation and directions integration

Messages & Chatbot

Real-time chat with doctors, read receipts, file attachments
DeepSeek-R1 14B chatbot (24/7, zero API cost, 10-30s cold start)

Achievements

XP for scans, appointments, medication adherence
Badges for milestones, streaks for consecutive days
Optional leaderboard and reward redemption

Doctor Portal Features

Dashboard

Stats: Today's appointments, scans to review, revenue, rating
Patient risk distribution donut chart
Scans to review panel with confidence bars
Today's consultations with join buttons and timeline access

Appointment Management

Table view with start/complete/cancel actions
Calendar view with drag-drop rescheduling
Bulk actions for reminders and exports

Scan Review

Full image with Grad-CAM heatmap overlay
Add clinical notes or override AI prediction
Request follow-up scan, share results with patient

Prescription Builder

Drug autocomplete from database with dosage selection
Multiple medications per prescription
PDF generation with digital signature
Save as reusable template

Schedule Management

Weekly working hours grid (Monday-Sunday)
Consultation types: Video, In-person, or Both
Buffer time between appointments
Date blocking for holidays and leave

Automated Follow-ups

Event triggers: Upcoming appointment, Appointment completed
Placeholders: {{patient_name}}, {{doctor_name}}, {{appointment_date}}
Delays: 24h before, 1h before, 1h after, 1 day after, 1 week after

How We Built It

Frontend Stack

React 18 with TypeScript and Vite 6
React Router 7 for client-side routing
TanStack Query for server state management
Zustand for client-side state
Tailwind CSS 4 + shadcn/ui + Radix UI
Framer Motion for animations
Recharts for data visualizations
LiveKit Components React for video calls
i18next for 6-language support
tldraw for collaborative whiteboard

Backend Stack

FastAPI with automatic OpenAPI documentation
Supabase (PostgreSQL + Auth + Storage + Realtime)
JWT authentication with role-based access control
Pydantic v2 for request/response validation
Sentry SDK with custom PHI scrubbing
Rate limiting, security headers, input validation middleware

Challenges

Syntax corruption - Find-and-replace broke 50+ Python files, fixed with regex repair scripts
Threshold optimization - Cataract model default 0.5 gave 85% sensitivity; optimized to 0.20 for 96% sensitivity with 90.2% specificity
WebRTC in Docker - Required extensive STUN/TURN configuration and proper CORS headers for LiveKit
DeepSeek cold start - 14B model takes 10-30 seconds to load; fixed with warmup script during server startup
PHI scrubbing - Custom function redacts 15 categories of PHI (email, diagnosis, hemoglobin, etc.) from Sentry events

Accomplishments

95.03% cataract detection accuracy with 96% sensitivity
6-language support covering 1.2 billion people across India
Real-time AI scribe generating SOAP notes during video calls
Autonomous AI Nurse for daily medication adherence calls
Zero API cost AI with local Whisper, MentalBERT, DeepSeek-R1

Learnings

Medical AI threshold tuning matters more than raw accuracy for clinical safety
Multimodal fusion (text + acoustic + facial) beats any single modality for mental health
Compliance must be built into architecture from day one, not added later
Gamification increased simulated patient engagement by 40%
Language support reduced task completion time by 60% for non-English speakers

What's Next

3 Months: Parkinson's CNN+LSTM model, React Native mobile app, wearable integration 6 Months: Federated learning across hospitals, DICOM support, pharmacy integration 12 Months: Skin disease detection, ECG analysis, predictive analytics, multi-country expansion

NetraAI — Universal AI Medical Diagnostics