Code Trek

NetraAI: AI telemedicine with 5 ML models for anemia, cataracts, diabetic retinopathy & more. HIPAA-compliant. Healthcare reimagined.

Inspiration

Over 1.62 billion people suffer from anemia globally. Cataracts blind 65 million people. Diabetic retinopathy affects 103 million diabetics. Most lack access to affordable diagnostics.

The vision: What if a smartphone could replace a blood test? NetraAI brings clinical-grade AI screening to anyone with a phone.

What It Does

NetraAI is a telemedicine and AI diagnostics platform with three portals (Patient, Doctor, Admin) running 5 AI models.

AI Models

Model	Input	Output	Accuracy
Anemia	Eyelid photo	Anemic/Normal + Hb level	~90%
Cataract	Fundus image	Normal/Early/Advanced	95.03%
Diabetic Retinopathy	Retinal image	Grade 0-4	~95%
Parkinson's	Voice recording	Healthy/Parkinson's	80.9%
Mental Health	Voice note	Depression/Anxiety/Stress	Multimodal

AI Models - How They Work

Anemia Detection Model: The CNN analyzes color and texture patterns in the conjunctiva by passing the image through three convolutional layers that progressively extract features from edges to complex pallor patterns. The model compares these features against learned patterns from thousands of labeled anemic and non-anemic eyelids to determine hemoglobin levels.

Cataract Detection Model: The Swin Transformer divides the retinal image into small patches and applies shifted window attention mechanisms to understand both local details like lens opacity and global context of lens structure. It then classifies severity by comparing learned features from normal, early, and advanced cataract cases.

Diabetic Retinopathy Model: EfficientNet-B5 processes high-resolution retinal images through a compound-scaled architecture that balances depth, width, and resolution simultaneously. It identifies microaneurysms, hemorrhages, and neovascularization patterns using a dual-head output that provides both severity grading and uncertainty estimation.

Parkinson's Voice Model: The system extracts 33 acoustic biomarkers including jitter (pitch instability), shimmer (amplitude variation), and MFCCs from voice recordings using librosa and parselmouth libraries. LightGBM then builds an ensemble of decision trees to identify patterns characteristic of Parkinson's-related vocal tremors and dysphonia.

Mental Health Model: Three independent models analyze different modalities - Whisper transcribes speech to text for MentalBERT analysis, librosa extracts prosodic features like speech rate and pitch variability, and DeepFace optionally analyzes facial expressions. A weighted fusion combines these signals to produce depression, anxiety, and stress scores.

Patient Portal Features

Video Consultations & Telemedicine

LiveKit WebRTC with end-to-end encryption for secure video calls
AI Scribe: Listens to conversation and generates structured SOAP notes in real-time
Live Translation: Bidirectional speech translation for 6 Indian languages
Collaborative Whiteboard: tldraw-powered drawing and annotation synchronized in real-time
Session recording with secure encrypted storage

Appointment Booking & Management

Book/Cancel/Reschedule with real-time availability checking
Find doctors by specialty, language, rating, and experience
Waiting room with estimated wait time and live status indicator
Automated reminders: 24 hours before and 1 hour before appointment

Medical History & Documents

Timeline view of all scans, appointments, prescriptions, lab results
Upload/download documents (PDF, images, DOCX) with auto-categorization
Semantic search using vector embeddings for natural language queries
FHIR export (HL7 FHIR R4 compliant JSON for EHR interoperability)

Lab Analyzer & Insurance

Upload lab report photo → OCR extracts vitals automatically
Key metrics: hemoglobin, blood sugar, cholesterol, creatinine, thyroid
Insurance verification showing policy limits, deductible, co-pay, covered procedures
Claim submission with AI pre-filled forms and status tracking

Health Risk Assessment

Multi-step questionnaire covering age, BMI, family history, lifestyle, symptoms
Risk scores for CVD, diabetes, hypertension, anemia, cancer
Personalized recommendations including lifestyle changes and screening schedules

Chronic Disease Tracker

Log daily vitals: blood pressure, blood sugar, weight, heart rate, SpO2
Line charts showing trends over 7, 30, or 90 days
Real-time alerts for out-of-range values with action recommendations
CSV export for sharing with doctors

Medication Reminders & AI Nurse

Add medications with name, dosage, frequency, and schedule
AI Nurse: Daily automated phone calls via Twilio checking adherence and side effects
Severe side effects trigger critical doctor alerts with full call transcript
Missed call retry after 30 minutes, escalation after 3 missed calls

AR Exercise Sessions

MediaPipe pose tracking with 33 body landmarks in real-time
Rep counting and form accuracy feedback using joint angle calculations
Exercises: squats, lunges, push-ups, jumping jacks
Session analytics saved to history (reps completed, form accuracy, time)

Nearby Hospitals & Maps

Leaflet map with browser geolocation showing nearby facilities
Filter by service type: emergency, lab, ophthalmology, pharmacy
Distance calculation and directions integration with Google Maps

Messages & AI Chatbot

Real-time chat with doctors including read receipts and file attachments
DeepSeek-R1 14B chatbot for 24/7 medical questions (zero API cost)
Chat history persisted and searchable

Achievements & Gamification

XP earned for scans, appointments, medication adherence, exercise
Badges for milestones, streaks for consecutive days of health activity
Optional leaderboard and reward redemption system

Profile Settings & Multi-language

Personal info, family members (dependents under same account), emergency contact
Theme: light/dark/system, font size adjustment for accessibility
Language switcher: 6 languages with instant UI update
MFA via TOTP authenticator app

Doctor Portal Features

Dashboard Overview

Stats cards: Today's appointments, scans to review, revenue, rating
Patient risk distribution donut chart
Revenue insights bar chart (weekly)
Scans to review panel with confidence bars
Today's consultations with join buttons

Appointment Management

Table view with start/complete/cancel actions
Calendar view with drag-drop rescheduling
Bulk actions for reminders and exports
Waiting room queue management

Scan Review System

Full image display with Grad-CAM heatmap overlay
AI prediction details with confidence scores
Add clinical notes or override AI prediction
Request follow-up scan, share results with patient

Schedule Management

Weekly working hours grid (Monday to Sunday)
Consultation types: Video, In-person, or Both
Buffer time between appointments
Date blocking for holidays, leave, conferences

Automated Follow-up Templates

Triggers: Upcoming appointment, Appointment completed, No-show
Placeholders: {{patient_name}}, {{doctor_name}}, {{appointment_date}}
Delays: 24h before, 1h before, 1h after, 1 day after, 1 week after
Channels: In-app, Email, SMS

Patient Timeline Management

Complete chronological health history for each patient
All scans with AI predictions and XAI heatmaps
Add clinical notes at any point with rich text formatting
Filter by event type and date range

AI Nurse Escalations

Critical side effects from daily AI Nurse calls appear as alerts
Color-coded severity: Red (immediate), Orange (follow-up), Yellow (monitor)
Full call transcript viewer with audio playback
Direct patient contact buttons from alert

Revenue & Withdrawals

Razorpay payment integration for consultation fees
Earnings breakdown by month with bar chart trends
Withdrawal/payout requests to bank account
Transaction history with receipt links

Ratings & Reviews

All patient reviews with star ratings (1-5) and comments
Response to review option with public reply
Rating trends chart over time
Flag inappropriate content for admin review

Referral Management

Refer patients to other specialists within the platform
Track status: Pending, Accepted, Scheduled, Completed, Declined
Attach clinical notes and relevant scan results
Referral history with analytics

Admin Portal Features

Dashboard & Analytics

Stats cards with growth indicators: Patients (+12%), Doctors (+5%), Appointments (+18%), AI Scans (+24%)
Platform growth area chart (users and scans over 6 months)
Weekly appointments bar chart with comparison to previous week
Attention section: pending doctor approvals, reported issues, system updates

Patient Management

Full list of all registered patients with search and filter
View patient detail with complete profile, scans, appointments, documents
Suspend/Activate account toggle, Delete account with confirmation
Bulk actions for selecting multiple patients

Doctor Management & Verification

List of all registered doctors with verification status badges
Review pending applications with uploaded credentials
Approve/Reject with optional comments and email notification
Verify doctor button enables patient search visibility

Appointment Oversight

All platform appointments with full details
Filter by status, date range, doctor, patient
Cancel or reschedule appointments with reason recorded
Audit log of all appointment changes

Scan Management

All AI scans across all patients
Filter by scan type, result, date range, patient
View scan images with AI results and XAI heatmaps
Flag scans for quality review, triggers compliance workflow

Audit Logs

Complete audit trail of all admin and system actions
Fields: timestamp, user, action type, affected resource, IP address, user agent
Filter by user, action type, date range, affected resource
Export logs as CSV for compliance audits

FDA APM Monitoring

Model performance charts: accuracy, sensitivity, specificity over time
Confidence calibration for each AI model
Alert thresholds for accuracy degradation (red line at 5% drop)
Incident reporting for model failures with FDA-mandated forms

IEC 62304 Traceability

Software lifecycle traceability matrix for medical device compliance
Maps: Requirements → Design → Implementation → Testing
Color-coded status for each traceability link
Export as PDF for regulatory submission

SOC 2 Evidence Collection

Control categories: Security, Availability, Processing Integrity, Confidentiality, Privacy
Evidence upload and management with automated collection
Compliance score cards with percentage completion
Alert badges for non-compliant items

FHIR Resource Manager

Manage HL7 FHIR R4 resources: Patient, Observation, Condition, MedicationRequest, DiagnosticReport
Create, Read, Update, Delete operations with JSON validation
FHIR-compliant export for EHR interoperability
Resource editor with schema validation

System Health Monitoring

Real-time status of all backend services
Response time, error rate (last hour/24 hours)
Memory and CPU usage where available
Service restart buttons for each service

Security Configuration

Active sessions management with remote logout
Failed login monitoring with IP addresses and timestamps
IP blocklist management with duration settings
Rate limit configuration per endpoint and user role

Platform Configuration

Maintenance mode toggle with custom message
Feature flags to enable/disable specific features per portal
Email/SMS provider configuration (API keys, sender details)
Payment gateway settings (Razorpay keys)

Epidemic Radar

Geographic heatmap of disease prevalence based on scan results
Cluster detection for potential outbreak locations
Filter by disease type, date range, geographic region
Export anonymized aggregate data for public health reporting

Content Management

Blogs: create, edit, publish health education posts with rich text editor
Newsletter: compose, send to patient segments, track open rates
Reviews: moderate patient reviews of doctors (approve/reject/flag)
Contact messages: respond to public form submissions, assign to team members

Team Management

Manage admin team members with role assignment
Roles: Super Admin, Admin, Support
Invite new team members via email with role-specific permissions
Audit log of team member actions

Custom Reports

Generate reports: patient activity, scan statistics, revenue, compliance
Schedule automated report delivery (daily/weekly/monthly)
Export as PDF or CSV with customizable columns

- Report history with download links

Challenges

Syntax corruption - Find-and-replace broke 50+ Python files, fixed with regex repair scripts
Threshold optimization - Cataract model default 0.5 gave 85% sensitivity; optimized to 0.20 for 96% sensitivity with 90.2% specificity
WebRTC in Docker - Required extensive STUN/TURN configuration and proper CORS headers for LiveKit
Parselmouth on Windows - Native C extension required full Python Docker image for voice analysis
DeepSeek cold start - 14B model takes 10-30 seconds to load; fixed with warmup script during server startup
PHI scrubbing - Custom function redacts 15 categories of PHI from Sentry events before transmission

Accomplishments

95.03% cataract detection accuracy with 96% sensitivity (clinically meaningful threshold)
6-language support covering 1.2 billion people across India's major language groups
Real-time AI scribe generating SOAP notes during live video consultations
Autonomous AI Nurse for daily medication adherence calls with side effect detection
Full compliance architecture: FDA APM, IEC 62304, SOC 2, HIPAA, FHIR R4
Explainable AI with Grad-CAM heatmaps on every model prediction
Zero API cost AI with local Whisper, MentalBERT, DeepSeek-R1 (14B)

Learnings

Medical AI threshold tuning matters more than raw accuracy for clinical safety
Multimodal fusion (text + acoustic + facial) beats any single modality for mental health
Compliance must be built into architecture from day one, not added later
Gamification increased simulated patient engagement by 40% in user testing
Language support reduced task completion time by 60% for non-English speakers