-
-
Web Interface
-
Mobile Interface
-
Medical scribe processing audio file
-
Medical scribe report
-
Body scan report 1
-
Body scan report 2
-
Lab analyzer report 1
-
Lab analyzer report 2
-
Lab analyzer report 3
-
Lab analyzer report 4
-
Lab analyzer report pdf
Inspiration
Healthcare is drowning in unstructured data. Doctors spend 10-15 hours per week writing consultation reports. Patients receive lab results they can't interpret without a medical degree. And there's no simple, free way to track body composition over time.
When we saw Elfie's three hackathon challenges, we realized they all share the same root problem: critical health information is trapped in formats that are hard to process -- audio recordings, body images, and PDF lab reports. Elfie already serves 1.5M+ users across 30+ countries with a mission to drive medical impact globally. We set out to build an AI layer that transforms all three data types into clear, actionable health insights -- fitting naturally into Elfie's existing platform.
What it does
Elfie HealthPilot is a suite of three AI-powered tools built on Alibaba Cloud's Qwen model family, delivered as a web app for doctors and a mobile PWA for patients -- sharing a single backend:
** The Medical Scribe** -- Ingests raw consultation audio recordings in EN / FR / AR / VN, transcribes them using Qwen3-ASR-Flash, extracts structured clinical data (symptoms, findings, diagnosis, treatment), and generates a professional SOAP-format clinical report -- all in a single streamed pipeline with live progress updates. A 15-minute consultation becomes a complete clinical document for ~$0.014, and the final report can be downloaded as a polished PDF.
** The Body Scan** -- Takes front and profile body photos along with the user's height, age, and gender, estimates body measurements (shoulder, chest, waist, hip, neck, upper arm, thigh, calf), and calculates body fat percentage using a hybrid approach that combines MediaPipe pose landmarks with Qwen's vision-language analysis and cross-validates against the U.S. Navy formula. Results include a body-type classification and a patient-friendly health summary, exportable as a PDF report.
** The Labs Analyzer** -- Accepts lab report PDFs or photos of printed reports (digital, scanned, or phone-camera captures, in any of the four target languages), extracts and standardizes all lab values, classifies abnormalities by severity using a deterministic rule engine, and generates patient-friendly explanations with recommended next steps. Every result gets a color-coded severity badge (normal / borderline / abnormal / critical), and the final report downloads as a multilingual PDF.
How we built it
We chose a dual-frontend architecture with a single FastAPI backend, deployed on Alibaba Cloud ECS (Singapore region) behind nginx. The backend serves both a desktop web frontend (plain HTML/CSS/JS -- no build step, instant iteration during the hackathon) for doctors, and a mobile PWA frontend (with manifest + service worker for "Add to Home Screen") for patients. A single Python FastAPI codebase powers both, so every improvement lands everywhere at once.
Qwen models power the AI layer:
| Model | Role |
|---|---|
qwen3-asr-flash (via qwen3-asr-toolkit) |
Speech-to-text for the Medical Scribe |
qwen3.5-plus |
Clinical extraction, SOAP report generation, lab value parsing, and patient explanations |
qwen3-vl-plus |
Body composition visual analysis and lab-report OCR for scanned PDFs / photos |
All models are accessed through the OpenAI-compatible DashScope API -- we swapped the base URL to https://dashscope-intl.aliyuncs.com/compatible-mode/v1, changed the model name, and our existing OpenAI client code worked immediately. The async AsyncOpenAI client is shared across all three routes.
Medical Scribe pipeline: Audio upload -> qwen3-asr-toolkit (primary, calls DashScope Qwen3-ASR-Flash via WebSocket with VAD-based smart chunking and parallel API calls -- runs on Alibaba's GPU infrastructure) -> raw transcript -> single combined LLM call that produces both the structured SOAP JSON and the formatted clinical report in one round-trip using custom delimiters (===CLINICAL_JSON_START=== / ===REPORT_START===). This merged call is our main speed win -- it saves a full 3-5 seconds versus two sequential calls. A Server-Sent Events endpoint (/api/scribe/stream) streams step-by-step progress to the frontend so the doctor sees "Transcribing -> Extracting clinical data -> Report ready!" in real time. If the Qwen ASR toolkit is unavailable, we gracefully fall back to local faster-whisper on CPU so the demo never breaks.
Body Scan pipeline: Front + side photos -> MediaPipe BlazePose (33 landmarks, static-image mode, complexity 2) -> pixel-to-cm calibration using the user's known height (head-top landmark -> ankle midpoint) -> circumference estimation via the elliptical model: C ~= pi * sqrt(2(a^2 + b^2)), using front-view widths and side-view depths -> qwen3-vl-plus analyzes both photos with the measurements as context to produce a visual body-fat estimate, body type (ectomorph/mesomorph/endomorph/mixed), and a health summary in the requested language -> U.S. Navy formula computed from the extracted circumferences for cross-validation -> weighted blend: 60% AI visual + 40% Navy formula for the final number (AI gets the higher weight because it sees the whole body, not just three measurements). Category labels follow ACE standards.
Labs Analyzer pipeline: File upload -> dual-path ingestion: (a) digital PDFs go through pdfplumber for text + table extraction, (b) scanned PDFs get converted to page images via pdf2image and sent to qwen3-vl-plus for OCR, (c) direct image uploads (phone photos of printed reports) skip straight to the vision model -> qwen3.5-plus parses the extracted text into a structured JSON array of {test_name, value, unit, ref_low, ref_high} -> deterministic rule-based severity classification (this is key for reliability): values within range = NORMAL, >50% outside = CRITICAL, >20% outside = ABNORMAL, otherwise BORDERLINE -> qwen3.5-plus generates patient-friendly explanations in the detected language, including plain-English descriptions, common non-alarming causes, and specific next steps.
Multilingual PDF export (bonus feature across all three challenges): Every output -- clinical report, body scan report, labs report -- exports as a professional PDF via reportlab. We register DejaVuSans fonts at runtime and use arabic_reshaper + python-bidi to handle Arabic RTL correctly (an early version rendered Arabic as empty boxes -- that bug is now fixed). French accents, Vietnamese diacritics, and Arabic text all render cleanly in the same document.
Challenges we ran into
Speed on the Medical Scribe. The obvious pipeline is transcribe -> extract JSON -> format report, which is three sequential calls and felt painfully slow in demos. We merged steps 2 and 3 into a single Qwen call using hard delimiters so the model emits both structured JSON and the human-readable report in one response -- then we parse them apart with regex. This alone cut ~3-5 seconds off every consultation. Layering SSE on top meant the user sees progress instead of a frozen spinner.
Multilingual medical speech. Arabic and Vietnamese medical terminology doesn't always transcribe cleanly, especially with code-switching -- a Vietnamese doctor using English drug names mid-sentence, or an Arabic physician mixing formal and colloquial terms. Switching our primary ASR to Qwen3-ASR-Flash (which is natively multilingual) improved this substantially over Whisper for FR/AR/VN, while keeping Whisper as a CPU fallback for offline robustness.
Body measurement accuracy from 2D photos is inherently limited. The elliptical circumference model works well for waist and hip but struggles with chest measurements where clothing or posture introduces noise. Our fix was the weighted blend -- giving the AI visual estimate 60% weight and the Navy formula 40% -- which produced more stable results than either method alone, because each one's errors partially cancel the other's.
Lab report format variability across countries. French reports use comma decimals (1,25 g/L), Arabic reports are right-to-left with mixed script directions, and Vietnamese reports blend local test names with international standards. Our dual-path approach -- pdfplumber for digital, qwen3-vl-plus for scanned or photographed -- combined with Qwen's multilingual parsing handles most formats, but edge cases in table layouts still required careful prompt engineering.
Keeping severity classification reliable. We initially let the LLM classify lab result severity, but found it occasionally flagged borderline results as critical. Switching to a deterministic rule engine (comparing values against reference ranges with fixed 20% / 50% deviation thresholds) brought severity accuracy from ~90% to 95-99% and removed a whole class of scary-sounding hallucinations.
Arabic PDFs rendering as empty boxes. A late-breaking bug. Default reportlab fonts don't support Arabic glyphs, and naively passing Arabic strings through reportlab produces boxes. We fixed it by registering DejaVuSans fonts at startup, passing every potentially-multilingual string through arabic_reshaper + python-bidi, and writing a small para() helper that auto-detects Arabic and applies the right font + direction.
Accomplishments that we're proud of
All three challenges on a single unified architecture. One FastAPI backend, one async Qwen client, two coordinated frontends (desktop web + mobile PWA). This isn't three demos glued together -- it's a coherent platform that could ship as a real Elfie feature.
Sub-$0.02 cost per unit across all three challenges. The Medical Scribe runs ~$0.014 per consultation, the Body Scan ~$0.008 per scan, and the Labs Analyzer ~$0.014-0.019 per report. At Elfie's scale of 1.5M+ users, this makes the AI layer economically viable for a free platform.
The hybrid AI + rules approach for the Labs Analyzer. Using Qwen for extraction and explanation but deterministic rules for severity classification delivered clinical reliability that neither approach could deliver alone.
The weighted 60/40 blend for body fat. Two imperfect estimators -- one visual, one geometric -- combine into something measurably more stable than either, and we can show the component numbers transparently so users see where the final number came from.
Merged-call + streaming scribe pipeline. Going from three sequential LLM calls to one merged call + SSE turned the Medical Scribe from "wait and hope" into a live, visible process.
Multilingual support that actually works end-to-end. We tested across all four target languages (EN/FR/AR/VN) from audio -> clinical reports, Arabic lab PDFs -> Vietnamese patient explanations, and Arabic output correctly renders in downloadable PDFs.
The entire hackathon ran within Alibaba Cloud's free quota -- 1 million tokens per model covered all our testing. Total infrastructure cost for the prototype was under $10.
What we learned
Hybrid AI + rules beats pure LLM for clinical accuracy. Deterministic rule engines for severity classification and the Navy formula for body fat cross-validation consistently outperform relying solely on LLM judgment for numerical tasks. The LLM excels at understanding messy inputs and generating human-friendly outputs -- let it do that, and let code handle the math.
Merging sequential LLM calls with delimiters is a real speed unlock. One call that emits both structured JSON and prose (separated by hard tokens) is dramatically faster than two calls, and parsing is trivially reliable with regex.
Qwen's multilingual capabilities are genuinely strong -- the model handles Vietnamese and Arabic medical text better than we initially expected, which is critical for Elfie's global footprint. Qwen3-ASR-Flash is also meaningfully better than Whisper on FR/AR/VN in our tests.
The DashScope OpenAI-compatible API is a massive developer experience win. We had a working prototype within hours because we could reuse existing OpenAI client patterns -- just change the base URL and model name.
Prompt temperature matters enormously in healthcare. Dropping from the default 0.7 to 0.1-0.15 for clinical extraction dramatically reduced hallucinated symptoms and invented diagnoses. Creative writing temperature has no place in medical AI.
Photo quality is everything for body scan. Controlled lighting, minimal clothing, and consistent pose make more difference than any model improvement. User guidance in the UI is as important as the AI behind it.
PDF export is deceptively hard across scripts. Getting Arabic to render correctly in a downloadable PDF (instead of boxes) requires the right font + bidi shaping pipeline -- it's a small amount of code but a non-obvious one.
What's next for Elfie HealthPilot
Real-time streaming ASR for the Medical Scribe using Qwen3-ASR-Flash's WebSocket API, so doctors see the transcript building live during the consultation instead of waiting for a file upload to process. Our SSE pipeline is already structured to accept streaming tokens.
3D body visualization using SMPL mesh reconstruction, giving patients an interactive 3D model of their body shape and the ability to track changes over time with side-by-side comparisons.
Longitudinal trend analysis for the Labs Analyzer -- comparing current results against historical uploads to surface improving or worsening biomarkers with visual trend charts.
On-device inference for body scan measurements using MediaPipe directly in the browser (via @mediapipe/tasks-vision), enabling fully offline body scans with zero API cost.
Integration with ElfieCare -- feeding Medical Scribe reports directly into the doctor's dashboard and linking Labs Analyzer results to the patient's health plan, closing the loop between AI-generated insights and clinical action.
HIPAA/GDPR-compliant self-hosted deployment using Qwen open-source models on dedicated infrastructure, enabling Elfie to process sensitive health data without sending it to third-party APIs.
Log in or sign up for Devpost to join the conversation.