Vision-Language-Model-ECG-Images

Master's research at Technion AIMLab — fine-tuning a vision-language model for ECG interpretation, benchmarked against GPT-4o, Gemini, Claude, and PULSE-7B

Comment

Vision-Language Model for ECG Diagnostics

Master's Research Thesis – Technion, Israel Institute of Technology (AIM Lab)
Supervised by Assoc. Prof. Joachim Behar (Head of AI in Medicine Laboratory - AIMLab) and Jonathan Fhima (PhD Candidate, AIMLab)

Why This Project

ECG interpretation still depends on clinical expertise that isn't always available — especially in emergencies or resource-limited settings. Large proprietary models (GPT-4o, Claude, Gemini) and specialized 7B+ models can read ECG images, but they're costly to run and hard to deploy at the point of care. This project asks a narrower question: how far can a small, open, deployable VLM (256M parameters) get on real ECG interpretation tasks — and where does it fall short?

Project Overview

This repository presents my Master's research project on Vision-Language Models (VLMs) applied to electrocardiogram (ECG) diagnostics.
The work builds upon the PULSE benchmark and evaluates SmolVLM-256M-Instruct, a lightweight open-source VLM fine-tuned for ECG reasoning and classification tasks.

Read the full thesis →

Objectives

  • Fine-tune SmolVLM-256M-Instruct (256M parameters) for ECG-specific reasoning tasks.
  • Benchmark performance against state-of-the-art proprietary and open-source models (PULSE-7B, GPT-4o, Gemini, Claude).
  • Focus on the ECGBench benchmark, particularly the CODE15 dataset, using metrics such as F1-score, AUC, Hamming Loss (HL).

Results – CODE15% (In-Domain)

Model F1 AUC HL
GPT-4o 24.9 59.9 15.7
Gemini 1.5 Pro 20.0 56.7 15.9
Claude 3.5 Sonnet 20.3 58.3 17.8
LLaVA-Med 27.0 69.2 33.4
Phi-3 Vision 22.6 69.6 38.8
PULSE-7B 85.4 90.7 5.0
SmolVLM-256M (Ours) 27.5 65.3 16.5

Key Insight: Our SmolVLM-256M performs in the upper range of open-source and generalist models — ahead of GPT-4o and Claude 3.5 Sonnet on F1, and ahead of GPT-4o, Gemini, and Claude on AUC — while remaining lightweight and efficient. PULSE-7B remains the clear leader, as expected for a specialized 7B-parameter model, but SmolVLM-256M demonstrates strong adaptability for its size.

Sample Predictions (CODE15)

Example 1 – Correct Prediction

Prompt:
The ECG image above belongs to a 54-year-old patient. Identify the cardiac rhythm and classify whether it is NORMAL or ABNORMAL.
Ground Truth: ABNORMAL – Atrial Fibrillation
Prediction (Ours): ABNORMAL – Atrial Fibrillation

Example 2 – Diagnostic Report

Prompt:
Evaluate the ECG and write a comprehensive diagnostic report.
Ground Truth: Normal sinus rhythm, no arrhythmia or conduction issues.
Prediction (Ours): Sinus rhythm, low QRS voltages (may suggest mild conduction abnormalities).

Example 3 – Incorrect Prediction

Prompt:
Classify the ECG condition (NORMAL or ABNORMAL) and specify the rhythm.
Ground Truth: ABNORMAL – Ventricular Tachycardia
Prediction (Ours): NORMAL – Sinus Rhythm

Research Context

This project draws inspiration from the PULSE benchmark, which sets a new standard for ECG reasoning with VLMs.
Our work demonstrates that smaller, optimized models like SmolVLM can achieve competitive results with targeted fine-tuning.

About Me

I hold a Master of Engineering in Computer Science — Diplôme d'Ingénieur from École Centrale d'Électronique (ECE Paris), and completed a Graduate Visiting Program (Master 2) in Data Science (Research Track) at the Technion – Israel Institute of Technology, where this research was conducted at the AI in Medicine Laboratory (AIMLab).

Focus areas:

  • Vision-Language Models (VLMs) and Multimodal AI
  • Deep Learning for Healthcare Applications
  • Model Efficiency and Domain Adaptation

References

Built With

Share this project:

Updates