Vision-language foundation model for echocardiogram interpretation.
Matthew ChristensenMilos VukadinovicNeal YuanDavid OuyangPublished in: Nature medicine (2024)
The development of robust artificial intelligence models for echocardiography has been limited by the availability of annotated clinical data. Here, to address this challenge and improve the performance of cardiac imaging models, we developed EchoCLIP, a vision-language foundation model for echocardiography, that learns the relationship between cardiac ultrasound images and the interpretations of expert cardiologists across a wide range of patients and indications for imaging. After training on 1,032,975 cardiac ultrasound videos and corresponding expert text, EchoCLIP performs well on a diverse range of benchmarks for cardiac image interpretation, despite not having been explicitly trained for individual interpretation tasks. EchoCLIP can assess cardiac function (mean absolute error of 7.1% when predicting left ventricular ejection fraction in an external validation dataset) and identify implanted intracardiac devices (area under the curve (AUC) of 0.84, 0.92 and 0.97 for pacemakers, percutaneous mitral valve repair and artificial aortic valves, respectively). We also developed a long-context variant (EchoCLIP-R) using a custom tokenizer based on common echocardiography concepts. EchoCLIP-R accurately identified unique patients across multiple videos (AUC of 0.86), identified clinical transitions such as heart transplants (AUC of 0.79) and cardiac surgery (AUC 0.77) and enabled robust image-to-text search (mean cross-modal retrieval rank in the top 1% of candidate text reports). These capabilities represent a substantial step toward understanding and applying foundation models in cardiovascular imaging for preliminary interpretation of echocardiographic findings.
Keyphrases
- left ventricular
- ejection fraction
- aortic stenosis
- artificial intelligence
- hypertrophic cardiomyopathy
- deep learning
- heart failure
- end stage renal disease
- newly diagnosed
- cardiac resynchronization therapy
- acute myocardial infarction
- left atrial
- cardiac surgery
- magnetic resonance imaging
- mitral valve
- aortic valve replacement
- machine learning
- chronic kidney disease
- computed tomography
- aortic valve
- pulmonary hypertension
- big data
- smoking cessation
- autism spectrum disorder
- transcatheter aortic valve implantation
- emergency department
- ultrasound guided
- prognostic factors
- acute kidney injury
- atrial fibrillation
- clinical practice
- acute coronary syndrome
- transcatheter aortic valve replacement
- mass spectrometry
- patient reported outcomes