Development and application of artificial intelligence in cardiac imaging
Abstract
In this review, we describe the technical aspects of artificial intelligence (AI) in cardiac imaging, starting with radiomics, basic algorithms of deep learning and application tasks of algorithms, until recently the availability of the public database. Subsequently, we conducted a systematic literature search for recently published clinically relevant studies on AI in cardiac imaging. As a result, 24 and 14 studies using CT and MRI, respectively, were included and summarized. From these studies, it can be concluded that AI is widely applied in cardiac applications in the clinic, including coronary calcium scoring, coronary CT angiography, fractional flow reserve CT, plaque analysis, left ventricular myocardium analysis, diagnosis of myocardial infarction, prognosis of coronary artery disease, assessment of cardiac function, and diagnosis and prognosis of cardiomyopathy. These advancements show that AI has a promising prospect in cardiac imaging.
Introduction
Cardiovascular diseases are the most common cause of mortality worldwide.1 The recent development of algorithms and methodologies for artificial intelligence (AI) accelerates quantitative automated imaging technology towards the diagnosis of diseases and personalized treatment strategies. Many algorithms have been developed for image detection, segmentation and classification, which significantly improved the clinical application in cardiac imaging. The AI-based clinical application has been developed, such as diagnosis of coronary artery disease, assessment of cardiac function, improvement of image quality. Since AI is currently revolutionizing the technical development and clinical application of cardiac imaging, in this review, we aim to give a broad overview of the development of AI in cardiac imaging, including CT and MRI. First, we narratively described the technical developments of AI. Then, we show the results of a systematic literature search for recent clinically relevant studies and discuss the clinical application of AI and its prospect.
Technical aspects
Radiomics
Radiomics is a process designed to extract a large number of quantitative image features using data-characterization algorithms.2,3 Radiomics allows data mining and statistical classifiers to determine the relevant features of an image to the target task and to build a prediction model, that is helpful to diagnose disorders in medical imaging. The radiomic features generally include size and shape based-features, intensity histogram, image voxel relationships, and filtered features and fractal features.4 Recently, radiomics showed to be able to differentiate hypertrophic cardiomyopathy from hypertensive heart disease; the integration of six texture and histogram features achieved an accuracy of 85.5%, outperforming the accuracy of conventional T1 weighted imaging of 64%.5 Radiomic texture analysis of late iodine enhancement on CT images reflects left ventricle remodeling and systolic–diastolic function, and may help to identify different patterns of structure remodeling.6 Coronary plaques are small and have a limited number of voxels, and are therefore very challenging for image analysis. Kolossvary et al demonstrated that the voxels of a coronary plaque were sufficient to perform a radiomic analysis, and found that 21% of radiomic parameters were significantly different between plaques with and without the napkin-ring sign and that radiomic parameters had a higher area under curve (AUC) than conventional parameters (0.92 vs 0.75).7 Kolossvary et al also performed a radiomic approach to identify advanced atherosclerotic lesions ex vivo, and showed a better AUC than visual assessment (0.73 vs 0.65).8
Basic algorithms of deep learning
In contrast to classical machine learning, which uses pre-computed features to build a predictive model for a diagnostic or prognostic task, deep learning simultaneously learns relevant features and builds up a predictive model from input images to the desired outcome.9 Convolutional neural network (CNN) is the most commonly used architecture in cardiac image analysis, which uses a convolution operation in the neural network. At present, CNN and CNN-derived networks have been widely used in detection, segmentation, and classification in cardiac imaging.10 Recurrent neural network (RNN) is another type of neural network, which is often used for processing sequential data. The commonly used RNN structures are long short-term memory (LSTM)11 and gated recurrent unit (GRU).12 There are many sequential data structures in cardiac imaging, such as cine MRI. RNN is especially useful to process the cardiac images for patients with arrhythmia or patients who have problems in breath-holding and even advanced techniques cannot generate good images.13–15
Application tasks of algorithms
The application of AI algorithms on cardiac imaging mainly consists of three parts: detection, segmentation, and classification.
In an image detection task, an object, such as a coronary plaque, has to be identified for further processing. For detection, CNN uses a bounding box to search for the target structure in the input two-dimensional or three-dimensional images. Subimages containing suspected structures are obtained and applied to another CNN model to discriminate between true or false target structures. Finally, the coordinates of detected target structures are obtained as the final output. Network architects like RNN and fully convolutional CNN (FCN) are also used for detection purposes. Xu et al used RNN to detect myocardial infarction areas in cardiac MR sequences and yielded an overall accuracy of 94.35%.16 Guo et al combined a multitask FCN to compute a local centerline distance map of a coronary artery tree, and to detect branch end points with minimal path extractor.17 They reduced missing coronary artery branches and improved the patient-level success rate of centerline extraction from 54.3 to 88.8%.
Image segmentation is the process of partitioning the cardiac image into different regions in order to separate certain structures or lesions, which is often used as a preprocessing step for feature extraction and classification. CNN and CNN-derived networks have been applied to segment left ventricle,18,19 right ventricle,20 coronary artery,21 epicardial and thoracic adipose tissue,22 and coronary calcified plaque.23,24Tran et al proposed to use FCN to segment left and right ventricle in cardiac MR25 FCN is an encoder–decoder-based CNN architecture that is widely used for image segmentation. The FCN model is trained end-to-end in a single learning stage and is easy to optimize. The segmentation task can be divided into several stages. Avendi et al used a segmentation pipeline in three stages.26 First, they used CNN to locate the left ventricle and obtained the location information as a bounding box. Second, a stacked autoencoder was applied to infer the shape of the left ventricle. Finally, a deformable model was proposed to do the segmentation with the inferred shape incorporated.
Image classification is used to predict a label for a given image and is often used to determine the presence or absence of an abnormality in cardiac imaging. For this purpose, CNN accepts the two-dimensional or three-dimensional images as input, forwards process and finalizes the fully connected layer and outputs a category label. Recently, unsupervised learning and ensemble learning have been applied to cardiac imaging.27,28 Zreik et al used an unsupervised convolutional autoencoder of the left ventricle myocardium and applied encoding statistics to classify patients into those with and without functionally significant coronary artery stenosis.27 Wu et al proposed a deep-learning architecture by combining multilayer perceptron (MLP) and a bidirectional tree-structural LSTM for anatomical classification or labeling of coronary arteries.28 The AUCs of labeling four main coronary branches and major side branches were 97 and 90%, respectively.
Deep learning can also resolve other imaging-related tasks, like image registration, generation, and enhancement. In the applications which incorporate the whole process of diagnosis, the algorithms are usually very complex and may consist of several different modules. Zhang et al introduced a method to diagnose chronic myocardial infarction on nonenhanced cardiac cine MRI. They firstly detected the left ventricle region with a localization deep learning network, then extracted motion features with LSTM, and finally used a fully connected discriminative network to distinguish myocardial infarction from normal tissues.19
Public database
The availability of high-quality image database is critical for the development of AI algorithms. The scope and quality of the data set determines the accuracy, generalizability, and robustness of the algorithm.29 The recent availability of public databases encourages researchers to pursue new AI algorithms. Public databases of lung nodules, e.g. the lung image database consortium (LIDC),30 greatly accelerated the development and application of deep learning in lung nodule detection. However, creating an image database is notoriously difficult, patient privacy must be protected and health information must be hidden.31 For developing deep learning algorithms, the database size and the coverage of the patient population should be ensured, which is critical for the generalizability. Meanwhile, labeling of a large number of medical images requires the expertise of experienced radiologists and great labor. In cardiology, several efforts are underway to create large databases from electronic health records like a cardiac arrhythmia database32 and a heart attack prediction database.33 These databases contain hundreds of variables, which can be used to develop new AI systems. There have been some initiatives to develop cardiac image databases. In 2016, Kaggle organized the second Data Science Bowl for left ventricular volume assessment, in which cardiac MR images from over 1000 subjects were provided by the National Institutes of Health and Children's National Medical Center.34 The goal of this competition was to develop algorithms to automatically measure cardiac volumes and to calculate ejection fractions. None of the images in the training and validation set were annotated. In 2017, the Medical Image Computing and Computer Assisted Intervention Society organized the Automated Cardiac Diagnosis Challenge for automatic segmentation of the left ventricular endocardium and epicardium, and a training set of 100 subjects was provided with manual annotation as ground truth.35 However, a major drawback is that those databases suffer from heterogeneity and low quality which narrows down the applicability of cardiac AI outcomes. Recently, a data set search engine has become available,36 listing data sets of multiple organized databases and prior competitions, potentially improved discoverability of existing data sources.
Clinical aspects
Scope and search strategy
A PubMed database search was performed using the terms related to AI, imaging methods and the heart. The searching strategy is shown in Table 1. Articles were included if they: (1) were clinically relevant studies and focused on patients; (2) used AI algorithms; (3) were published in the last 5 years (2015–2019); (4) investigated the detection, diagnosis, prognosis and quality control. Articles were excluded if they: (1) were reviews, abstracts, case reports, letters or conference papers; (2) did not use CT or MRI; (3) did not focus on the heart; (4) were technical studies; (5) did not investigate humans, but for experimental animals or phantom; (6) had a sample size <50, or did not mention the sample size.
| Search terms used to identify relevant citations |
|---|
| PubMed #1: (“tomography, X-ray computed”[MeSH] OR “Magnetic Resonance Imaging”[Mesh] OR “comput* tomography”[tiab] OR “radiography”[tiab] OR CT[tiab] OR MDCT[tiab] OR MR [tiab] OR MRI[tiab] OR CMR[tiab]) #2: (“Artificial Intelligence”[Mesh] OR “machine learning”[tiab] OR “deep learning”[tiab] OR “neural network*“[tiab] OR “convolutional neural network*“[tiab] OR “generative network*"[tiab] OR CNN*[tiab] OR AI[tiab]) #3: (“Heart”[Mesh] OR cardiac[tiab] OR cardi*[tiab]) Grammar in advanced search: #1 AND #2 AND #3 |
The flowchart of the literature review and selection is shown in Figure 1. The categories of AI tasks in clinical cardiac radiology are shown in Figure 2. The included literature are listed in Table 2.
Flowchart of literature review and selection. AI, artificial intelligence.
The four major categories (with example applications) of artificial intelligence tasks in clinical cardiac radiology. CAC, coronary artery calcium; CAD, coronary artery disease; CCTA, coronary computedtomographic angiography; EAT, epicardial adipose tissue; FFR-CT, fractional flow reserve-CT; PVAT, perivascular adipose tissue.
| Reference | Scan method | Application | AI method | Patient, n | Performance | |
|---|---|---|---|---|---|---|
| Tourassi et al. (2016)37 | Chest CT | Automatic CAC scoring | DL | CNN | 1,028 | Accuracy: 0.844 |
| Wolterink et al. (2016)24 | CCTA | Automatic CAC scoring | DL | CNN | 250 | Accuracy: 0.83 CC: 0.944 (correlation with cardiac calcium scoring CT) |
| Wang et al. (2019)38 | CT | Automatic CAC scoring | DL | 530 | CC: 0.77 (correlation with manual analysis) | |
| Cano-Espinosa et al. (2018)39 | CT | Automatic CAC scoring | DL | CNN | 5,973 | Accuracy: 0.726 CC: 0.93 (correlation with manual reference) |
| Commandeur et al. (2018)22 | CT | Automatic EAT quanification | DL | CNN | 250 | DC: 0.823 |
| Oikonomou et al. (2019)40 | CCTA | Automatic PVAT analysis | ML | 1,575 | A new AI-powered imaging biomarker leads to a striking improvement of cardiac risk prediction | |
| Zreik et al. (2019)41 | CCTA | Detection and classification of plaque and stenosis | DL | RNN | 163 | Accuracy: Plaque: 0.77 Stenosis: 0.80 |
| Zreik et al. (2019)42 | CCTA | Detection of functional stenosis | DL | convolutional autoencoders | 187 | AUC: 0.87 |
| Kumamaru et al. (2019)43 | CCTA | Automated estimation of CT-FFR | DL | CNN and GAN | 1,052 | Accuracy: 0.76 AUC: 0.78 |
| Itu et al. (2016)44 | CCTA | Detection of functional stenosis by CT-FFR | ML | 87 | Accuracy: 0.832 CC: 0.994 (correlation with CFD predictions) | |
| Coenen et al. (2018)45 | CCTA | Detection of functional stenosis by CT-FFR | ML | 351 | Accuracy: 0.85 CC: 0.997 (correlation with CFD predictions) | |
| Dey et al. (2018)46 | CCTA | Detection of functional stenosis by plaque quantification | ML | 80 | AUC: 0.84 | |
| Gaur et al. (2016)47 | CCTA | Detection of functional stenosis by CT-FFR and plaque quantification | ML | 254 | AUC: 0.90 | |
| von Knebel Doeberitz et al. (2019)48 | CCTA | Detection of functional stenosis by CT-FFR and plaque quantification | ML | 84 | AUC: 0.93 | |
| Han et al. (2018)49 | CCTA | Detection of functional stenosis by CTP | ML | 252 | ||
| Zreik et al. (2018)27 | CCTA | Detection of functional stenosis by myocardium analysis | DL | CNN | 166 | AUC: 0.74 DC: 0.91 |
| van Hamersvelt et al. (2019)50 | CCTA | Detection of functional stenosis by myocardium analysis | DL | CNN | 126 | AUC: 0.76 |
| Baessler et al. (2018)51 | Cine MRI | Diagnosis of subacute and chronic MI | ML | 120 | AUC: Subacute MI: 0.93 Chronic MI: 0.92 | |
| Larroza et al. (2018)52 | LGE and cine MRI | Diagnosis of chronic MI | ML | 50 | AUC: 0.849 | |
| Zhang et al. (2019)19 | Cine MRI | Diagnosis of chronic MI | DL | 212 | AUC: 0.94 | |
| Mannil et al. (2018)53 | CT | Diagnosis of subacute and chronic MI | ML | 57 | AUC: 0.78 | |
| Motwani et al. (2017)54 | CCTA | Prediction of all-cause mortality of CAD | ML | Information gain ranking | 10,030 | AUC: 0.79 |
| Nakanishi et al. (2018)55 | CT | Prognosis of CAD | ML | 6,814 | AUC: 0.765 | |
| Johnson et al. (2019)56 | CCTA | Prognosis of CAD | ML | 6,892 | AUC: 0.72–0.85 | |
| van Assen et al. (2019)57 | CCTA | Prognosis of CAD by plaque analysis | ML | 45 | AUC: 0.94 | |
| von Knebel Doeberitz et al. (2019)58 | CCTA | Prognosis of CAD by CT-FFR and plaque quantification | ML | 82 | AUC: 0.94 | |
| Tao et al. (2019)59 | Cine MRI | Automatic left ventricular function analysis | DL | CNN | 400 | Correlation with manual analysis (r = 0.98) |
| Ruijsink et al. (2019)60 | Cine MRI | Automatic cardiac function analysis | DL | 2,029 | Correlation with manual analysis (r = 0.89–0.95) | |
| Baessler et al. (2018)61 | MRI | Diagnosis of hypertrophic cardiomyopathy | ML | 62 | AUC: 0.95 | |
| Fahmy et al. (2019)62 | MRI | Automated myocardial scar quantification in hypertrophic cardiomyopathy | DL | CNN | 1,073 | Accuracy: 0.98 |
| Alis et al. (2019)63 | LGE MRI | Assessment of ventricular tachyarrhythmia in hypertrophic cardiomyopathy | ML | 64 | Accuracy: 0.941 | |
| Neisius et al. (2019)5 | MRI | Discriminates between hypertensive heart disease and hypertrophic cardiomyopathy | ML | SVM | 232 | Accuracy: 0.800 |
| Shao et al. (2018)64 | MRI | Diagnosis of dilated cardiomyopathy | ML | SVM | 50 | Accuracy: 0.85 |
| Gopalakrishnan et al. (2015)65 | MRI | Diagnosis of pediatric cardiomyopathies | ML | Bayesian rule learning | 83 | Accuracy: 0.807 AUC: 0.796 |
| Samad et al. (2018)66 | MRI | Prognosis of ventricular function after repairing tetralogy of Fallot | ML | SVM | 153 | AUC: 0.82 |
| Fries et al. (2019)67 | MRI | Classification of aortic valve malformations | ML | weak supervision | 4,000 | Enable classification of the aortic valve by weak supervision |
| Wolterink et al. (2017)68 | CT | Image denoising | DL | GAN | 284 | Cost:<10 s / case |
| Kustner et al. (2019)69 | MRI | Image deblurring | DL | GAN | 18 | Structural similarity index >0.8 Normalized mutual information >0.9 |
Automatic coronary artery calcium scoring
Coronary artery calcium (CAC) score on cardiac CT has been recognized as an independent predictor of cardiovascular events and mortality.70 Manual CAC measurement is time-consuming and its accuracy is heavily affected by motion artifacts, image noise, or blooming artifacts caused by multiple or large calcifications. The utilization of deep learning can fully automate this task, resulting in significant time savings and improved accuracy.24,37–39 Previous studies using a dedicated electrocardiogram (ECG)-triggered cardiac CT to acquire cardiac images to calculate CAC. Recently, it has been demonstrated that CAC scoring is also feasible on conventional non-ECG-triggered chest CT scans, although hampered by a substantial underestimation of the cardiac event risk. Tourassi et al showed that low-dose chest CT could achieve automatic CAC scoring.37 Using 1028 non-enhanced and non-ECG-triggered low-dose chest CT scans, CNN was trained to estimate cardiac bounding boxes, from which voxels ≥130 HU were extracted as candidates for CAC, and subsequently, Agatston scores were automatically calculated for a standard five-class cardiovascular risk classification, with an accuracy of 0.844. In another study, Wolterink et al showed that CAC quantification could be automatically derived from coronary CT angiography (CCTA).24 Based on a CNN trained with 250 patients who had both CCTA and non-enhanced cardiac CT scans, CCTA yielded results highly consistent with non-enhanced cardiac CT with an intraclass correlation coefficient of 0.944. This allows patients to have an automated CAC scoring while being screened for lung cancer by low-dose CT or undergoing CCTA scans, without the need for another dedicated cardiac CT scan for the coronary score. Therefore, the clinical workflow can be simplified and the radiation dose of CT for patients is reduced.
Automatic epicardial and perivascular adipose tissue analysis for improving cardiac risk prediction
Epicardial adipose tissue (EAT) is metabolically active fat in the pericardium directly surrounding the coronary arteries. EAT associates with diseases in the coronary arteries and the heart, such as arteriosclerosis, coronary calcification, and ventricular fibrillation, as well as adverse cardiovascular events. Commandeur et al developed a fully automated deep learning framework to quantify the EAT and thoracic adipose tissue of 250 asymptomatic individuals on CT scans, with a good correlation between automatic and expert manual quantification (r = 0.924–0.945), and a dice coefficient (DC) of 0.823 and 0.905, respectively.22 This rapid and automatic procedure has the potential to improve cardiovascular risk stratification in patients undergoing routine CT scans.
In addition, coronary artery inflammation can cause dynamic changes in the lipid balance of perivascular adipose tissue (PVAT), which can be detected by the perivascular fat attenuation index on CCTA. Oikonomou et al found that radiotranscriptomic analysis of coronary PVAT on CCTA could reveal coronary inflammation and structural remodeling. In their study, testing on 1575 patients, the analysis significantly improved the prediction of cardiac risk with respect to a traditional risk factor stratification (p < 0.001).40
Coronary CT angiography for coronary artery stenosis
CCTA plays a key role in the non-invasive assessment of coronary artery disease (CAD). With up to 99% negative predictive value, CCTA can effectively exclude obstructive CAD. CCTA can also be used to determine the degree of stenosis and quantitatively evaluate the arteriosclerotic plaque features, including high-risk plaque markers such as low attenuation, positive remodeling, spotty calcification, and nap-ring signs.71,72 But such measurements require complex post-processing by an experienced observer. Therefore, deep learning has been used to optimize the information extracted from CCTA, especially to develop algorithms that can perform plaque analysis in an automated, accurate, and objective manner. Zreik et al used recurrent CNN to automatically detect and classify the coronary artery plaque and diagnose the degree of coronary artery stenosis in 163 patients on CCTA, with an accuracy of 0.77 and 0.80, respectively.41 Most recently, AI has been shown to automatically segment coronary arteries, extract center lines and generate post-processed CCTA images (Figure 3), i.e. volume rendering, maximal intensity projection and curved multiplanar reconstructions. These procedures used automatic detection and measurement of stenosis, and finally generated structural or free-text reports. Although a multicenter study is necessary to validate these techniques, cardiac AI has shown a great potential to improve not only clinical workflow for CCTA but also to improve the accuracy of stenosis detection and clinical decision-making.
Schematic diagram of Shukun coronary artificial intelligence algorithm workflow. (A) axial image with annotations of aorta and coronary arteries (green); (B) VR image of initial segmentation including vessel image rupture (red) and coronary vein (blue); (C) VR image of corrected segmentation; (D) extracted centerline of coronary arteries; (E1) cMPR image with a detected stenosis (red); (E2) straightened rendering of the curved path shows a detected stenosis (red). cMPR, curved multiplanar reformation; VR, volume rendering.
Fractional flow reserve CT and CCTA plaque analysis for significant functional stenosis
CCTA mainly provides anatomical information of the coronary arteries, but there are still limitations in the diagnosis of functional coronary stenosis.73 Invasive fractional flow reserve (FFR) is regarded as the gold-standard for evaluating the significance of coronary hemodynamics, although recently, it has been shown in a non-inferior design study that non-invasive MR myocardial perfusion can replace invasive FFR.74 But FFR measurements with a pressure wire are invasive, relatively expensive and in a small number of patients FFR causes adverse events. The recently introduced FFR-CT, based on conventional CCTA images, can assess coronary hemodynamics and is highly consistent with invasive FFR,75 although in the critical stenosis category FFR-CT produces indeterminate outcomes. Furthermore, FFR-CT requires complex computer fluid dynamics (CFD) computations, that are time-consuming and heavily affected by the image quality. The utilization of AI on FFR-CT can replace the intensive computing of CFD, and can significantly reduce analyzing time.44,45,47,48 Itu et al trained 12,000 anatomic coronary artery structures using an AI algorithm.44 Their model can predict FFR values on each point on the central line of the coronary artery tree. The accuracy was 83.2% in 87 patients compared to invasive FFR as a gold-standard, and the correlation with the predictions based on the CFD model was excellent (r = 0.9994). The mean analyzing time was reduced from 196.4 s (CFD model) to 2.4 s (AI). In a multicenter study, FFR-CT was performed on 351 patients using a deep learning algorithm to detect functionally obstructive CAD,45 which ran on an ordinary computer. The diagnostic accuracy was significantly improved from 71% for pure visual CCTA to 85%, and the AUC from 0.69 to 0.84.
Some studies have found that quantitative plaque analysis can also predict lesion-specific ischemia. In a multicenter trial of 254 patients, Dey et al used quantitative plaque analysis to predict lesion-specific ischemia by quantifying stenosis, plaque volumes (non-calcified plaque, low-density plaque, and calcified plaque), contrast density difference and plaque length, and finally achieved an AUC up to 0.84.46 Some studies combined the two approaches, using machine learning method of FFR-CT and CCTA plaque analysis, and showed that the prediction of lesion-specific ischemia was improved to 0.90 by Gaur et al,47 and 0.93 by Von Knebel Doeberitz et al.48
Automatic left ventricular myocardium analysis for significant functional stenosis
Because an obstruction in a coronary artery may lead to left ventricular myocardial ischemia, for patients with moderate to severe coronary artery stenosis, a myocardial analysis can be performed in addition to direct analysis of the coronary artery in order to determine its functional significance. Zreik et al27 analyzed the left ventricular myocardium of 166 patients on CCTA by using a deep learning algorithm to automatically identify the significant functional coronary artery stenosis. They showed that left ventricular myocardial segmentation DC and AUC was 0.91 and 0.74, respectively. A single CCTA scan can automatically analyze the left ventricular myocardium to identify significant functional stenosis, which may strengthen the clinical evidence of non-invasive CCTA procedure, thus reduce the false positive rate for further invasive coronary angiography.
Diagnosis of myocardial infarction
The diagnosis of myocardial infarction (MI) by machine learning is mainly realized in cardiac MR (CMR).19,51,52,76 The most common imaging technique in CMR is cine sequence which can provide serial images and are mainly used to assess the contractile function of the heart. Late gadolinium enhancement (LGE) is a well-established sequence for quantifying the extent of tissue abnormalities in or around the cardiac muscle and can be used to detect and validate MI.77 Larroza et al analyzed 50 patients on LGE and cine CMR with chronic myocardial infarction to identify the infarcted non-viable, viable, and remote segments with an AUC of 0.849.52 Baessler et al used radiomic approach to diagnose subacute and chronic MI on non-contrast-enhanced cine CMR, with an AUC of 0.93 and 0.92, respectively.51 Zhang et al used deep learning to detect chronic myocardial infarction on non-enhanced cine CMR in 212 patients and reached an AUC of 0.94.19 Several studies have shown that a non-enhanced cine CMR can diagnose myocardial infarction with the similar efficiency as LGE CMR. Therefore, AI may have the potential to reduce the use of gadolinium contrast administration. MI can also be diagnosed with CT. Mannil et al used 6 commonly used machine learning methods to diagnose 27 patients with acute MI and 30 patients with chronic MI on low-dose CT, with an AUC of 0.78.53
Prognosis of coronary artery disease and adverse cardiovascular events
CCTA shows pathological information, that may be associated with prognosis, but the optimal way to extract the data is still under investigation. Machine learning can be used to model vascular features to better predict CAD and adverse cardiovascular events.54–57 The prediction of 5 year all-cause mortality in 10,030 patients in CCTA was studied by Motwani et al, where 25 clinical factors and 44 CCTA features were evaluated, and the AUC was 0.79.54 This was significantly better than the existing clinical or CCTA parameters, such as the Framingham risk with an AUC of only 0.61. Johnson et al used machine learning to assess the prognosis of 6892 patients on CCTA.56 The AUC for all-cause death, CAD death, coronary heart disease death, and non-fatal MI was 0.77, 0.72, 0.85, and 0.79, respectively. Van Assen et al reported an accuracy of 0.77 to predict adverse cardiovascular events in 45 patients with suspected CAD by plaque analysis, and the accuracy improved to 0.87 in combination with traditional clinical risk factors.57 Machine learning can better mine the prognosis information in CCTA and provide improved prognosis evaluation than with existing parameters.
Cardiac magnetic resonance imaging for cardiac function
Most cardiovascular imaging diagnoses are based on anatomical segmentation of the heart cavities, valves, coronary arteries, and precise measurements of various cardiac function parameters, including ejection fraction or perfusion defects. CMR is a non-invasive imaging technique that produces high-quality images with good tissue contrast between different soft tissues without exposure to ionizing radiation. CMR can obtain a specific anatomical plane in any direction, and there are many imaging methods such as cine, flow, tagged, LGE, and fusion CMR. Machine learning/deep learning based on CMR can now perform rapid and automatic segmentation of heart chambers such as left ventricle, right ventricle, bi-ventricles, and analysis of heart function. Tao et al used CNN to perform a multicenter quantitative study on 400 cine CMR patients,59 and obtained an excellent correlation with a manual expert analysis (r = 0.98). Deep learning is used to train highly variable data sets, which can be used for automatic and accurate cine CMR analysis of multivendor and multicenter data. Ruijsink et al used a deep learning framework to perform automatic cardiac analysis of 2029 patients,60 and applied comprehensive image quality control before and after the analysis. The results of the automatic analysis highly correlated with a manual analysis (r = 0.89–0.95), with a sensitivity of 0.95. Therefore, deep learning can be used to perform a fully automated, quality-controlled CMR analysis without the need for clinician supervision.
Diagnosis and prognosis of cardiomyopathy
CMR plays an increasingly important role in the diagnosis, treatment planning, and prognosis of cardiomyopathy.5,35,61–66 Baessler et al used radiomic analysis and machine learning to detect hypertrophied cardiomyopathy on non-enhanced CMR of 62 patients.61 Four radiomic features were identified with 100% sensitivity and 90% specificity. Changes in heart tissue can be measured with extremely high accuracy without enhancement. Shao et al used a support vector machine learning algorithm to perform a texture analysis of T1 weighted CMR images in 74 patients with an accuracy of 0.85.64 Gopalakrishnan et al analyzed 83 children's CMR using machine learning algorithms to diagnose pediatric cardiomyopathy with an accuracy of 0.807 and an AUC of 0.796.65 Using a machine learning algorithm, Samad et al were able to predict the risk of deterioration after repair surgery of tetralogy of Fallot.66 Two CMR examinations performed at least 6 months apart in 153 patients were analyzed. The AUC for differentiating two categories (deterioration and non-deterioration) was 0.82, and for three categories (major-, minor- and non-deterioration) was 0.77.
Image quality control and improvement
Generative adversarial network (GAN) uses deep learning‐based generative algorithms, which have shown to be able to solve generation and transformation problems in image processing, including image denoising and image deblurring.78 Recently, GAN methods have been applied to medical images. Wolterink et al used GAN to reduce noise on low-dose CT,68 as well as non-enhanced cardiac CT. After GAN training, the noise of low-dose CT was reduced, and the image quality was similar to conventional-dose CT. Similarly, GAN can also be used on MRI. Kustner et al performed retrospective motion correction via GAN to correct artifacts caused by respiration or cardiac motion with high evaluation metrics (normalized mean squared error 0.08, structural similarity index 0.8, normalized mutual information 0.9).69 These explorations are limited to the research environment, that would help to accomplish further practical application.
Prospect of clinical application
Many applications of cardiac AI have been used to make human work more accurate and faster in cardiac imaging. But most studies emphasized on the use of single-purpose algorithms or tools at the proof-of-concept stage. Although there is a long way between technological developments and a routine implementation in clinical practice, several specific AI applications have already demonstrated expert-level performance, and it is likely that these applications will be integrated into the imaging workflow of the radiologist.
One important future prospect of AI is the ability to perform beyond human perception and capability. The first example thereof is CNN powered FFR-CT. FFR is the golden standard for coronary hemodynamics. Traditional CCTA provides anatomic and pathological vision. Cardiac FFR-CT AI provides coronary hemodynamic information, that is not visible to the human eye and outperforms fluid dynamic analysis. The second example is the use of non-contrast cine cardiac MRI data to detect the presence of myocardial infarction which cannot be clearly identified by the human eyes and has been validated by using machine learning, radiomics and deep learning.19,51,52 These approaches may have the potential to reduce the use of gadolinium contrast and largely shorten scan duration. Another example is the use of CNN to improve image quality or to correct motion artifact, which can be the main barrier for a clear cardiac image and is beyond human capability. Zhang et al did an experimental study showing that CNN trained by motion artifacts can largely reduce the CAC scoring variation, and to correct CAC scores of blurred images.79
Another future application field is AI-driven prognosis assessment. Traditionally, clinicians choose treatment methods according to their knowledge and existing evidence. Some studies have demonstrated optimal cardiac AI results for the prediction of prognosis, which will potentially impact therapy planning.54,56 Nevertheless, more well-designed clinical trials are needed to confirm these early results and to open the perspective of clinical implementation of cardiac AI.
Conclusion
Cardiac AI algorithms have been developed for image detection, segmentation, and classification, as well as recently available advanced algorithms, which can significantly improve the clinical application in cardiac imaging. In the past few years, numerous studies have shown the great potential of AI in the diagnosis of CAD and cardiomyopathy, and the assessment of cardiac function, as well as the prediction of prognosis. Recently, cardiac AI exhibits its ability to outperform human perception in several specific cardiac fields. These developments in cardiac imaging improved the daily practice of cardiac imaging and will greatly show its impact in the near future.
Acknowledgment We thank Dr. Marcel J. W. Greuter for his kindly assistance with language editing.
Competing interests Dr. Ning Guo is an employee of Shukun (Beijing) Technology Co., Ltd.
Funding National Natural Science Foundation of China (project no. 81471662 and 81971612), Ministry of Science and Technology of China (2016YFE0103000), Shanghai Municipal Education Commission – Gaofeng Clinical Medicine Grant Support (20181814), Shanghai Jiao Tong University (ZH2018ZDB10), and Clinical Research Innovation Plan of Shanghai General Hospital (CTCCR-2018B04, CTCCR-2019D05). The funders played no role in the study design, data collection, and analysis, decision to publish, or preparation of the manuscript.
REFERENCES
1. . Ncds: a challenge to sustainable human development. The Lancet 2013; 381: 510–1. doi: https://doi.org/10.1016/S0140-6736(13)60058-6
2. . Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer 2012; 48: 441–6. doi: https://doi.org/10.1016/j.ejca.2011.11.036
3. . Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat Commun 2014; 5:
4006 . doi: https://doi.org/10.1038/ncomms50064. . Cardiac computed tomography Radiomics: a comprehensive review on radiomic techniques. J Thorac Imaging 2018; 33: 26–34. doi: https://doi.org/10.1097/RTI.0000000000000268
5. . Radiomic Analysis of Myocardial Native T1 Imaging Discriminates Between Hypertensive Heart Disease and Hypertrophic Cardiomyopathy. JACC Cardiovasc Imaging 2019; 12: 1946–54. doi: https://doi.org/10.1016/j.jcmg.2018.11.024
6. . Assessment of remote myocardium heterogeneity in patients with ventricular tachycardia using texture analysis of late iodine enhancement (lie) cardiac computed tomography (CCT) images. Mol Imaging Biol 2018; 20: 816–25. doi: https://doi.org/10.1007/s11307-018-1175-1
7. . Radiomic features are superior to conventional quantitative computed tomographic metrics to identify coronary plaques with napkin-ring sign. Circ Cardiovasc Imaging 2017; 10. doi: https://doi.org/10.1161/CIRCIMAGING.117.006843
8. . Radiomics versus visual and histogram-based assessment to identify atheromatous lesions at coronary CT angiography: an ex vivo study. Radiology 2019; 293: 89–96. doi: https://doi.org/10.1148/radiol.2019190407
9. . State-Of-The-Art deep learning in cardiovascular image analysis. JACC Cardiovasc Imaging 2019; 12(8 Pt 1): 1549–65. doi: https://doi.org/10.1016/j.jcmg.2019.06.009
10. . Deep learning in radiology: an overview of the concepts and a survey of the state of the art with focus on MRI. J Magn Reson Imaging 2019; 49: 939–54. doi: https://doi.org/10.1002/jmri.26534
11. . Long short-term memory. Neural Comput 1997; 9: 1735–80. doi: https://doi.org/10.1162/neco.1997.9.8.1735
12. . Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv e-prints [serial on the Internet]. 2014. Available from: https://ui.adsabs.harvard.edu/abs/2014arXiv1412.3555C.
13. . Recurrent neural networks for aortic image sequence segmentation with sparse annotations. arXiv e-prints [serial on the Internet]. 2018. Available from: https://ui.adsabs.harvard.edu/abs/2018arXiv180800273B.
14. . Cardiac-deepied: automatic pixel-level deep segmentation for cardiac bi-ventricle using improved end-to-end encoder-decoder network. IEEE J Transl Eng Health Med 2019; 7: 1:
1900110 –10. doi: https://doi.org/10.1109/JTEHM.2019.290062815. . Convolutional recurrent neural networks for dynamic Mr image reconstruction. IEEE Trans Med Imaging 2019; 38: 280–90. doi: https://doi.org/10.1109/TMI.2018.2863670
16. . Direct Detection of Pixel-Level Myocardial Infarction Areas via a Deep-Learning Algorithm. Cham: Springer International Publishing; 2017.
17. . DeepCenterline: A Multi-task Fully Convolutional Network for Centerline Extraction. Cham: Springer International Publishing; 2019..
18. . Automatic localization of the left ventricle in cardiac MRI images using deep learning. Conf Proc IEEE Eng Med Biol Soc 2015; 2015: 683–6. doi: https://doi.org/10.1109/EMBC.2015.7318454
19. . Deep learning for diagnosis of chronic myocardial infarction on nonenhanced cardiac cine MRI. Radiology 2019; 291: 606–17. doi: https://doi.org/10.1148/radiol.2019182304
20. . Automatic segmentation of the right ventricle from cardiac MRI using a learning-based approach. Magn Reson Med 2017; 78: 2439–48. doi: https://doi.org/10.1002/mrm.26631
21. . Coronary Artery Segmentation in Cardiac CT Angiography Using 3D Multi-Channel U-net. arXiv e-prints [serial on the Internet]. 2019. Available from: https://ui.adsabs.harvard.edu/abs/2019arXiv190712246C.
22. . Deep learning for quantification of epicardial and thoracic adipose tissue from non-contrast CT. IEEE Trans Med Imaging 2018; 37: 1835–46. doi: https://doi.org/10.1109/TMI.2018.2804799
23. . Automatic calcium scoring in low-dose chest CT using deep neural networks with dilated convolutions. IEEE Trans Med Imaging 2018; 37: 615–25. doi: https://doi.org/10.1109/TMI.2017.2769839
24. . Automatic coronary artery calcium scoring in cardiac CT angiography using paired convolutional neural networks. Med Image Anal 2016; 34: 123–36. doi: https://doi.org/10.1016/j.media.2016.04.004
25. . A fully convolutional neural network for cardiac segmentation in short-axis MRI. arXiv e-prints [serial on the Internet]. 2016. Available from: https://ui.adsabs.harvard.edu/abs/2016arXiv160400494T.
26. . A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI. arXiv e-prints [serial on the Internet]. 2015. Available from: https://ui.adsabs.harvard.edu/abs/2015arXiv151207951A.
27. . Deep learning analysis of the myocardium in coronary CT angiography for identification of patients with functionally significant coronary artery stenosis. Med Image Anal 2018; 44: 72–85. doi: https://doi.org/10.1016/j.media.2017.11.008
28. . Automated anatomical labeling of coronary arteries via bidirectional tree LSTMs. Int J Comput Assist Radiol Surg 2019; 14: 271–80. doi: https://doi.org/10.1007/s11548-018-1884-6
29. . Artificial intelligence in cardiovascular imaging: JACC state-of-the-art review. J Am Coll Cardiol 2019; 73: 1317–35. doi: https://doi.org/10.1016/j.jacc.2018.12.054
30. . The lung image database Consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans. Med Phys 2011; 38: 915–31. doi: https://doi.org/10.1118/1.3528204
31. . Challenges related to artificial intelligence research in medical imaging and the importance of image analysis competitions. Radiology: Artificial Intelligence 2019; 1.
32. . A formal definition of big data based on its essential features. Library Review 2016; 65: 122–35. doi: https://doi.org/10.1108/LR-06-2015-0061
33. . Imaging in population science: cardiovascular magnetic resonance in 100,000 participants of UK Biobank - rationale, challenges and approaches. J Cardiovasc Magn Reson 2013; 15. doi: https://doi.org/10.1186/1532-429X-15-46
34. . Protocol and quality assurance for carotid imaging in 100,000 participants of UK Biobank: development and assessment. Eur J Prev Cardiol 2017; 24: 1799–806. doi: https://doi.org/10.1177/2047487317732273
35. . Disproportionate cardiac hypertrophy during early postnatal development in infants born preterm. Pediatr Res 2017; 82: 36–46. doi: https://doi.org/10.1038/pr.2017.96
36. Dataset search. Available from: https://toolbox.google.com/datasetsearch [Accessed September 7, 2018].
37. . Deep convolutional neural networks for automatic coronary calcium scoring in a screening study with low-dose chest CT. Medical Imaging 2016: Computer-Aided Diagnosis 2016;.
38. . Coronary artery calcium score quantification using a deep-learning algorithm. Clin Radiol 2019. doi: https://doi.org/10.1016/j.crad.2019.10.012
39. . Automated agatston score computation in non-ECG gated CT scans using deep learning. Proc SPIE Int Soc Opt Eng 2018; 1057402 03 2018. doi: https://doi.org/10.1117/12.2293681
40. . A novel machine learning-derived radiotranscriptomic signature of perivascular fat improves cardiac risk prediction using coronary CT angiography. Eur Heart J 2019; 40: 3529–43. doi: https://doi.org/10.1093/eurheartj/ehz592
41. . A recurrent cnn for automatic detection and classification of coronary artery plaque and stenosis in coronary CT angiography. IEEE Trans Med Imaging 2019; 38: 1588–98. doi: https://doi.org/10.1109/TMI.2018.2883807
42. . Deep learning analysis of coronary arteries in cardiac CT angiography for detection of patients requiring invasive coronary angiography. IEEE Trans Med Imaging 2019;:
1 . doi: https://doi.org/10.1109/TMI.2019.295305443. . Diagnostic accuracy of 3D deep-learning-based fully automated estimation of patient-level minimum fractional flow reserve from coronary computed tomography angiography. Eur Heart J Cardiovasc Imaging 2019; 80. doi: https://doi.org/10.1093/ehjci/jez160
44. . A machine-learning approach for computation of fractional flow reserve from coronary computed tomography. J Appl Physiol 2016; 121: 42–52. doi: https://doi.org/10.1152/japplphysiol.00752.2015
45. . Diagnostic accuracy of a machine-learning approach to coronary computed tomographic angiography-based fractional flow reserve: result from the machine Consortium. Circ Cardiovasc Imaging 2018; 11: e007217:
e007217 . doi: https://doi.org/10.1161/CIRCIMAGING.117.00721746. . Integrated prediction of lesion-specific ischaemia from quantitative coronary CT angiography using machine learning: a multicentre study. Eur Radiol 2018; 28: 2655–64. doi: https://doi.org/10.1007/s00330-017-5223-z
47. . Coronary plaque quantification and fractional flow reserve by coronary computed tomography angiography identify ischaemia-causing lesions. Eur Heart J 2016; 37: 1220–7. doi: https://doi.org/10.1093/eurheartj/ehv690
48. . Coronary CT angiography-derived plaque quantification with artificial intelligence CT fractional flow reserve for the identification of lesion-specific ischemia. Eur Radiol 2019; 29: 2378–87. doi: https://doi.org/10.1007/s00330-018-5834-z
49. . Incremental role of resting myocardial computed tomography perfusion for predicting physiologically significant coronary artery disease: a machine learning approach. J Nucl Cardiol 2018; 25: 223–33. doi: https://doi.org/10.1007/s12350-017-0834-y
50. . Deep learning analysis of left ventricular myocardium in CT angiographic intermediate-degree coronary stenosis improves the diagnostic accuracy for identification of functionally significant stenosis. Eur Radiol 2019; 29: 2350–9. doi: https://doi.org/10.1007/s00330-018-5822-3
51. . Subacute and chronic left ventricular myocardial scar: accuracy of texture analysis on nonenhanced cine Mr images. Radiology 2018; 286: 103–12. doi: https://doi.org/10.1148/radiol.2017170213
52. . Texture analysis of cardiac cine magnetic resonance imaging to detect nonviable segments in patients with chronic myocardial infarction. Med Phys 2018; 45: 1471–80. doi: https://doi.org/10.1002/mp.12783
53. . Texture analysis and machine learning for detecting myocardial infarction in noncontrast low-dose computed tomography: unveiling the invisible. Invest Radiol 2018; 53: 338–43. doi: https://doi.org/10.1097/RLI.0000000000000448
54. . Machine learning for prediction of all-cause mortality in patients with suspected coronary artery disease: a 5-year multicentre prospective registry analysis. Eur Heart J 2017; 38: 500–7. doi: https://doi.org/10.1093/eurheartj/ehw188
55. . Machine learning in predicting coronary heart disease and cardiovascular disease events: results from the multi-ethnic study of atherosclerosis (MESA. J Am Coll Cardiol 2018; 71:
A1483 . doi: https://doi.org/10.1016/S0735-1097(18)32024-256. . Scoring of coronary artery disease characteristics on coronary CT angiograms by using machine learning. Radiology 2019; 292: 354–62. doi: https://doi.org/10.1148/radiol.2019182061
57. . Automated plaque analysis for the prognostication of major adverse cardiac events. Eur J Radiol 2019; 116: 76–83. doi: https://doi.org/10.1016/j.ejrad.2019.04.013
58. . Impact of coronary computerized tomography angiography-derived plaque quantification and machine-learning computerized tomography fractional flow reserve on adverse cardiac outcome. Am J Cardiol 2019; 124: 1340–8. doi: https://doi.org/10.1016/j.amjcard.2019.07.061
59. . Deep learning-based method for fully automatic quantification of left ventricle function from cine Mr images: a multivendor, multicenter study. Radiology 2019; 290: 81–8. doi: https://doi.org/10.1148/radiol.2018180513
60. . Fully automated, quality-controlled cardiac analysis from CMR: validation and large-scale application to characterize cardiac function. JACC Cardiovasc Imaging 2019;17 Jul 2019. doi: https://doi.org/10.1016/j.jcmg.2019.05.030
61. . Texture analysis and machine learning of non-contrast T1-weighted Mr images in patients with hypertrophic cardiomyopathy-Preliminary results. Eur J Radiol 2018; 102: 61–7. doi: https://doi.org/10.1016/j.ejrad.2018.03.013
62. . Three-Dimensional deep convolutional neural networks for automated myocardial scar quantification in hypertrophic cardiomyopathy: a multicenter multivendor study. Radiology 2019;:
190737 .63. . Assessment of ventricular tachyarrhythmia in patients with hypertrophic cardiomyopathy with machine learning-based texture analysis of late gadolinium enhancement cardiac MRI. Diagn Interv Imaging 2019. doi: https://doi.org/10.1016/j.diii.2019.10.005
64. . Texture analysis of magnetic resonance T1 mapping with dilated cardiomyopathy: a machine learning approach. Medicine 2018; 97:
e12246 . doi: https://doi.org/10.1097/MD.000000000001224665. . cMRI-BED: a novel informatics framework for cardiac MRI biomarker extraction and discovery applied to pediatric cardiomyopathy classification. Biomed Eng Online 2015; 14 Suppl 2(Suppl 2):
S7 . doi: https://doi.org/10.1186/1475-925X-14-S2-S766. . Predicting deterioration of ventricular function in patients with repaired tetralogy of Fallot using machine learning. Eur Heart J Cardiovasc Imaging 2018; 19: 730–8. doi: https://doi.org/10.1093/ehjci/jey003
67. . Weakly supervised classification of aortic valve malformations using unlabeled cardiac MRI sequences. Nat Commun 2019; 10:
3111 . doi: https://doi.org/10.1038/s41467-019-11012-368. . Generative adversarial networks for noise reduction in low-dose CT. IEEE Trans Med Imaging 2017; 36: 2536–45. doi: https://doi.org/10.1109/TMI.2017.2708987
69. . Retrospective correction of motion-affected Mr images using deep learning frameworks. Magn Reson Med 2019; 82: 1527–40. doi: https://doi.org/10.1002/mrm.27783
70. . Comparison of novel risk markers for improvement in cardiovascular risk assessment in intermediate-risk individuals. JAMA 2012; 308: 788–95. doi: https://doi.org/10.1001/jama.2012.9624
71. . The high-risk criteria low-attenuation plaque. Eur Heart J Cardiovasc Imaging 2017; 18: 772–9.
72. . Evaluation of coronary plaque characteristics with coronary computed tomography angiography in patients with non-obstructive coronary artery disease: a long-term follow-up study. Eur Heart J Cardiovasc Imaging 2016;.
73. . Evolving concepts of angiogram: fractional flow reserve discordances in 4000 coronary stenoses. Eur Heart J 2014; 35: 2831–8. doi: https://doi.org/10.1093/eurheartj/ehu094
74. . Magnetic resonance perfusion or fractional flow reserve in coronary disease. N Engl J Med 2019; 380: 2418–28. doi: https://doi.org/10.1056/NEJMoa1716734
75. . Computational fluid dynamics applied to cardiac computed tomography for noninvasive quantification of fractional flow reserve: scientific basis. J Am Coll Cardiol 2013; 61: 2233–41. doi: https://doi.org/10.1016/j.jacc.2012.11.083
76. . Interpreting CNNs via decision trees. arXiv e-prints [serial on the Internet]. 2018. Available from: https://ui.adsabs.harvard.edu/abs/2018arXiv180200121Z.
77. . Cardiac MRI: a general overview with emphasis on current use and indications. Med Clin North Am 2015; 99: 849–61. doi: https://doi.org/10.1016/j.mcna.2015.02.011
78. . Generative adversarial networks. arXiv e-prints [serial on the Internet]. 2014. Available from: https://ui.adsabs.harvard.edu/abs/2014arXiv1406.2661G.
79. . Motion-corrected coronary calcium scores by a convolutional neural network: a robotic simulating study. Eur Radiol 2020; 30: 1285–94. doi: https://doi.org/10.1007/s00330-019-06447-7
