SEMIAUTOMATIC DETECTION OF STENOSIS AND OCCLUSION OF PULMONARY ARTERIES FOR PATIENTS WITH CHRONIC THROMBOEMBOLIC PULMONARY HYPERTENSION

Chronic thromboembolic pulmonary hypertension (CTEPH) is a severe lung disease defined by the presence of chronic blood clots in the pulmonary arteries accompanied by severe health complications. It is necessary to go through a large set of axial sections from Computed tomography pulmonary angiogram (CTPA) for diagnosing the disease, which is difficult and time consuming for the radiologist. The radiologist's experience plays a significant role, same as subjective factors such as attention and fatigue. In this work we pursued the design and development of the algorithm for semiautomatic detection of pulmonary artery stenoses and clots for diagnosing CTEPH, which is based on the implementation of semantic segmentation using deep convolutional neural networks. Specifically, it is about the use of the DeepLab V3 + model embedded in the Xception architecture. Within this work we focused on stenoses and clots located in larger pulmonary arteries. Anonymized data of patients diagnosed with CTEPH and one healthy patient in the term of the presence of the disease were used for realization of this work. Statistical analysis of the results is divided into two parts: analysis of the created algorithm based on comparison of outputs with ground truth data (manually marked references) and analysis of pathology detection on new data based on comparison of predictions with reference images from the radiologist. The proposed algorithm correctly detects present vascular pathology in 83% of cases (sensitivity) and precisely selects cases where the investigated pathology does not occur in 72% of cases (specificity). The calculated Matthews correlation coefficient is 0.53. This means that the predictive ability of the algorithm is moderate positive. The designed and developed image analysis algorithm offers the radiologist a "second opinion" and it also could enable to increase the sensitivity of CTEPH diagnostics in cooperation with a radiologist


Introduction
Chronic thromboembolic pulmonary hypertension (CTEPH) is a serious pulmonary disease defined by the presence of chronic blood clots in the pulmonary arteries accompanied by vascular remodeling, pulmonary vascular resistance and right-sided heart failure in severe cases [1].
The most common method of diagnosing CTEPH is computed tomography [1,2].Specifically, the name CTPA (Computed tomography pulmonary angiogram) is used for the diagnosis of pulmonary artery disease.CT scans allow the finding of thrombi adjacent to the vessel walls, assessment of right ventricular (RV) status and changes in the lung parenchyma [1].
CTPA has emerged as a potentially useful diagnostic method for the evaluation of the pulmonary circulation, including the diagnosis of CTEPH [3][4][5][6].The main advantage of CTPA is the possibility of non-invasive diagnosis of pulmonary disease as opposed to conventional angiography [3,4].The sensitivity of computed tomography in the diagnosis of CTEPH varies between 83-100% (which is higher compared to conventional angiography and MRI) and specificity varies between 89-96% (higher than V/Q scintigraphy) ISSN 0301-5491 (Print), ISSN 2336-5552 (Online) ORIGINAL RESEARCH [1].Due to the high sensitivity, the number of false negative (FN) results is low [1,7].
Precisely timed intravenous injection of contrast agent is essential for quality visualization of the vascular lumen.Pathological findings from helical CT scans play an important role not only in the diagnosis of CTEPH, but also in the selection of patients for thromboendarterectomy or in the decision to treat the disease differently [8].
Although CTPA has a number of advantages, it also has a significant disadvantage in the form of processing a large set of images by a physician.The set of CT images from a single patient necessary for diagnosis includes an average of 600 submillimeter slices in the axial plane.In order to detect CTEPH, it is necessary to view most of them carefully and to pay attention to the presence of small formations inside the vessels (stenoses and occlusions), which may pose a potential danger to the patient's health.Although the overall size of the area of interest is much smaller, successive slices must be viewed several times to verify the accuracy of a single finding [7].
Diagnosis is primarily based on the detection of common symptoms that can unambiguously reveal the disease.Symptoms suggestive of CTEPH can be divided into four groups: vascular, parenchymal symptoms, symptoms of pulmonary hypertension and collateral circulation [8,9]: Vascular pathological changes are unambiguous signs of the presence of the disease.They include complete or partial obstruction of the pulmonary arteries, acute and chronic thrombi (acute and chronic pulmonary emboli can coexist [9]), calcified thrombi, various curvatures, septal defects, and poststenotic dilatations [10].
Computer-Aided Detection (CAD) is a novel technology to facilitate the search for pulmonary emboli (PE) on CTPA.CAD algorithms increase the sensitivity of PE diagnostics in cooperation with a physician [11,12].
There is a set of mathematical methods and procedures for implementing automatic pathology detection, although all of them involve an identical algorithm that includes the following steps: image preprocessing, segmentation, feature extraction and the actual implementation of detection.
CTPA-CAD methods have also been developed by well-known medical imaging suppliers systems such as Philips, Siemens and GE, but have not found wide commercial application due to high number of FP findings [7,11].
In this work we developed an algorithm for semiautomatic detection of pulmonary artery stenosis and occlusion in patients with CTEPH using convolutional neural networks.

Methods
Semi-automatic detection of vascular pathologies is based on the use of a supervised learning algorithm performed in Matlab2019b (The MathWorks, Inc., USA).Specifically, it is the application of semantic segmentation using deep convolutional neural networks.
Anonymized data of patients diagnosed with CTEPH and one healthy patient in terms of the presence of the disease were used.The Radiodiagnostic Clinic of General University Hospital in Prague provided fully anonymized data of CTPA chest images in DICOM format.The 512×512 images were obtained from a Philips iCT256 spiral multidetector CT scanner.The data set from one patient includes: 1. Axial submillimeter cuts with a thickness of 0.9 mm; 2. Axial, sagittal and coronal sections with a thickness of 3.0 mm; 3. Spatial model of the heart and vascular structures (VRT -volume rendering technique); 4. Reference images with problem areas marked.
Submillimeter axial sections were used to create the training set, with pathological sites marked by the physician.The training set contains data from 7 patients (26 variable vascular patterns).
The functionality of the semi-automated detection algorithm was verified on a pilot dataset of 25 patients with a confirmed diagnosis of CTEPH and one patient in whom the disease was excluded.

Train in g set
Since the variability of the symptoms of the disease is very high, in this work we focused on the most important vascular pathologies that can cause the greatest health complications.The training set contains sections showing thromboses and stenoses in the major pulmonary arteries (Figure 1).
A total of 26 variable patterns of vascular pathologies are included in the training set.Since thromboses and stenoses appear within several related images or slices, both the image or slice preceding the reference image and the image following it must be considered for processing.In the case of large pathologies, two images before the reference and two images after the reference are used.The training set includes 87 slices.ISSN 0301-5491 (Print), ISSN 2336-5552 (Online) The augmentation was performed by copying the set three times, since the appearance of pathologies does not vary.In total the training set contained 261 images.Geometric transforms were excluded as an option for dataset augmentation due to deforming stenoses shape and changing their localization.

ORIGINAL RESEARCH
The creation of the ground truth dataset is implemented in the Image Labeler application, which is part of Matlab2019b.Ground truth data serves as a standard for the proposed detection algorithm.Each image in the training set is divided into five regions (labels) which are manually marked (Figure 2): 1. Non-empty areas where pulmonary vascular pathology cannot occur-trunk section, cardiac musculature, ascending and descending aorta, bronchi, etc.; 2. Lung windows and the smallest vascular structures; 3. The area of interest where the pathology may potentially be located-the pulmonary arteries and the systemic pulmonary arteries (ROI); 4. Pathological manifestations of CTEPH that is indicated by the physician on the corresponding reference image.5. Background -no label area.We used the DeepLab V3+ model with weights initialized from the pre-trained Xception network.The Xception model was introduced in 2017 and has demonstrated significant performance in image classification with high computational speed [15,16].
The insertion of the DeepLab V3+ architecture into the Xception structure mainly affects the layers in the Exit Flow.The classification layers are replaced by a decoder.In addition, the changes affect the Entry Flow, where low-level flags from the Entry Flow and the input image are used in the decoding.The proposed CNN architecture contains a total of 205 layers [16,17].
A frequent phenomenon in biomedical images is the imbalance of anatomical structures in terms of the number of pixels that specific structures occupy.The default weights in the Pixel Classification Layer used by CNN do not account for this issue.
The proposal was to modify the weights in the classification layer with respect to the frequency of pixels occupied by each label.The frequency of pixels occupied by each label was calculated as follows: The modification of the weights calculated using the median showed the best results.The weights for a given label represent the ratio of the median frequency vector, and was calculated as follows: The number of iterations is automatically defined in Matlab based on the mini-batch size and the number of epochs.The number of iterations per epoch is 65, the total number of iterations is 6500.
Training is implemented on the NVIDIA GeForce GTX 1050 Ti 4.0 GB GPU.Training took 518 minutes.Number of epochs  = 100 was chosen experimentally.

Statist ic al an alysis of p roposed algorithm
The developed detection algorithm is evaluated by comparing the semantic segmentation result with ground truth.In the analysis of the algorithm, the confusion matrix, normalized confusion matrix, metrics of each image, each class and dataset are computed [18].

ORIGINAL RESEARCH
Quality metrics for semantic segmentation include the following parameters: 1. Global accuracy -the ratio of correctly classified pixels to the total number of pixels regardless of class calculated as follows: 2. Accuracy -the ratio of correctly classified pixels in the i-th class to the total number of pixels: 3. Mean accuracy -the ratio of correctly classified pixels in each class to the total number of pixels averaged over all classes.The value is equal to the mean class metrics accuracy (5) [19].
Where  is the number of labels.
4. The IoU score (Intersection over Union), which expresses the similarity of two finite sets and is defined as the ratio of the size of the intersection of the sets to the size of the union of the sets [18].
5. Average IoU score (mean IoU) -averaged IoU value from all classes calculated as follows [18]: The BF score indicates how well the predicted object boundary matches the boundary in the ground truth image.It is defined as the harmonic mean (F1-measure) of the sensitivity and the positive predictive value (PPV) with distance error tolerance to decide whether a point on the predicted boundary has a match with the ground truth boundary.The BF score is calculated as follows [20]: Metrics are calculated per class, per image and per dataset (Table 1).

Statist ic al an alysis of n ew data
The test set includes a total of 55 images.Statistical analysis of the detection results is performed by creating a confusion matrix for binary classification.Two labels out of five are considered: Background and Pathology.The assignment of the results to the different groups is described as on Figure 3:

Fig. 3: Visualization of semantic segmentation results. -TP -the location marked by the algorithm coincides with the marked location on the reference image. -FP -the location marked by the algorithm does not match the marked location on the reference image, and it also includes locations marked by the algorithm on images of a healthy patient. -FN -a place that is not marked by the algorithm, although it is a pathology according to the reference. -TN -a place that is not marked either by the algorithm or in the reference (background).
Basic statistical calculations from the confusion matrix include sensitivity and specificity.The Matthews correlation coefficient (MCC) is used to evaluate the quality of the proposed algorithm in terms of its ability to correctly detect pulmonary artery pathology.The MCC represents the correlation between the actual target (reference images with marked problem areas) and the network prediction.
The MCC value is in the range <-1,1>.˗ MCC = 1, means perfect prediction capability of the algorithm, ˗ MCC = 0, then the prediction is random, ˗ MCC = -1, indicates complete disagreement of the prediction with the reference.

Evalu at ion o f the qualit y o f the imp leme nted algorithm
Evaluation of the quality of the implemented algorithm was performed on training dataset by comparing the semantic segmentation algorithm output (using the evaluateSemanticSegmentation function in Matlab) with manually marked labels (ground truth).The function allows the automatic calculation of the confusion matrix, the normalized confusion matrix, per image (per image), per class (per class) and per dataset (per dataset) metrics (Tables 2-6).
From the confusion matrix, the metrics for each class (per class) are automatically computed: accuracy, IoU and mean BF score.The results are shown in the Tab. 4.
The metrics for each image include the computation of global and average accuracy, average IoU, weighted IoU, and mean BF score.Due to the large number of training data, we provided a summary of the maximum and minimum metric values per image (Table 5).
By averaging the metrics per image, the metrics for the entire dataset are calculated (Table 6).
According to the normalized confusion matrix (Table 3), all the pixels belonging to the Pathology label in ground truth were correctly assigned by the algorithm, but at the same time a small number of pixels from other labels were considered by the algorithm as belonging to the Pathology class.
The Figure 4 shows a selection of semantic segmentation results, the corresponding ground truth data, and also describes the color coding of each class.

App licat ion of the algor ithm to ne w data
The ability of the proposed algorithm to detect pathological findings on new data (different from training data) was verified on 55 images from 26 patients (including 20 images of a patient with excluded CTEPH).
The results of the semantic segmentation are represented as a confusion matrix for binary classification (Table 7).TP and TN findings are highlighted in color.A summary of the results is given below: TP: in 25 cases one pathology of pulmonary vessels is correctly detected on one image and in three patients two pathologies are correctly detected on one image.FP (type I error): in 17 cases, the algorithm flagged nonpathological areas in the patients and also flagged pathologies in four images of the healthy patient.FN (type II error): the algorithm failed to detect 7 pathological changes in the pulmonary vessels.The sensitivity of the proposed algorithm is 83% specificity is 72% and MCC is 0.53.

Discussion
The main finding of the study is that the predictive ability of the semantic segmentation using CNN is moderately positive with sensitivity 83% and specificity 72%.
This work is focused on specific manifestations of stenosis and thrombosis in the larger pulmonary arteries.This approach could increase the specificity of the algorithm.CNN-based image analysis is mainly based on local contrast changes in the image.Local contrast change consists of the appearance of a dark hypodense region embedded in a group of bright pixels that present a circular or elongated shape.
Manual augmentation by copying the training data resulted in the most accurate localization of the pathological site.The best results were obtained by defining five labels.Binary classification was applied.However, pathology recognition showed no efficiency due to the huge imbalance of the classes (15-50 pixels versus two hundred thousand in the Background class).Training took twice less time than using Xception.The network proved to be particularly effective in solving the problems network overlearning due to deeper architectures [21].Although there are a number of advantages, the proposed CNN performed worse in terms of finding vascular pathologies, meaning that the incidence of FN findings was higher.
Another attempt to address the complication of class imbalance was to replace the pixelClassificationLayer with a dicePixelClassificationLayer [22].The default setting of the loss function in pixelClassificationLayer is cross entropy loss, which theoretically complicates semantic segmentation for small classes [22].Class balancing in dicePixelClassificationLayer is implemented through the computation of Dice scores.This approach allows to increase the accuracy of semantic segmentation for small objects [22][23][24].However, the result of semantic segmentation using Dice score had worse results.
Another attempt was the use of the U-net [25,26], with one modification, namely mini-batch size = 2 due to GPU capacity limitations.When U-net was implemented on images of CTEPH patients, the network did not converge to 100% accuracy for a long time.The algorithm marked the locations in the image where there are no pathologies.
In terms of marking problematic spots in the image, no other procedure showed better results than modifying the weights using the median of the frequencies using cross-entropy in the classification layer.The proposed structure of CNN using DeepLab V3+ model with weights initialized from the pre-trained Xception network was considered as the most successful for CTEPH vascular pathology detection.
The number of correctly detected pathological sites is TP = 35.Four pathologies among them were not first marked by the physician but were subsequently checked after prediction by the algorithm.Processing such a large dataset from 25 patients diagnosed with CTEPH did not mean marking all pulmonary artery pathologies.In the first instance, the most significant and highly visible sites were marked by the radiologist.
The number of false positives is 21.These are mainly sites where the nature of the contrast change resembles true positives.Another reason is the observation of significant changes in anatomical structures depending on the location of the section.Incidental findings on the trunk are not counted in the FP.
The overall image contrast plays a role in the discovery of FP findings, in such a way that findings by a very small number of pixels appear randomly.When the contrast changes, large findings do not change location or disappear (unlike low-pixel findings), only the number of marked pixels changes.In terms of FP finds that are in the area of interest, the most frequent errors of the algorithm are marked pulmonary veins and arterial bifurcations.The algorithm marked four FP findings on the images of a healthy patient.
The algorithm did not detect seven pathological sites (FN).
Calculated sensitivity is 83%, specificity is 72%.This means that in 83% of cases the proposed algorithm correctly detects the present vascular pathology and in 72% it accurately selects cases in which the investigated pathology does not occur.The predictive ability of the algorithm is expressed by the MCC.The computed MCC is 0.53 which means that the prediction by the algorithm has a direct correlation with the observation and the locations in the image are not marked randomly.The prediction ability of the proposed algorithm can be characterized as moderately positive [27].
The algorithm for semi-automated detection of stenoses and thromboses has its limitations.The main one is the algorithm's focus on searching for pathological manifestations only in larger pulmonary arteries and only in those representing a septum running homogeneously across a filled vessel.Such findings represent the greatest medical complications for patients and play a minor role in the decision on treatment [10,28].On the other hand, this limitation brings a great advantage in terms of increasing the specificity of the algorithm, which is what has limited the commercial application of CAD.Another limitation is the sensitivity of the algorithm to changes in image contrast.This is obvious because the main principle of CNN is to work with local contrast changes.

Conclusion
The aim of this work was to propose an algorithm for semi-automatic detection of thromboses and occlusions in pulmonary arteries based on CTA images of chest of patients with CTEPH.The design and implementation of the algorithm for finding problem areas was performed using CNN-based semantic segmentation in Matlab environment.The performance of the algorithm was verified on a pilot cohort of patients with anonymized data and statistically evaluated.The number of correctly detected pathological sites is TP = 35.While 4 of them were not first identified by the physician but were subsequently checked after prediction by the algorithm.
The proposed semi-automated detection of stenoses and thromboses consists of providing the radiologist with "second opinion", whereby the doctor only needs to carefully examine the suspicious areas marked by the algorithm.This procedure could partially minimize the influence of subjective factors in the diagnosis.

Fig. 1 :
Fig. 1: CTEPH symptoms on reference images from the training set.

Fig. 2 : 2
Fig. 2: Labeled ground truth images with corresponding originals.Exporting labels from Image Labeler creates a PixelLabelData folder with ground truth data.The labels are stored as pixels with values from 1 to 4. The background is marked as 0. By converting to a grayscale image, we obtain the form of the original image.The pixel values in the greyscale images are as follows (Figure2):

7 ) 6 .
Weighted IoU -the averaged IoU value from all classes, weighted by the number of pixels in each class.7. The mean BF score (mean boundary F1 score) is the average of the BF scores from all data.

Figure 5
Figure 5 represents a selection of semantic segmentation results and the corresponding references.Findings are highlighted in green (TP) and red (FP) in the figures for better clarity.A summary of the results is given below: TP: in 25 cases one pathology of pulmonary vessels is correctly detected on one image and in three patients two pathologies are correctly detected on one image.FP (type I error): in 17 cases, the algorithm flagged nonpathological areas in the patients and also flagged pathologies in four images of the healthy patient.FN (type II error): the algorithm failed to detect 7 pathological changes in the pulmonary vessels.The sensitivity of the proposed algorithm is 83% specificity is 72% and MCC is 0.53.

Fig. 5 :
Fig. 5: Selection of semantic segmentation results applied on new data.
55 ISSN 0301-5491 (Print), ISSN 2336-5552 (Online) ORIGINAL RESEARCH Other CNN architectures have been investigated to implement pathology detection.All experiments were performed under the same conditions: the same training set, number of epochs and modification of the weights in the classification layer.The DeepLab V3+ model was embedded in the pre-trained ResNet-18 network.

Table 1 :
Metrics for algorithm evaluation.

Table 2 :
Confusion matrix, cells corresponding to the number of correctly assigned pixels (TP) are highlighted.

Table 3 :
Normalized confusion matrix, where each element of the matrix (, ) is the ratio of the number of pixels that belong to group  but are assigned to group  by the algorithm to the total number of pixels in that label.

Table 5 :
Maximum and minimum of metrics per image.

Table 7 :
Confusion matrix for binary classification.