Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Multi-feature fusion method for medical image retrieval using wavelet and bag-of-features

Multi-feature fusion method for medical image retrieval using wavelet and bag-of-features COMPUTER ASSISTED SURGERY 2019, VOL. 24, NO. S1, 72–80 https://doi.org/10.1080/24699322.2018.1560087 RESEARCH ARTICLE Multi-feature fusion method for medical image retrieval using wavelet and bag-of-features a,b a c d Liu Shuang , Chen Deyun , Chen Zhifeng and Pang Ming Center for Post-Doctoral Studies of Computer Science and Technology, Harbin University of Science and Technology, Harbin, Heilongjiang, China; College of Computer and Information Engineering, Harbin University of Commerce, Harbin, Heilongjiang, China; c d College of Energy and Architectural Engineering, Harbin University of Commerce, Harbin, Heilongjiang, China; College of Automation, Harbin Engineering University, Harbin, Heilongjiang, China KEYWORDS ABSTRACT Word; medical image Color, texture, and shape are the common features used for the retrieval systems. However, retrieval; bag-of-feature; many medical images have a spot of color information. Therefore, the discriminative texture and texture feature; LBP feature shape features should be extracted to obtain a satisfied retrieval result. In order to increase the credibility of the retrieval process, many features can be combined to be used for medical image retrieval. Meanwhile, more features require more processing time, which will decrease the retrieval speed. In this paper, wavelet decomposition is adopted to generate different reso- lution images. Bag-of-feature, texture, and LBP feature are extracted from three different-level wavelet images. Finally, the similarity measure function is obtained by fusing these three types of features. Experimental results show that the proposed multi-feature fusion method can achieve a higher retrieval accuracy with an acceptable retrieval time. 1. Introduction feature is characterized by combining texture with shape information [5]. Cardiac CT images are import- The medical content-based image retrieval (M-CBIR) ant parts of the medical database, and related systems have been developed and used for path- retrieval methods have been formed by taking ology, radiology, and clinical laboratory diagnostics advantage of the heart shape. A contour and texture [1]. In these systems, some databases only contain a based image retrieval technology has been put for- single kind of medical image, while others contain ward and applied to the liver image database [6]. many kinds of medical images. Radiology images not Patch-based features have been applied to X-ray only have a large number but also play an important medical image retrieval, while scale invariant feature role in auxiliary diagnosis. Therefore, the radiation transform (SIFT) has been selected as well as a new image retrieval system has become a hot research feature for medical image retrieval. area [2]. Since wavelet transformation can extend an image Since it iseasy toidentify color and texture in the into different size and obtain idiographic feature infor- pathological image, CBIR is often used in the mation, it is also introduced to medical image research of retrieval system of pathological images. retrieval. By combining Gabor filter and Euclidean dis- A computer-aided diagnosis system was proposed tance, wavelet transform has achieved better perform- for pigmented skin lesions and multiple classifier sys- ance on image retrieval. When features have been tems were used for melanoma diagnosis [3]. To selected, image retrieval needs good methods to clas- retrieve the medical image, the traditional global sify these features in order to obtain the best result features have been widely used including color fea- similar to query image [7]. Support vector machine is tures, texture features, and shape features. Compared with text retrieval, image retrieval often in common use for retrieval in medical image data- constructs visual words by utilizing all kinds of fea- base including X-Rays, MRI, and CT. IRMA is an effi- tures [4]. ASSERT is a retrieval system for lung CT cient image retrieval system using feature fusion and image. In the retrieval algorithm of ASSERT, lesion support vector machine [8]. CONTACT Pang Ming pangm@hrbeu.edu.cn College of Automation, Harbin Engineering University, Harbin, Heilongjiang, China 2019 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. COMPUTER ASSISTED SURGERY 73 Figure 1. Framework of the proposed method. To improve the performance of CBIR for a medical the proposed method is founded in Section 2.In image, machine learning has been used in pre-filtering Section 3, mathematical expression and execution image and statistical similarity has been used in the details of the proposed method are listed. matching of multi-feature between query image and Experiments are shown in Section 4 and conclusion is database [9]. UMLS is a successful medical image listed in Section 5. retrieval system which has a structured learning frame- work and modular design based on support vector 2. Retrieval framework of the machine. To find out the possible lesion region in proposed method brain images, domain knowledge has been used in retrieving the image sequence. A boosting framework The retrieval framework of the proposed method is has been proposed to improve the performance of shown as in Figure.1 medical image retrieval. Evaluation results show that From Figure 1, we can see that this method will be this method has a high retrieval accuracy with a low implemented by seven steps: computational cost [10]. 1. Different resolution images are obtained by using In this paper, a multi-feature fusion method was proposed for medical image retrieval based on wave- wavelet decomposition. let transform and bag-of-features. The remaining parts 2. Gray-based bag-of-features are computed accord- of this paper are organized as follows: Framework of ing to first-level resolution image. 74 L. SHUANG ET AL. Hash Image of Feature Feature Hash Code Coding Extraction Database Vectors Hanming Distance Feature Feature Hash Hash Code Query Image Extraction Coding Vectors Figure 2. Feature selection. (2) (2) LL LH (1) (1) (1) LL LH LH (2) (2) HL HH (0) LL (1) (1) (1) (1) HL HH HL HH Figure 3. Wavelet decomposition. 3. Texture features are extracted from second-level vectors of different dimensions are formed, and hash resolution image. coding is generated by using these vectors as the 4. LBP features are computed by utilizing third-level input of hash algorithm. And then the Hamming dis- resolution image. tance is compared to determine the effectiveness of 5. Retrieval feature is obtained by fusing bag-of-fea- various features. It is shown as shown in Figure 2. tures, texture feature, and LBP feature. Finally, three features are proved to be more effect- 6. Comparing retrieval features in between query ive for image retrieval in the medical image library. image and image of medical database. Bag-of-feature can better describe the local grayscale 7. Retrieval results are output by the order statistical information of medical images. Texture fea- of similarity. ture is the most abundant expression in medical image content. LBP feature has grayscale invariance and rotation invariance. It is also very important for 3. Multi-feature fusion retrieval method based medical image analysis. on wavelet and bag-of-feature 3.1. Feature selection by hash coding 3.2. Wavelet decomposition In order to increase the reliability of medical image retrieval, more image features should be extracted. The process of wavelet decomposition can be However, the medical image is generally large so that described as Figure.3. ð0Þ ð1Þ more time will be expended for computing these fea- Where LL is the original medical image, LL , ð1Þ ð1Þ ð1Þ tures on the original image. For this reason, wavelet LH , HL , HH are the results after one-time wave- ðkÞ ðkÞ ðkÞ ðkÞ decomposition was introduced to obtain multi-level let decomposition, and LL , LH , HL , HH are the resolution expression of the original image. Then, dif- results after k-time wavelet decomposition. ðkÞ ðkÞ ðkÞ ðkÞ ferent features would be obtained on different level From Figure.2, we can find LL , LH , HL , HH are the same-size sub-image and express partial infor- resolution image. ðk1Þ ðkÞ In order to get more reliable features of medical mation of LL . Here, LL has more information image retrieval, a certain number of images are ran- than other three components and it can be taken as ðk1Þ domly selected from the database as training images the most powerful expression of LL . From the ðkÞ ðk1Þ for feature extraction. The extracted features include image size, LL is a quarter of LL . When the same ðkÞ color features, bag-of-features, texture features, shape computation is implemented on LL , there will be ðk1Þ features, LBP local features, and so on. The feature less time cost than on LL . COMPUTER ASSISTED SURGERY 75 original image.A thoracic medical image (pixels size: 256  256) had been segmented 16 small images shown as Figure 4. The number of small images should be determined by the size of original image. In Figure 4, the size of thoracic image is 256  256 and the number of small image is 16. If the size of original image is bigger, the number of small image is also increased. After segmentation, the information of all small images can be described with a corresponding matrix ð0Þ Figure 4. Segmentation of LL image. just like formula (5). 2 3 B B B B 11 12 13 14 6 7 B B B B 21 22 23 24 6 7 B ¼ (5) If wavelet decomposition is taken on an image with 4 5 B B B B 31 32 33 34 the size M  N, the process can be described in a set B B B B 41 42 43 44 of mathematical formulae as follows: Where, B is a feature clustering function which ij hi ðÞ ðÞ k k1 can express corresponding small image. In this LL m; n ¼ LL  H  H (1) ðÞ rows 2#1columns 1#2 paper, general gray-statistic function had been k k selected as a feature computation formula. It is Here, m ¼ 1; :::; M=2 ; n ¼ 1; :::; N=2 . hi shown as below: ðÞ ðÞ k k1 LH m; n ¼ LL  H  G (2) ðÞ w c rows y y w c x x h 2#1columns X X 1#2 Ix þ t; y þ s  Mx; y  (6) ðÞ ðÞ k k1 k s¼c t¼c Here, m ¼ M=2 þ 1; :::; M=2 ; n ¼ 1; :::; N=2 . y x hi Where, w and w are the width and height of the ðÞ ðÞ x y k k1 HL m; n ¼ LL  G  H (3) ðÞ rows 2#1columns small image, (c , c ) is the coordinate of the center 1#2 x y pixel of small image, (x, y) is the coordinate of the k k k1 Here, m ¼ 1; :::; M=2 ; n ¼ N=2 þ 1; :::; N=2 . arbitrary pixel of small image. Mðx; yÞ can be com- hi ðÞ k ðÞ k1 puted as formula (7). HH ðÞ m; n ¼ LL  G  G (4) rows 2#1columns 1#2 w c y y w c x x X X Mx; y ¼ IiðÞ ; j (7) ðÞ Here, w  w y x i¼c i¼c k k1 k k1 y x m ¼ M=2 þ 1; :::; M=2 ; n ¼ N=2 þ 1; :::; N=2 . H and G represents the low-pass filter and high- pass filter after the wavelet decomposition respect- 3.4. Texture feature computation ively, 2#1（ 1#2） represents the sampling along the column/row, k represents the level of the wavelet Texture feature is more important to medical image decomposition. retrieval because that color information is not rich. In this paper, computation about texture feature is ð1Þ implemented on LL image. In order to obtain tex- 3.3. Bag-of-feature computation ture feature, gray level co-occurrence matrix (GLCM) is ð0Þ Original image can be looked on as LL image of used. GLCM characterizes the correlation between any wavelet decomposition, and it includes the true two gray level of the medical image. As a common ð0Þ information of disease. So LL imageplays themost means of describing image texture, GLCM can be nor- role of extracting feature during medical image malized to pðg ; g Þ. GLCM-based texture feature is i j ð0Þ retrieval. Because LL image has too large size, a generally divided into four kinds: long time will be cost for computing retrieval feature ð0Þ on it. Therefore, it is a good idea to segment LL 3.4.1. Energy image to several small images. And these images are Energy describes the uniformity of gray distribution so small that retrieval feature can be computed via about an image. If data is more concentrated near the less time. The information of every bag-of-feature obtained from small image will be used in forming a main diagonal, we can consider that gray distribution feature set which can express the true information of is relatively symmetrical. In this case, big energy value 76 L. SHUANG ET AL. shows that image texture is coarse. Energy can be From formula (13), energy, entropy, contrast, and computed according to formula (8). correlation is considered to have the same impact on the whole texture feature. G G XX T ¼ pg ; g (8) E ðÞ i j i¼1 j¼1 3.5. LBP feature computation LBP (local binary patterns) feature is also an important 3.4.2. Entropy feature which can be used in expressing image tex- Entropy is a measure of the amount of information. If ture. However, computing time of LBP feature will entropy value is large, we can consider that there is increase significantly when calculation neighborhood more texture information. Entropy can be computed increases. In this paper, LBP feature is computed on ð2Þ according to formula (9). LL image in order to spend save time. Take a local image of 3  3 pixels for an example, G G XX T ¼ pg ; g logpg ; g (9) LBP feature can be calculated as follows: H ðÞ i j ðÞ i j i¼1 j¼1 At first, center pixel fðx ; y Þ is selected as the pixel c c to be processed and its gray value is selected as a threshold. Other pixels can be binarized by comparing 3.4.3. Contrast their gray values with this threshold. This process is Contrast is a measure of the clarity of image texture. If shown as formula (14). texture feature is more clear, the value of contrast 1; g  g i c should be large. Contrast can be computed according sg ; g ¼ (14) ðÞ i c 0 g < g i c to formula (10). Where, g is the gray value of center pixel, g is other G G c i XX pixel in local image. T ¼ ðÞ i  j pg ; g (10) C ðÞ i j i¼1 j¼1 After binarization, all sðg ; g Þ will be combined into i c an 8-bit binary number and the decimal number of a binary number is LBP feature value of the center pixel. 3.4.4. Correlation This calculation process is just like formula (15). Correlation is the gray similarity about the row or col- umn elements of GLCM. Correlation can be computed LBP ¼ sg ; g 2 (15) ðÞ i c i¼0 according to formula (11). G G XX An LBP image can be obtained when LBP feature ijp g ; g  l l ðÞ i j x y T ¼ (11) calculation is carried out over the whole image and r r x y i¼1 j¼1 every pixel is replaced by LBP value. Because texture Where, l , l , r , and r are mean and variance of feature is more legible in LBP image, LBP value has x y x y GCLM. They can be computed by using formula (12). been selected as important retrieval feature. LBP results of a medical image and its histogram are G G > X X > shown as Figure 5. l ¼ g pg ; g > i ðÞ i j i¼1 j¼1 In fact, LBP feature computation can be extended G G > X X > to the neighborhood of arbitrary size. Because this l ¼ g pg ; g > j ðÞ i j y ð2Þ process is carried out on LL image in this paper, the j¼1 i¼1 (12) size of 3  3 pixels neighborhood is enough. G G X X 2 2 > r ¼ g  l pg ; g ðÞ i ðÞ i j > x > i¼1 j¼1 > G G 3.6. Feature fusion and similarity measurement X X r ¼ g  l pg ; g ðÞ j ðÞ i j y x Retrieval results can be obtained by comparing the j¼1 i¼1 similarity between query image and database image. In this paper, the whole texture feature is made up So an efficient similarity measure function is very of energy, entropy, contrast, and correlation, and it is important to image retrieval. In this paper, bag-of-fea- just like formula (13). ture, texture feature, and LBP feature should have a proper position in similarity measure function which is T ¼ T þ T þ T þ T (13) E H C R shown as formula (16). COMPUTER ASSISTED SURGERY 77 Figure 5. Computation result of LBP feature of a medical image. Figure 6. Experimental results for chest CT image. 78 L. SHUANG ET AL. S ¼jðÞ g B þ g T þ g L  1j (16) T can be calculated by the formula (18). B S T S L S S T T þ T þ T þ T D DE DH DC DR Where, B , T , and L represents the component of S S S T ¼ ¼ (18) T T þ T þ T þ T bag-of-feature, texture feature, and LBP feature, g , g , Q QE QH QC QR B T and g is the weight of B , T , and L . S S S The computation of L is shown as formula (19). B can be computed according to formula (17). Di w c y y w c x x P P L ¼ s log (19) S Qi I x þ t; y þ s  M x; y ðÞ Qi D DðÞ i¼1 s¼c t¼c y x B ¼ (17) S th w c y y w c x x Where, s and s represents the i probability of P P Qi Di I x þ t; y þ s  M x; y ðÞ ðÞ Q Q LBP histogram of query image and medical image s¼c t¼c y x of database. Where D represents the image in medical image data- Since three features were calculated on different base, Q represents query image. levels of wavelet image, they should have a different effect to similarity measure function. As an expression of original image information, bag-of-feature should occupy the most important position. LBP feature is more reliable than general image texture, but Figure 7. Experimental results for pelvic CT image. COMPUTER ASSISTED SURGERY 79 (a) Comparison of the accuracy of three methods 100% 95% 90% 85% 80% 75% Leg Hand Head Chest Pelvic Different parts of CT image (b) Comparison of retrieval time of three methods 1.1 1.0 0.9 0.8 0.7 0.6 Leg Hand Head Chest Pelvic Different parts of CT image Figure 8. Comparison of the accuracy and retrieval time of three methods. ð2Þ information of LL is not rich. Therefore, g , g , and including head CT, chest CT, pelvic CT, and spine CT. In B T g is given by 0.4, 0.3, and 0.3. the experiment, the key parameters are as follows: the bit number of hash codes is 32; layer number of wave- let decomposition is 2; Bags of image are 16; LBP tem- 4. Experiments and analysis platesizeis 3 3; fusion parameters g , g ,andg of B T L In order to test the performance of the retrieval algo- three features is given by 0.4, 0.3, and 0.3. rithm proposed in this paper, an experimental system Three experimental results are shown as Figure 6 had been built for medical images and some experi- and Figure 7. ments had been carried out. The medical image data From Figure 6 and Figure 7, wecan seethat set is selected from the CT Department of Harbin head CT, chest CT, and pelvic CT are selected as Medical University, including different CT images of all query image and retrieval results are placed order parts of the human body. Because of the confidentiality according to similarity. In fact, there are much more of the dataset, only a small amount of results can be kinds of medical images. Limited by the capability shown here. The primary processor of the system is a of this paper, many experimental results can not computer which has a 2.5 GHz two core Intel CPU and be shown. an 8 G RAM. The image retrieval software is coded with In order to compare retrieval performance between Cþþ language, and key retrieval technology is based the proposed method and others, we had carried out image retrieval by using three different methods. on multi-feature fusion algorithm proposed in this paper. The retrieval database has 1000 medical images Retrieval time Retrieval accuracy 80 L. SHUANG ET AL. The first method is the combination of wavelet Funding transform and energy extraction (WAVE) in the litera- This study was supported by International S&T Cooperation ture [7]. The second method is the multi-feature SVM Program of China [Grant No. 2014DFA70470] and National fusion method (SVM) in the literature [9], and the third Natural Science Foundation of China [Grant No. 61774107]. method is the multi-feature fusion method (OUR) pro- posed in this paper. References Comparison of the accuracy and retrieval time of [1] Li Z, Zhang X, Muller H, et al. Large-scale retrieval for the three methods is shown in Figure 8. medical image analytics: a comprehensive review[J]. From Figure 8, we can see that the proposed Med Image Anal. 2018;43:66–72. method has the highest retrieval accuracy because [2] MM, Rahman S, Antani GR. Thoma A classification- multi-feature was fused in it. Moreover, retrieval time driven similarity matching framework for retrieval of of the proposed method is not significantly increased biomedical images[C]. 11th ACM International because different feature had been computed on dif- Conference on Multimedia Information Retrieval, ferent wavelet- level image. Philadelphia, USA; 2010. p.147–154. [3] Shi X, Xing F, Xu K, et al. Supervised graph hashing Although our method takes a longer time than the for histopathology image retrieval and single feature method in the literature [7], our method classification[J]. Med Image Anal. 2017;42:117–126. is very close with the method of multi-feature fusion [4] Bai X, Yang X, Latechi J. Learning context sensitive in the literature [9]. According to the accuracy of com- shape similarity by graph transduction[J]. IEEE Tran prehensive retrieval, our method is obviously Pattern Anal Mach Intell 2010;32:861–874. more ideal. [5] Avni U, Greenspan H, Sharon M. X-ray image categor- ization and retrieval using patch-based visual words representation. Vol. 1. In Proceedings of IEEE 5. Conclusion International Symposium on Biomedical Imaging: From Nano to Macro, Rotterdam, Netherlands; 2009. Since a single feature can bring on false retrieval p. 350–353. results, much more reliable feature should be used for [6] Nowakova J. M. Prilepok, V. Snasel. Medical image medical image retrieval. However, retrieval time will retrieval using vector quantization and fuzzys-tree[J]. increase evidently when many features were com- J Med Sys 2017;41:18. puted during retrieval. In this paper, a multi-feature [7] K, Rajakumar S. Muttan. Medical image retrieval using fusion method was proposed for medical image energy efficient wavelet transform[C]. Second International Conference on Computing, Communication retrieval. At first, wavelet decomposition was used in and Networking Technologies, Wuzhen, China; 2010. obtaining multi-resolution expression of the original p. 1–5. image. Then, bag-of-feature, texture-feature, LBP fea- [8] Sasi Kumar M, Kumaraswamy YS. An improved sup- ture was computed on different resolution images. port vector machine kernel for medical image These three features were fused when retrieval similar- retrieval system[C]. Proceedings of the International ity was measured between the query image and Conference on Pattern Recognition, Informatics and image of the medical image database. Experimental Medical Engineering; 2012. p. 257–260. [9] Yonggang H, Jun Z, Yongwang Z, et al. Medical image results show that the proposed multi-feature fusion retrieval with query-dependent feature fusion based on method has a higher retrieval accuracy and its one-class SVM[C]. 13th IEEE International Conference retrieval time is not added comparing with those on Computational Science and Engineering, Faro, methods using a single feature. Portugal; 2010. p. 176–183. [10] Liu Y, Rahul S, Steven CH. A boosting framework for visuality preserving distance metric learning and its Disclosure statement application to medical image retrieval[J]. IEEE Trans No potential conflict of interest was reported by the authors. Pattern Anal Mach Intell 2010;32(1):30–44. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Computer Assisted Surgery Taylor & Francis

Multi-feature fusion method for medical image retrieval using wavelet and bag-of-features

Shuang, Liu; Deyun, Chen; Zhifeng, Chen; Ming, Pang
Computer Assisted Surgery , Volume 24 (sup1): 9 – Oct 1, 2019
Download PDF
9 pages

Loading next page...
 
/lp/taylor-francis/multi-feature-fusion-method-for-medical-image-retrieval-using-wavelet-QsZ6XzOzmw

References (10)

Publisher
Taylor & Francis
Copyright
© 2019 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.
ISSN
2469-9322
DOI
10.1080/24699322.2018.1560087
Publisher site
See Article on Publisher Site

Abstract

COMPUTER ASSISTED SURGERY 2019, VOL. 24, NO. S1, 72–80 https://doi.org/10.1080/24699322.2018.1560087 RESEARCH ARTICLE Multi-feature fusion method for medical image retrieval using wavelet and bag-of-features a,b a c d Liu Shuang , Chen Deyun , Chen Zhifeng and Pang Ming Center for Post-Doctoral Studies of Computer Science and Technology, Harbin University of Science and Technology, Harbin, Heilongjiang, China; College of Computer and Information Engineering, Harbin University of Commerce, Harbin, Heilongjiang, China; c d College of Energy and Architectural Engineering, Harbin University of Commerce, Harbin, Heilongjiang, China; College of Automation, Harbin Engineering University, Harbin, Heilongjiang, China KEYWORDS ABSTRACT Word; medical image Color, texture, and shape are the common features used for the retrieval systems. However, retrieval; bag-of-feature; many medical images have a spot of color information. Therefore, the discriminative texture and texture feature; LBP feature shape features should be extracted to obtain a satisfied retrieval result. In order to increase the credibility of the retrieval process, many features can be combined to be used for medical image retrieval. Meanwhile, more features require more processing time, which will decrease the retrieval speed. In this paper, wavelet decomposition is adopted to generate different reso- lution images. Bag-of-feature, texture, and LBP feature are extracted from three different-level wavelet images. Finally, the similarity measure function is obtained by fusing these three types of features. Experimental results show that the proposed multi-feature fusion method can achieve a higher retrieval accuracy with an acceptable retrieval time. 1. Introduction feature is characterized by combining texture with shape information [5]. Cardiac CT images are import- The medical content-based image retrieval (M-CBIR) ant parts of the medical database, and related systems have been developed and used for path- retrieval methods have been formed by taking ology, radiology, and clinical laboratory diagnostics advantage of the heart shape. A contour and texture [1]. In these systems, some databases only contain a based image retrieval technology has been put for- single kind of medical image, while others contain ward and applied to the liver image database [6]. many kinds of medical images. Radiology images not Patch-based features have been applied to X-ray only have a large number but also play an important medical image retrieval, while scale invariant feature role in auxiliary diagnosis. Therefore, the radiation transform (SIFT) has been selected as well as a new image retrieval system has become a hot research feature for medical image retrieval. area [2]. Since wavelet transformation can extend an image Since it iseasy toidentify color and texture in the into different size and obtain idiographic feature infor- pathological image, CBIR is often used in the mation, it is also introduced to medical image research of retrieval system of pathological images. retrieval. By combining Gabor filter and Euclidean dis- A computer-aided diagnosis system was proposed tance, wavelet transform has achieved better perform- for pigmented skin lesions and multiple classifier sys- ance on image retrieval. When features have been tems were used for melanoma diagnosis [3]. To selected, image retrieval needs good methods to clas- retrieve the medical image, the traditional global sify these features in order to obtain the best result features have been widely used including color fea- similar to query image [7]. Support vector machine is tures, texture features, and shape features. Compared with text retrieval, image retrieval often in common use for retrieval in medical image data- constructs visual words by utilizing all kinds of fea- base including X-Rays, MRI, and CT. IRMA is an effi- tures [4]. ASSERT is a retrieval system for lung CT cient image retrieval system using feature fusion and image. In the retrieval algorithm of ASSERT, lesion support vector machine [8]. CONTACT Pang Ming pangm@hrbeu.edu.cn College of Automation, Harbin Engineering University, Harbin, Heilongjiang, China 2019 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. COMPUTER ASSISTED SURGERY 73 Figure 1. Framework of the proposed method. To improve the performance of CBIR for a medical the proposed method is founded in Section 2.In image, machine learning has been used in pre-filtering Section 3, mathematical expression and execution image and statistical similarity has been used in the details of the proposed method are listed. matching of multi-feature between query image and Experiments are shown in Section 4 and conclusion is database [9]. UMLS is a successful medical image listed in Section 5. retrieval system which has a structured learning frame- work and modular design based on support vector 2. Retrieval framework of the machine. To find out the possible lesion region in proposed method brain images, domain knowledge has been used in retrieving the image sequence. A boosting framework The retrieval framework of the proposed method is has been proposed to improve the performance of shown as in Figure.1 medical image retrieval. Evaluation results show that From Figure 1, we can see that this method will be this method has a high retrieval accuracy with a low implemented by seven steps: computational cost [10]. 1. Different resolution images are obtained by using In this paper, a multi-feature fusion method was proposed for medical image retrieval based on wave- wavelet decomposition. let transform and bag-of-features. The remaining parts 2. Gray-based bag-of-features are computed accord- of this paper are organized as follows: Framework of ing to first-level resolution image. 74 L. SHUANG ET AL. Hash Image of Feature Feature Hash Code Coding Extraction Database Vectors Hanming Distance Feature Feature Hash Hash Code Query Image Extraction Coding Vectors Figure 2. Feature selection. (2) (2) LL LH (1) (1) (1) LL LH LH (2) (2) HL HH (0) LL (1) (1) (1) (1) HL HH HL HH Figure 3. Wavelet decomposition. 3. Texture features are extracted from second-level vectors of different dimensions are formed, and hash resolution image. coding is generated by using these vectors as the 4. LBP features are computed by utilizing third-level input of hash algorithm. And then the Hamming dis- resolution image. tance is compared to determine the effectiveness of 5. Retrieval feature is obtained by fusing bag-of-fea- various features. It is shown as shown in Figure 2. tures, texture feature, and LBP feature. Finally, three features are proved to be more effect- 6. Comparing retrieval features in between query ive for image retrieval in the medical image library. image and image of medical database. Bag-of-feature can better describe the local grayscale 7. Retrieval results are output by the order statistical information of medical images. Texture fea- of similarity. ture is the most abundant expression in medical image content. LBP feature has grayscale invariance and rotation invariance. It is also very important for 3. Multi-feature fusion retrieval method based medical image analysis. on wavelet and bag-of-feature 3.1. Feature selection by hash coding 3.2. Wavelet decomposition In order to increase the reliability of medical image retrieval, more image features should be extracted. The process of wavelet decomposition can be However, the medical image is generally large so that described as Figure.3. ð0Þ ð1Þ more time will be expended for computing these fea- Where LL is the original medical image, LL , ð1Þ ð1Þ ð1Þ tures on the original image. For this reason, wavelet LH , HL , HH are the results after one-time wave- ðkÞ ðkÞ ðkÞ ðkÞ decomposition was introduced to obtain multi-level let decomposition, and LL , LH , HL , HH are the resolution expression of the original image. Then, dif- results after k-time wavelet decomposition. ðkÞ ðkÞ ðkÞ ðkÞ ferent features would be obtained on different level From Figure.2, we can find LL , LH , HL , HH are the same-size sub-image and express partial infor- resolution image. ðk1Þ ðkÞ In order to get more reliable features of medical mation of LL . Here, LL has more information image retrieval, a certain number of images are ran- than other three components and it can be taken as ðk1Þ domly selected from the database as training images the most powerful expression of LL . From the ðkÞ ðk1Þ for feature extraction. The extracted features include image size, LL is a quarter of LL . When the same ðkÞ color features, bag-of-features, texture features, shape computation is implemented on LL , there will be ðk1Þ features, LBP local features, and so on. The feature less time cost than on LL . COMPUTER ASSISTED SURGERY 75 original image.A thoracic medical image (pixels size: 256  256) had been segmented 16 small images shown as Figure 4. The number of small images should be determined by the size of original image. In Figure 4, the size of thoracic image is 256  256 and the number of small image is 16. If the size of original image is bigger, the number of small image is also increased. After segmentation, the information of all small images can be described with a corresponding matrix ð0Þ Figure 4. Segmentation of LL image. just like formula (5). 2 3 B B B B 11 12 13 14 6 7 B B B B 21 22 23 24 6 7 B ¼ (5) If wavelet decomposition is taken on an image with 4 5 B B B B 31 32 33 34 the size M  N, the process can be described in a set B B B B 41 42 43 44 of mathematical formulae as follows: Where, B is a feature clustering function which ij hi ðÞ ðÞ k k1 can express corresponding small image. In this LL m; n ¼ LL  H  H (1) ðÞ rows 2#1columns 1#2 paper, general gray-statistic function had been k k selected as a feature computation formula. It is Here, m ¼ 1; :::; M=2 ; n ¼ 1; :::; N=2 . hi shown as below: ðÞ ðÞ k k1 LH m; n ¼ LL  H  G (2) ðÞ w c rows y y w c x x h 2#1columns X X 1#2 Ix þ t; y þ s  Mx; y  (6) ðÞ ðÞ k k1 k s¼c t¼c Here, m ¼ M=2 þ 1; :::; M=2 ; n ¼ 1; :::; N=2 . y x hi Where, w and w are the width and height of the ðÞ ðÞ x y k k1 HL m; n ¼ LL  G  H (3) ðÞ rows 2#1columns small image, (c , c ) is the coordinate of the center 1#2 x y pixel of small image, (x, y) is the coordinate of the k k k1 Here, m ¼ 1; :::; M=2 ; n ¼ N=2 þ 1; :::; N=2 . arbitrary pixel of small image. Mðx; yÞ can be com- hi ðÞ k ðÞ k1 puted as formula (7). HH ðÞ m; n ¼ LL  G  G (4) rows 2#1columns 1#2 w c y y w c x x X X Mx; y ¼ IiðÞ ; j (7) ðÞ Here, w  w y x i¼c i¼c k k1 k k1 y x m ¼ M=2 þ 1; :::; M=2 ; n ¼ N=2 þ 1; :::; N=2 . H and G represents the low-pass filter and high- pass filter after the wavelet decomposition respect- 3.4. Texture feature computation ively, 2#1（ 1#2） represents the sampling along the column/row, k represents the level of the wavelet Texture feature is more important to medical image decomposition. retrieval because that color information is not rich. In this paper, computation about texture feature is ð1Þ implemented on LL image. In order to obtain tex- 3.3. Bag-of-feature computation ture feature, gray level co-occurrence matrix (GLCM) is ð0Þ Original image can be looked on as LL image of used. GLCM characterizes the correlation between any wavelet decomposition, and it includes the true two gray level of the medical image. As a common ð0Þ information of disease. So LL imageplays themost means of describing image texture, GLCM can be nor- role of extracting feature during medical image malized to pðg ; g Þ. GLCM-based texture feature is i j ð0Þ retrieval. Because LL image has too large size, a generally divided into four kinds: long time will be cost for computing retrieval feature ð0Þ on it. Therefore, it is a good idea to segment LL 3.4.1. Energy image to several small images. And these images are Energy describes the uniformity of gray distribution so small that retrieval feature can be computed via about an image. If data is more concentrated near the less time. The information of every bag-of-feature obtained from small image will be used in forming a main diagonal, we can consider that gray distribution feature set which can express the true information of is relatively symmetrical. In this case, big energy value 76 L. SHUANG ET AL. shows that image texture is coarse. Energy can be From formula (13), energy, entropy, contrast, and computed according to formula (8). correlation is considered to have the same impact on the whole texture feature. G G XX T ¼ pg ; g (8) E ðÞ i j i¼1 j¼1 3.5. LBP feature computation LBP (local binary patterns) feature is also an important 3.4.2. Entropy feature which can be used in expressing image tex- Entropy is a measure of the amount of information. If ture. However, computing time of LBP feature will entropy value is large, we can consider that there is increase significantly when calculation neighborhood more texture information. Entropy can be computed increases. In this paper, LBP feature is computed on ð2Þ according to formula (9). LL image in order to spend save time. Take a local image of 3  3 pixels for an example, G G XX T ¼ pg ; g logpg ; g (9) LBP feature can be calculated as follows: H ðÞ i j ðÞ i j i¼1 j¼1 At first, center pixel fðx ; y Þ is selected as the pixel c c to be processed and its gray value is selected as a threshold. Other pixels can be binarized by comparing 3.4.3. Contrast their gray values with this threshold. This process is Contrast is a measure of the clarity of image texture. If shown as formula (14). texture feature is more clear, the value of contrast 1; g  g i c should be large. Contrast can be computed according sg ; g ¼ (14) ðÞ i c 0 g < g i c to formula (10). Where, g is the gray value of center pixel, g is other G G c i XX pixel in local image. T ¼ ðÞ i  j pg ; g (10) C ðÞ i j i¼1 j¼1 After binarization, all sðg ; g Þ will be combined into i c an 8-bit binary number and the decimal number of a binary number is LBP feature value of the center pixel. 3.4.4. Correlation This calculation process is just like formula (15). Correlation is the gray similarity about the row or col- umn elements of GLCM. Correlation can be computed LBP ¼ sg ; g 2 (15) ðÞ i c i¼0 according to formula (11). G G XX An LBP image can be obtained when LBP feature ijp g ; g  l l ðÞ i j x y T ¼ (11) calculation is carried out over the whole image and r r x y i¼1 j¼1 every pixel is replaced by LBP value. Because texture Where, l , l , r , and r are mean and variance of feature is more legible in LBP image, LBP value has x y x y GCLM. They can be computed by using formula (12). been selected as important retrieval feature. LBP results of a medical image and its histogram are G G > X X > shown as Figure 5. l ¼ g pg ; g > i ðÞ i j i¼1 j¼1 In fact, LBP feature computation can be extended G G > X X > to the neighborhood of arbitrary size. Because this l ¼ g pg ; g > j ðÞ i j y ð2Þ process is carried out on LL image in this paper, the j¼1 i¼1 (12) size of 3  3 pixels neighborhood is enough. G G X X 2 2 > r ¼ g  l pg ; g ðÞ i ðÞ i j > x > i¼1 j¼1 > G G 3.6. Feature fusion and similarity measurement X X r ¼ g  l pg ; g ðÞ j ðÞ i j y x Retrieval results can be obtained by comparing the j¼1 i¼1 similarity between query image and database image. In this paper, the whole texture feature is made up So an efficient similarity measure function is very of energy, entropy, contrast, and correlation, and it is important to image retrieval. In this paper, bag-of-fea- just like formula (13). ture, texture feature, and LBP feature should have a proper position in similarity measure function which is T ¼ T þ T þ T þ T (13) E H C R shown as formula (16). COMPUTER ASSISTED SURGERY 77 Figure 5. Computation result of LBP feature of a medical image. Figure 6. Experimental results for chest CT image. 78 L. SHUANG ET AL. S ¼jðÞ g B þ g T þ g L  1j (16) T can be calculated by the formula (18). B S T S L S S T T þ T þ T þ T D DE DH DC DR Where, B , T , and L represents the component of S S S T ¼ ¼ (18) T T þ T þ T þ T bag-of-feature, texture feature, and LBP feature, g , g , Q QE QH QC QR B T and g is the weight of B , T , and L . S S S The computation of L is shown as formula (19). B can be computed according to formula (17). Di w c y y w c x x P P L ¼ s log (19) S Qi I x þ t; y þ s  M x; y ðÞ Qi D DðÞ i¼1 s¼c t¼c y x B ¼ (17) S th w c y y w c x x Where, s and s represents the i probability of P P Qi Di I x þ t; y þ s  M x; y ðÞ ðÞ Q Q LBP histogram of query image and medical image s¼c t¼c y x of database. Where D represents the image in medical image data- Since three features were calculated on different base, Q represents query image. levels of wavelet image, they should have a different effect to similarity measure function. As an expression of original image information, bag-of-feature should occupy the most important position. LBP feature is more reliable than general image texture, but Figure 7. Experimental results for pelvic CT image. COMPUTER ASSISTED SURGERY 79 (a) Comparison of the accuracy of three methods 100% 95% 90% 85% 80% 75% Leg Hand Head Chest Pelvic Different parts of CT image (b) Comparison of retrieval time of three methods 1.1 1.0 0.9 0.8 0.7 0.6 Leg Hand Head Chest Pelvic Different parts of CT image Figure 8. Comparison of the accuracy and retrieval time of three methods. ð2Þ information of LL is not rich. Therefore, g , g , and including head CT, chest CT, pelvic CT, and spine CT. In B T g is given by 0.4, 0.3, and 0.3. the experiment, the key parameters are as follows: the bit number of hash codes is 32; layer number of wave- let decomposition is 2; Bags of image are 16; LBP tem- 4. Experiments and analysis platesizeis 3 3; fusion parameters g , g ,andg of B T L In order to test the performance of the retrieval algo- three features is given by 0.4, 0.3, and 0.3. rithm proposed in this paper, an experimental system Three experimental results are shown as Figure 6 had been built for medical images and some experi- and Figure 7. ments had been carried out. The medical image data From Figure 6 and Figure 7, wecan seethat set is selected from the CT Department of Harbin head CT, chest CT, and pelvic CT are selected as Medical University, including different CT images of all query image and retrieval results are placed order parts of the human body. Because of the confidentiality according to similarity. In fact, there are much more of the dataset, only a small amount of results can be kinds of medical images. Limited by the capability shown here. The primary processor of the system is a of this paper, many experimental results can not computer which has a 2.5 GHz two core Intel CPU and be shown. an 8 G RAM. The image retrieval software is coded with In order to compare retrieval performance between Cþþ language, and key retrieval technology is based the proposed method and others, we had carried out image retrieval by using three different methods. on multi-feature fusion algorithm proposed in this paper. The retrieval database has 1000 medical images Retrieval time Retrieval accuracy 80 L. SHUANG ET AL. The first method is the combination of wavelet Funding transform and energy extraction (WAVE) in the litera- This study was supported by International S&T Cooperation ture [7]. The second method is the multi-feature SVM Program of China [Grant No. 2014DFA70470] and National fusion method (SVM) in the literature [9], and the third Natural Science Foundation of China [Grant No. 61774107]. method is the multi-feature fusion method (OUR) pro- posed in this paper. References Comparison of the accuracy and retrieval time of [1] Li Z, Zhang X, Muller H, et al. Large-scale retrieval for the three methods is shown in Figure 8. medical image analytics: a comprehensive review[J]. From Figure 8, we can see that the proposed Med Image Anal. 2018;43:66–72. method has the highest retrieval accuracy because [2] MM, Rahman S, Antani GR. Thoma A classification- multi-feature was fused in it. Moreover, retrieval time driven similarity matching framework for retrieval of of the proposed method is not significantly increased biomedical images[C]. 11th ACM International because different feature had been computed on dif- Conference on Multimedia Information Retrieval, ferent wavelet- level image. Philadelphia, USA; 2010. p.147–154. [3] Shi X, Xing F, Xu K, et al. Supervised graph hashing Although our method takes a longer time than the for histopathology image retrieval and single feature method in the literature [7], our method classification[J]. Med Image Anal. 2017;42:117–126. is very close with the method of multi-feature fusion [4] Bai X, Yang X, Latechi J. Learning context sensitive in the literature [9]. According to the accuracy of com- shape similarity by graph transduction[J]. IEEE Tran prehensive retrieval, our method is obviously Pattern Anal Mach Intell 2010;32:861–874. more ideal. [5] Avni U, Greenspan H, Sharon M. X-ray image categor- ization and retrieval using patch-based visual words representation. Vol. 1. In Proceedings of IEEE 5. Conclusion International Symposium on Biomedical Imaging: From Nano to Macro, Rotterdam, Netherlands; 2009. Since a single feature can bring on false retrieval p. 350–353. results, much more reliable feature should be used for [6] Nowakova J. M. Prilepok, V. Snasel. Medical image medical image retrieval. However, retrieval time will retrieval using vector quantization and fuzzys-tree[J]. increase evidently when many features were com- J Med Sys 2017;41:18. puted during retrieval. In this paper, a multi-feature [7] K, Rajakumar S. Muttan. Medical image retrieval using fusion method was proposed for medical image energy efficient wavelet transform[C]. Second International Conference on Computing, Communication retrieval. At first, wavelet decomposition was used in and Networking Technologies, Wuzhen, China; 2010. obtaining multi-resolution expression of the original p. 1–5. image. Then, bag-of-feature, texture-feature, LBP fea- [8] Sasi Kumar M, Kumaraswamy YS. An improved sup- ture was computed on different resolution images. port vector machine kernel for medical image These three features were fused when retrieval similar- retrieval system[C]. Proceedings of the International ity was measured between the query image and Conference on Pattern Recognition, Informatics and image of the medical image database. Experimental Medical Engineering; 2012. p. 257–260. [9] Yonggang H, Jun Z, Yongwang Z, et al. Medical image results show that the proposed multi-feature fusion retrieval with query-dependent feature fusion based on method has a higher retrieval accuracy and its one-class SVM[C]. 13th IEEE International Conference retrieval time is not added comparing with those on Computational Science and Engineering, Faro, methods using a single feature. Portugal; 2010. p. 176–183. [10] Liu Y, Rahul S, Steven CH. A boosting framework for visuality preserving distance metric learning and its Disclosure statement application to medical image retrieval[J]. IEEE Trans No potential conflict of interest was reported by the authors. Pattern Anal Mach Intell 2010;32(1):30–44.

Journal

Computer Assisted Surgery – Taylor & Francis

Published: Oct 1, 2019

Keywords: Word; medical image retrieval; bag-of-feature; texture feature; LBP feature

There are no references for this article.