Paper Info
Title
Multimodal medical image retrieval: image categorization to improve search precision
Abstract
Effective medical image retrieval can be useful in the clinical care of patients, education and research. Traditionally, image retrieval systems have been text-based, relying on the annotations or captions associated with the images. Although text-based information retrieval methods are mature and well researched, they are limited by the quality and availability of the annotations associated with the images. Advances in computer vision have led to methods for using the image itself as the search entity. However, the success of purely content-based techniques, when applied to a diverse set of clinical images, has been somewhat limited and these systems have not had much success in the medical domain. On the other hand, as demonstrated in recent years, a combination of text-based and content-based image retrieval techniques can achieve improved retrieval performance if combined effectively. There are many approaches to multimodal retrieval including early and late fusion of weighed results from the different search engines. In this work, we use automatic annotation based on visual attributes to label images as part of the indexing process and the subsequently use these labels to filter or reorder the results during the retrieval process. Labels for medical images can be categorized along three dimensions - imaging modality, anatomical location and image finding or pathology. Our previous research has indicated that the imaging modality is most easily identified using visual techniques whereas the caption or textual annotation frequently contains the finding or pathological information about the image. Thus, it is best to use visual methods to filter the modality and occasionally, anatomy while it is better to use the textual annotation to find the finding of interest. We have created a modality classifier for the weakly labeled images in our collection using a novel approach that combines affinity propagation for the selection of class exemplars, textons and patch-based descriptors as visual features and a NaiveBayes Nearest Neighbor technique for the classification of modality using visual features. We demonstrate significant improvement in precision attained using this technique for the ImageCLEF medical retrieval task 2009 using both our textual runs as well as runs from all participants in 2009
Year
DOI
Venue
2010
10.1145/1743384.1743415
Multimedia Information Retrieval
Keywords
Field
DocType
image categorization,search precision,effective medical image retrieval,text-based information retrieval method,imageclef medical retrieval task,textual annotation,content-based image retrieval technique,retrieval process,improved retrieval performance,imaging modality,visual feature,image retrieval system,multimodal medical image retrieval,information retrieval,affinity propagation,machine learning,search engine,three dimensions,image retrieval,nearest neighbor,indexation,classification,computer vision
Annotation,Automatic image annotation,Human–computer information retrieval,Information retrieval,Affinity propagation,Pattern recognition,Computer science,Image retrieval,Search engine indexing,Artificial intelligence,Classifier (linguistics),Visual Word
Conference
Citations 
PageRank 
References 
11
0.57
9
Authors
2
Name
Order
Citations
PageRank
Jayashree Kalpathy-Cramer184361.53
William Hersh22491307.00