Paper Info
Title
A new approach to cross-modal multimedia retrieval
Abstract
The problem of joint modeling the text and image components of multimedia documents is studied. The text component is represented as a sample from a hidden topic model, learned with latent Dirichlet allocation, and images are represented as bags of visual (SIFT) features. Two hypotheses are investigated: that 1) there is a benefit to explicitly modeling correlations between the two components, and 2) this modeling is more effective in feature spaces with higher levels of abstraction. Correlations between the two components are learned with canonical correlation analysis. Abstraction is achieved by representing text and images at a more general, semantic level. The two hypotheses are studied in the context of the task of cross-modal document retrieval. This includes retrieving the text that most closely matches a query image, or retrieving the images that most closely match a query text. It is shown that accounting for cross-modal correlations and semantic abstraction both improve retrieval accuracy. The cross-modal model is also shown to outperform state-of-the-art image retrieval systems on a unimodal retrieval task.
Year
DOI
Venue
2010
10.1145/1873951.1873987
ACM Multimedia 2001
Keywords
Field
DocType
retrieval accuracy,text component,state-of-the-art image retrieval system,query image,image component,unimodal retrieval task,cross-modal document retrieval,cross-modal model,multimedia retrieval,new approach,query text,cross-modal correlation,latent dirichlet allocation,multimedia,image retrieval,feature space,document retrieval,document processing,canonical correlation analysis
Latent Dirichlet allocation,Computer science,Canonical correlation,Image retrieval,Explicit semantic analysis,Natural language processing,Artificial intelligence,Document retrieval,Information retrieval,Topic model,Multimedia,Concept search,Visual Word
Conference
Citations 
PageRank 
References 
503
11.41
24
Authors
7
Search Limit
100503
Name
Order
Citations
PageRank
N Rasiwasia1117334.61
Jose Costa Pereira268717.58
Emanuele Coviello379823.59
Gabriel Doyle472321.91
Gert R. G. Lanckriet54769296.98
Roger Levy699667.40
Nuno Vasconcelos75410273.99