Name
Abstract | ||
|---|---|---|
In this paper, we propose a novel method that exploits correlation between audio-visual dynamics of a video to segment and localize objects that are the dominant source of audio. Our approach consists of a two-step spatiotemporal segmentation mechanism that relies on velocity and acceleration of moving objects as visual features. Each frame of the video is segmented into regions based on motion and appearance cues using the QuickShift algorithm, which are then clustered over time using K-means, so as to obtain a spatiotemporal video segmentation. The video is represented by motion features computed over individual segments. The Mel-Frequency Cepstral Coefficients (MFCC) of the audio signal, and their first order derivatives are exploited to represent audio. The proposed framework assumes there is a non-trivial correlation between these audio features and the velocity and acceleration of the moving and sounding objects. The canonical correlation analysis (CCA) is utilized to identify the moving objects which are most correlated to the audio signal. In addition to moving-sounding object identification, the same framework is also exploited to solve the problem of audio-video synchronization, and is used to aid interactive segmentation. We evaluate the performance of our proposed method on challenging videos. Our experiments demonstrate significant increase in performance over the state-of-the-art both qualitatively and quantitatively, and validate the feasibility and superiority of our approach. |
Year | DOI | Venue |
|---|---|---|
2013 | 10.1109/TMM.2012.2228476 | IEEE Transactions on Multimedia |
Keywords | Field | DocType |
audio signal processing,image motion analysis,image representation,image segmentation,learning (artificial intelligence),object detection,pattern clustering,statistical analysis,synchronisation,video signal processing,K-means clustering,Mel-frequency cepstral coefficients,QuickShift algorithm,appearance cue,audio dominant source,audio signal correlation,audio-video synchronization,audio-visual dynamics,canonical correlation analysis,motion cue,motion feature representation,moving object acceleration,moving object velocity,moving-sounding object,multimodal analysis,object identification,object segmentation,two-step spatiotemporal segmentation mechanism,video dynamics,Audio-visual analysis,audio-visual synchronization,canonical correlation analysis,video segmentation | Mel-frequency cepstrum,Audio signal,Object detection,Computer vision,Synchronization,Pattern recognition,Computer science,Segmentation,Image segmentation,Artificial intelligence,Acceleration,Audio signal processing | Journal |
Volume | Issue | ISSN |
15 | 2 | 1520-9210 |
Citations | PageRank | References |
17 | 0.84 | 20 |
Authors | ||
3 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Hamid Izadinia | 1 | 164 | 11.16 |
| Imran Saleemi | 2 | 425 | 15.76 |
| Mubarak Shah | 3 | 16522 | 943.74 |