Title
Statistical and Visual Analysis of Audio, Text, and Image Features for Multi-Modal Music Genre Recognition
Abstract
We present a multi-modal genre recognition framework that considers the modalities audio, text, and image by features extracted from audio signals, album cover images, and lyrics of music tracks. In contrast to pure learning of features by a neural network as done in the related work, handcrafted features designed for a respective modality are also integrated, allowing for higher interpretability of created models and further theoretical analysis of the impact of individual features on genre prediction. Genre recognition is performed by binary classification of a music track with respect to each genre based on combinations of elementary features. For feature combination a two-level technique is used, which combines aggregation into fixed-length feature vectors with confidence-based fusion of classification results. Extensive experiments have been conducted for three classifier models (Naive Bayes, Support Vector Machine, and Random Forest) and numerous feature combinations. The results are presented visually, with data reduction for improved perceptibility achieved by multi-objective analysis and restriction to non-dominated data. Feature- and classifier-related hypotheses are formulated based on the data, and their statistical significance is formally analyzed. The statistical analysis shows that the combination of two modalities almost always leads to a significant increase of performance and the combination of three modalities in several cases.
Year
DOI
Venue
2021
10.3390/e23111502
ENTROPY
Keywords
DocType
Volume
music genre recognition, multi-modal classification, feature evaluation, audio signal features, album cover images, lyrics
Journal
23
Issue
ISSN
Citations 
11
1099-4300
0
PageRank 
References 
Authors
0.34
0
3
Name
Order
Citations
PageRank
Ben Wilkes100.34
Igor Vatolkin200.34
Heinrich Müller300.34