Name
Title | ||
|---|---|---|
Evaluating music recommendation in a real-world setting: On data splitting and evaluation metrics |
Abstract | ||
|---|---|---|
Evaluation is important to assess the performance of a computer system in fulfilling a certain user need. In the context of recommendation, researchers usually evaluate the performance of a recommender system by holding out a random subset of observed ratings and calculating the accuracy of the system in reproducing such ratings. This evaluation strategy, however, does not consider the fact that in a real-world setting we are actually given the observed ratings of the past and have to predict for the future. There might be new songs, which create the cold-start problem, and the users' musical preference might change over time. Moreover, the user satisfaction of a recommender system may be related to factors other than accuracy. In light of these observations, we propose in this paper a novel evaluation framework that uses various time-based data splitting methods and evaluation metrics to assess the performance of recommender systems. Using millions of listening records collected from a commercial music streaming service, we compare the performance of collaborative filtering (CF) and content-based (CB) models with low-level audio features and semantic audio descriptors. Our evaluation shows that the CB model with semantic descriptors obtains a better trade-off among accuracy, novelty, diversity, freshness and popularity, and can nicely deal with the cold-start problems of new songs. |
Year | DOI | Venue |
|---|---|---|
2015 | 10.1109/ICME.2015.7177456 | 2015 IEEE International Conference on Multimedia and Expo (ICME) |
Keywords | Field | DocType |
Collaborative filtering,content-based recommendation,cold-start,data splitting,evaluation metrics | Recommender system,Mel-frequency cepstrum,Evaluation strategy,Collaborative filtering,Computer science,Popularity,Active listening,Artificial intelligence,Novelty,Machine learning,Encoding (memory) | Conference |
ISSN | Citations | PageRank |
1945-7871 | 3 | 0.42 |
References | Authors | |
21 | 3 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Szu-Yu Chou | 1 | 49 | 6.82 |
| Yi-Hsuan Yang | 2 | 1022 | 84.71 |
| Yu-Ching Lin | 3 | 389 | 28.19 |