Name
Abstract | ||
|---|---|---|
Until recently, QoE (Quality of Experience) experiments had to be conducted in academic laboratories; however, with the advent of ubiquitous Internet access, it is now possible to ask an Internet crowd to conduct experiments on their personal computers. Since such a crowd can be quite large, crowdsourcing enables researchers to conduct experiments with a more diverse set of participants at a lower economic cost than would be possible under laboratory conditions. However, because participants carry out experiments without supervision, they may give erroneous feedback perfunctorily, carelessly, or dishonestly, even if they receive a reward for each experiment. In this paper, we propose a crowdsourceable framework to quantify the QoE of multimedia content. The advantages of our framework over traditional MOS ratings are: 1) it enables crowdsourcing because it supports systematic verification of participants' inputs; 2) the rating procedure is simpler than that of MOS, so there is less burden on participants; and 3) it derives interval-scale scores that enable subsequent quantitative analysis and QoE provisioning. We conducted four case studies, which demonstrated that, with our framework, researchers can outsource their QoE evaluation experiments to an Internet crowd without risking the quality of the results; and at the same time, obtain a higher level of participant diversity at a lower monetary cost. |
Year | DOI | Venue |
|---|---|---|
2009 | 10.1145/1631272.1631339 | ACM Multimedia 2001 |
Keywords | Field | DocType |
crowdsourceable qoe evaluation framework,academic laboratory,multimedia content,lower monetary cost,internet crowd,crowdsourceable framework,traditional mos rating,ubiquitous internet access,qoe evaluation experiment,case study,lower economic cost,diverse set,quantitative analysis,crowdsourcing,internet access,paired comparison | Ask price,Computer science,Crowdsourcing,Outsourcing,Provisioning,Quality of experience,Economic cost,Internet access,Multimedia,The Internet | Conference |
ISSN | Citations | PageRank |
1792-4308 | 92 | 6.82 |
References | Authors | |
15 | 4 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Kuan-Ta Chen | 1 | 1896 | 136.86 |
| Chen-Chi Wu | 2 | 238 | 17.12 |
| Yu-Chun Chang | 3 | 448 | 25.55 |
| Chin-Laung Lei | 4 | 1686 | 201.07 |