Paper Info
Title
Study of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera Content
Abstract
We need data sets of images and subjective scores to develop robust no reference (or blind) visual quality metrics for consumer applications. These applications have many uncontrolled variables because the camera creates the original media and the impairment simultaneously. We do not fully understand how this impacts the integrity of our subjective data. We put forward two new data sets of images from consumer cameras. The first data set, CCRIQ2, uses a strict experiment design, more suitable for camera performance evaluation. The second data set, VIME1, uses a loose experiment design that resembles the behavior of consumer photographers. We gather subjective scores through a subjective experiment with 24 participants using the Absolute Category Rating method. We make these two new data sets available royalty-free on the Consumer Digital Video Library. We also present their integrity analysis (proposing one new approach) and explore the possibility of combining CCRIQ2 with its legacy counterpart. We conclude that the loose experiment design yields unreliable data, despite adhering to international recommendations. This suggests that the classical subjective study design may not be suitable for studies using consumer content. Finally, we show that Hossfeld-Schatz-Egger alpha failed to detect important differences between the two data sets.
Year
DOI
Venue
2020
10.3390/jimaging6030007
JOURNAL OF IMAGING
Keywords
DocType
Volume
image quality,data integrity,consumer camera,blind quality assessment,evaluation,subjective data,no reference,NR metrics,subjective study
Journal
6
Issue
ISSN
Citations 
3
2313-433X
0
PageRank 
References 
Authors
0.34
0
4
Name
Order
Citations
PageRank
Jakub Nawala101.35
Margaret H. Pinson200.34
Mikołaj Leszczuk300.34
Lucjan Janowski431331.55