Paper Info
Title
MetricQ: A Scalable Infrastructure for Processing High-Resolution Time Series Data
Abstract
In this paper we present MetricQ, a novel infrastructure for collecting, archiving, and analyzing sensor data. Core components of MetricQ are a scalable message broker based on the Advanced Message Queuing Protocol, and a newly developed Hierarchical Timeline Aggregation (HTA) storage concept that is specifically designed for timeseries data. HTA requires moderate data processing during data collection and a storage space overhead of about 10 %, and in turn reduces the complexity of typical timeline request from O(N) to O(1). This enables access to very large metric timelines spanning years and billions of data points at a performance level that is sufficient for interactive use cases. In contrast to existing solutions in this domain, no relevant information such as very short peaks in the data is discarded. We demonstrate how we use MetricQ with few metrics at very high update rates, e.g., for energy efficiency research, and for a very large number of metrics at moderate update rates, e.g., monitoring data from the electrical and cooling infrastructure of our data center.
Year
DOI
Venue
2019
10.1109/DAAC49578.2019.00007
2019 IEEE/ACM Industry/University Joint International Workshop on Data-center Automation, Analytics, and Control (DAAC)
Keywords
Field
DocType
MetricQ,high-resolution time series data,message broker,Advanced Message Queuing Protocol,data processing,data collection,electrical cooling infrastructure,data center,Hierarchical Timeline Aggregation storage
Data point,Data collection,Data processing,Efficient energy use,Computer science,Computer network,Message broker,Message queue,Data center,Scalability
Conference
ISBN
Citations 
PageRank 
978-1-7281-5992-8
2
0.44
References 
Authors
6
5
Name
Order
Citations
PageRank
Thomas Ilsche117214.92
Daniel Hackenberg240028.07
Robert Schöne323519.27
Mario Höpfner420.44
Wolfgang E. Nagel51800167.93