Paper Info
Title
Efficient Incremental Data Analytics With Apache Spark
Abstract
As smart electricity meters are becoming more popular and starting to replace conventional meters worldwide, new area of research for meter data analytics has emerged. Wide spectrum of computations in this context has been applied, ranging from computationally inexpensive tasks such as calculating monthly bills and peak usage, to elaborate computations to provide energy saving feedback to consumers in order to reduce peak energy demand. Examples include model building approaches for usage predictions and recommendations. Although research efforts in this field are progressing, majority of research in this domain still has overlooked the incremental aspects of energy data analytics, or in best cases, researches have not been able to properly utilize the incremental nature of the energy data. We have noticed that incremental approaches can significantly improve performance of smart meter analytics. For example, per-hour readings of a smart meter can efficiently become integrated with the previous readings and result in an incremental re-computation of a particular smart meter task.In this paper, we introduce UW Incremental Spark Analytics (UWISA), our incremental smart meter data platform, which applies efficient incremental techniques for calculating "energy-temperature" model (also called three-line model) [ 9]. Our platform can achieve better multi-core scalability and speedup of 4.5x (on average) compared to non-incremental implementation and speedup of higher than 2x when compared to previous incremental research for smart meter datasets up to tens of GBs. We also investigate the reasons behind better performance of incremental method when compared to the non-incremental and Spark Streaming approaches.
Year
DOI
Venue
2017
10.1109/BigData.2017.8258254
2017 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA)
Keywords
DocType
ISSN
Smart meters, Apache Spark, incremental analytics, energy-temperature model, batch processing, Hadoop file system
Conference
2639-1589
Citations 
PageRank 
References 
0
0.34
0
Authors
3
Name
Order
Citations
PageRank
Sina Gholamian1132.25
Wojciech Golab221017.22
Paul A. S. Ward313211.22