Paper Info
Title
CloudETL: scalable dimensional ETL for hive
Abstract
Extract-Transform-Load (ETL) programs process data into data warehouses (DWs). Rapidly growing data volumes demand systems that scale out. Recently, much attention has been given to MapReduce for parallel handling of massive data sets in cloud environments. Hive is the most widely used RDBMS-like system for DWs on MapReduce and provides scalable analytics. It is, however, challenging to do proper dimensional ETL processing with Hive; e.g., the concept of slowly changing dimensions (SCDs) is not supported (and due to lacking support for UPDATEs, SCDs are complex to handle manually). Also the powerful Pig platform for data processing on MapReduce does not support such dimensional ETL processing. To remedy this, we present the ETL framework CloudETL which uses Hadoop to parallelize ETL execution and to process data into Hive. The user defines the ETL process by means of high-level constructs and transformations and does not have to worry about technical MapReduce details. CloudETL supports different dimensional concepts such as star schemas and SCDs. We present how CloudETL works and uses different performance optimizations including a purpose-specific data placement policy to co-locate data. Further, we present a performance study and compare with other cloud-enabled systems. The results show that CloudETL scales very well and outperforms the dimensional ETL capabilities of Hive both with respect to performance and programmer productivity. For example, Hive uses 3.9 times as long to load an SCD in an experiment and needs 112 statements while CloudETL only needs 4.
Year
DOI
Venue
2014
10.1145/2628194.2628249
IDEAS
Keywords
Field
DocType
algorithms,data warehouse and repository,experimentation,parallel programming,mapreduce,etl,hive
Data warehouse,Data mining,Data set,Data processing,Programmer,Star schema,Computer science,Analytics,Database,Scalability,Cloud computing
Conference
Citations 
PageRank 
References 
9
0.61
19
Authors
3
Name
Order
Citations
PageRank
Xiufeng Liu110814.69
Christian Thomsen2282.25
Torben Bach Pedersen32102181.24