Name
Abstract | ||
|---|---|---|
Hadoop is a very popular general purpose framework for many different classes of data-intensive applications. However, it is not good for iterative operations because of the cost paid for the data reloading from disk at each iteration. As an emerging framework, Spark, which is designed to have a global cache mechanism, can achieve better performance in response time since the in-memory access over the distributed machines of cluster will proceed during the entire iterative process. Although the performance on time has been evaluated for Spark over Hadoop [1], the memory consumption, another system performance criteria, is not deeply analyzed in the literature. In this work, we conducted extensive experiments for iterative operations to compare the performance in both time and memory cost between Hadoop and Spark. We found that although Spark is in general faster than Hadoop in iterative operations, it has to pay for more memory consumption. Also, its speed advantage is weakened at the moment when the memory is not sufficient enough to store newly created intermediate results. |
Year | DOI | Venue |
|---|---|---|
2013 | 10.1109/HPCC.and.EUC.2013.106 | 2013 IEEE 15TH INTERNATIONAL CONFERENCE ON HIGH PERFORMANCE COMPUTING AND COMMUNICATIONS & 2013 IEEE INTERNATIONAL CONFERENCE ON EMBEDDED AND UBIQUITOUS COMPUTING (HPCC_EUC) |
Keywords | Field | DocType |
central processing unit,distributed databases,iterative methods,generators | Central processing unit,Spark (mathematics),Iterative and incremental development,General purpose,Cache,Computer science,Iterative method,Parallel computing,Response time,Real-time computing,Distributed database,Distributed computing | Conference |
Citations | PageRank | References |
25 | 1.15 | 11 |
Authors | ||
2 | ||