Paper Info
Title
HiFrames: High Performance Data Frames in a Scripting Language.
Abstract
Data frames in scripting languages are essential abstractions for processing structured data. However, existing data frame solutions are either not distributed (e.g., Pandas in Python) and therefore have limited scalability, or they are not tightly integrated with array computations (e.g., Spark SQL). This paper proposes a novel compiler-based approach where we integrate data frames into the High Performance Analytics Toolkit (HPAT) to build HiFrames. It provides expressive and flexible data frame APIs which are tightly integrated with array operations. HiFrames then automatically parallelizes and compiles relational operations along with other array computations in end-to-end data analytics programs, and generates efficient MPI/C++ code. We demonstrate that HiFrames is significantly faster than alternatives such as Spark SQL on clusters, without forcing the programmer to switch to embedded SQL for part of the program. HiFrames is 3.6x to 70x faster than Spark SQL for basic relational operations, and can be up to 20,000x faster for advanced analytics operations, such as weighted moving averages (WMA), that the map-reduce paradigm cannot handle effectively. HiFrames is also 5x faster than Spark SQL for TPCx-BB Q26 on 64 nodes of Cori supercomputer.
Year
Venue
Field
2017
arXiv: Distributed, Parallel, and Cluster Computing
SQL,Stored procedure,Programming language,Spark (mathematics),Computer science,Parallel computing,Embedded SQL,Analytics,Data model,Scalability,Scripting language,Distributed computing
DocType
Volume
Citations 
Journal
abs/1704.02341
0
PageRank 
References 
Authors
0.34
25
4
Name
Order
Citations
PageRank
Ehsan Totoni1747.77
Wajih Ul Hassan2547.07
Todd A Anderson315121.51
Tatiana Shpeisman443632.69