Paper Info
Title
Large-scale Predictive Analytics in Vertica: Fast Data Transfer, Distributed Model Creation, and In-database Prediction
Abstract
A typical predictive analytics workflow will pre-process data in a database, transfer the resulting data to an external statistical tool such as R, create machine learning models in R, and then apply the model on newly arriving data. Today, this workflow is slow and cumbersome. Extracting data from databases, using ODBC connectors, can take hours on multi-gigabyte datasets. Building models on single-threaded R does not scale. Finally, it is nearly impossible to use R or other common tools, to apply models on terabytes of newly arriving data. We solve all the above challenges by integrating HP Vertica with Distributed R, a distributed framework for R. This paper presents the design of a high performance data transfer mechanism, new data-structures in Distributed R to maintain data locality with database table segments, and extensions to Vertica for saving and deploying R models. Our experiments show that data transfers from Vertica are 6x faster than using ODBC connections. Even complex predictive analysis on 100s of gigabytes of database tables can complete in minutes, and is as fast as in-memory systems like Spark running directly on a distributed file system.
Year
DOI
Venue
2015
10.1145/2723372.2742789
ACM SIGMOD Conference
Keywords
Field
DocType
HP Vertica,R,Machine Learning,In-database
Distributed File System,Data mining,Locality,Spark (mathematics),Predictive analytics,Computer science,Terabyte,Open Database Connectivity,Workflow,Database,Table (database)
Conference
Citations 
PageRank 
References 
7
0.51
14
Authors
8
Name
Order
Citations
PageRank
Shreya Prasad180.86
Arash Fard270.51
Vishrut Gupta370.85
Jorge Martinez470.51
Jeff LeFevre515511.32
Vincent Xu670.51
Meichun Hsu73437778.34
Indrajit Roy8121.30