Paper Info
Title
Parallel index and query for large scale data analysis
Abstract
Modern scientific datasets present numerous data management and analysis challenges. State-of-the-art index and query technologies are critical for facilitating interactive exploration of large datasets, but numerous challenges remain in terms of designing a system for processing general scientific datasets. The system needs to be able to run on distributed multi-core platforms, efficiently utilize underlying I/O infrastructure, and scale to massive datasets. We present FastQuery, a novel software framework that address these challenges. FastQuery utilizes a state-of-the-art index and query technology (FastBit) and is designed to process massive datasets on modern supercomputing platforms. We apply FastQuery to processing of a massive 50TB dataset generated by a large scale accelerator modeling code. We demonstrate the scalability of the tool to 11,520 cores. Motivated by the scientific need to search for interesting particles in this dataset, we use our framework to reduce search time from hours to tens of seconds.
Year
DOI
Venue
2011
10.1145/2063384.2063424
SC
Keywords
Field
DocType
large scale data analysis,query technology,massive datasets,modern scientific datasets,state-of-the-art index,parallel index,modern supercomputing platform,scientific need,large scale accelerator modeling,novel software framework,large datasets,general scientific datasets,indexing,indexation,software framework,parallel processing,data management,data analysis
Supercomputer,Computer science,Parallel computing,Parallel processing,Search engine indexing,Data management,Software framework,Distributed computing,Scalability
Conference
Citations 
PageRank 
References 
38
1.68
14
Authors
10
Name
Order
Citations
PageRank
Jerry Chou1553.69
Mark Howison21069.21
Brian Austin3484.12
Kesheng Wu41231108.30
Ji Qiang57910.07
E. Wes Bethel643839.76
Arie Shoshani71701675.01
Oliver Rübel810311.78
Prabhat945634.79
Rob D. Ryne10381.68