Name
Title | ||
|---|---|---|
Large-scale compute-intensive analysis via a combined in-situ and co-scheduling workflow approach |
Abstract | ||
|---|---|---|
Large-scale simulations can produce hundreds of terabytes to petabytes of data, complicating and limiting the efficiency of workflows. Traditionally, outputs are stored on the file system and analyzed in post-processing. With the rapidly increasing size and complexity of simulations, this approach faces an uncertain future. Trending techniques consist of performing the analysis in-situ, utilizing the same resources as the simulation, and/or off-loading subsets of the data to a compute-intensive analysis system. We introduce an analysis framework developed for HACC, a cosmological N-body code, that uses both in-situ and co-scheduling approaches for handling petabyte-scale outputs. We compare different analysis set-ups ranging from purely off-line, to purely in-situ to in-situ/co-scheduling. The analysis routines are implemented using the PISTON/VTK-m framework, allowing a single implementation of an algorithm that simultaneously targets a variety of GPU, multi-core, and many-core architectures. |
Year | DOI | Venue |
|---|---|---|
2015 | 10.1145/2807591.2807663 | International Conference for High Performance Computing, Networking, Storage, and Analysis |
Keywords | Field | DocType |
many-core architectures,multi-core architectures,GPU,PISTON/VTK-m framework,petabyte-scale output handling,cosmological N-body code,HACC,data off-loading subsets,in-situ analysis,post-processing analysis,file system storage,large-scale simulations,co-scheduling workflow,in-situ workflow,large-scale compute-intensive analysis | Data modeling,File system,Algorithm design,GPU cluster,Terabyte,Computer science,Parallel computing,N-body simulation,Workflow,Big data,Distributed computing | Conference |
ISBN | Citations | PageRank |
978-1-5090-0273-3 | 7 | 0.45 |
References | Authors | |
22 | 12 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Christopher Sewell | 1 | 164 | 14.96 |
| Katrin Heitmann | 2 | 144 | 14.49 |
| Hal Finkel | 3 | 114 | 18.43 |
| George Zagaris | 4 | 20 | 3.54 |
| Suzanne Parete-Koon | 5 | 20 | 1.73 |
| Patricia K. Fasel | 6 | 46 | 6.46 |
| Adrian Pope | 7 | 67 | 4.45 |
| Nicholas Frontiere | 8 | 61 | 4.13 |
| Li-ta Lo | 9 | 75 | 7.03 |
| O. E. Bronson Messer | 10 | 25 | 3.50 |
| Salman Habib | 11 | 98 | 15.24 |
| James Ahrens | 12 | 233 | 35.07 |