Name
Abstract | ||
|---|---|---|
We present further work on SciSpark, a Big Data framework that extends Apache Spark's in-memory parallel computing to scale scientific computations. SciSpark's current architecture and design includes: time and space partitioning of high-resolution geo-grids from NetCDF3/4; a sciDataset class providing N-dimensional array operations in Scala/Java and CF-style variable attributes (an update of our prior sciTensor class); parallel computation of time-series statistical metrics; and an interactive front-end using science (code & visualization) Notebooks. We demonstrate how SciSpark achieves parallel ingest and time/space partitioning of Earth science satellite and model datasets. We illustrate the usability, extensibility, and early performance of SciSpark using several Earth science Use cases, here presenting benchmarks for sciDataset Readers and parallel time-series analytics. A three-hour SciSpark tutorial was taught at an ESIP Federation meeting using a dozen "live" Notebooks. |
Year | Venue | Keywords |
|---|---|---|
2016 | 2016 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA) | Apache Spark, in-memory distributed computing, large scientific datasets, SciSpark |
Field | DocType | Citations |
Space partitioning,Data mining,Spark (mathematics),Scala,Visualization,Computer science,Usability,Artificial intelligence,Analytics,Big data,Java,Machine learning | Conference | 1 |
PageRank | References | Authors |
0.40 | 11 | 9 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Brian D. Wilson | 1 | 10 | 1.62 |
| Rahul Palamuttam | 2 | 1 | 0.40 |
| Kim Whitehall | 3 | 17 | 1.91 |
| Chris A. Mattmann | 4 | 200 | 25.39 |
| Alex Goodman | 5 | 1 | 0.40 |
| Maziyar Boustani | 6 | 8 | 1.70 |
| Sujen Shah | 7 | 1 | 0.40 |
| Paul Zimdars | 8 | 9 | 2.44 |
| Paul M. Ramirez | 9 | 11 | 1.65 |