Name
Abstract | ||
|---|---|---|
Big Data (as embodied by Hadoop clusters) and Big Compute (as embodied by MPI clusters) provide unique capabilities for storing and processing large volumes of data. Hadoop clusters make distributed computing readily accessible to the Java community and MPI clusters provide high parallel efficiency for compute intensive workloads. Bringing the big data and big compute communities together is an active area of research. The LLGrid team has developed and deployed a number of technologies that aim to provide the best of both worlds. LLGrid MapReduce allows the map/reduce parallel programming model to be used quickly and efficiently in any language on any compute cluster. D4M (Dynamic Distributed Dimensional Data Model) provided a high level distributed arrays interface to the Apache Accumulo database. The accessibility of these technologies is assessed by measuring the effort to use these tools and is typically a few lines of code. The performance is assessed by measuring the insert rate into the Accumulo database. Using these tools a database insert rate of 4M inserts/second has been achieved on an 8 node cluster. |
Year | DOI | Venue |
|---|---|---|
2012 | 10.1109/HPEC.2012.6408678 | HPEC |
Keywords | DocType | ISSN |
node cluster,database management systems,java community,application program interfaces,big data,big compute,d4m,dynamic distributed dimensional data model,parallel programming,mpi clusters,llgridmapreduce,high level distributed arrays interface,hadoop clusters,scheduler,data models,data handling,distributed computing,concurrent query,apache accumulo database,message passing,map-reduce parallel programming model,parallel matlab,java,parallel ingestion,hdfs,llgrid mapreduce | Conference | 2377-6943 |
ISBN | Citations | PageRank |
978-1-4673-1577-7 | 12 | 0.77 |
References | Authors | |
0 | 14 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Chansup Byun | 1 | 180 | 19.21 |
| William Arcand | 2 | 175 | 17.77 |
| David Bestor | 3 | 181 | 19.08 |
| Bill Bergeron | 4 | 168 | 16.57 |
| Matthew Hubbell | 5 | 192 | 20.93 |
| Jeremy Kepner | 6 | 606 | 61.58 |
| Andrew McCabe | 7 | 12 | 0.77 |
| Peter Michaleas | 8 | 201 | 20.93 |
| Julie Mullen | 9 | 138 | 15.22 |
| David O'Gwynn | 10 | 35 | 3.42 |
| Andrew Prout | 11 | 182 | 18.78 |
| Albert Reuther | 12 | 335 | 37.32 |
| Antonio Rosa | 13 | 170 | 17.67 |
| Charles Yee | 14 | 147 | 15.14 |