Paper Info
Title
A hierarchical indexing strategy for optimizing Apache Spark with HDFS to efficiently query big geospatial raster data
Abstract
Earth observations and model simulations are generating big multidimensional array-based raster data. However, it is difficult to efficiently query these big raster data due to the inconsistency among the geospatial raster data model, distributed physical data storage model, and the data pipeline in distributed computing frameworks. To efficiently process big geospatial data, this paper proposes a three-layer hierarchical indexing strategy to optimize Apache Spark with Hadoop Distributed File System (HDFS) from the following aspects: (1) improve I/O efficiency by adopting the chunking data structure; (2) keep the workload balance and high data locality by building the global index (k-d tree); (3) enable Spark and HDFS to natively support geospatial raster data formats (e.g., HDF4, NetCDF4, GeoTiff) by building the local index (hash table); (4) index the in-memory data to further improve geospatial data queries; (5) develop a data repartition strategy to tune the query parallelism while keeping high data locality. The above strategies are implemented by developing the customized RDDs, and evaluated by comparing the performance with that of Spark SQL and SciSpark. The proposed indexing strategy can be applied to other distributed frameworks or cloud-based computing systems to natively support big geospatial data query with high efficiency.
Year
DOI
Venue
2020
10.1080/17538947.2018.1523957
INTERNATIONAL JOURNAL OF DIGITAL EARTH
Keywords
DocType
Volume
Big data,hierarchical indexing,multi-dimensional,Apache Spark,HDFS,distributed computing,GIS
Journal
13.0
Issue
ISSN
Citations 
3.0
1753-8947
1
PageRank 
References 
Authors
0.36
10
9
Name
Order
Citations
PageRank
Fei Hu11348.33
Fei Hu210.36
Chaowei Yang384156.47
Yongyao Jiang4484.44
Yun Li5382.85
Weiwei Song6426.05
Daniel Q. Duffy7625.54
John L. Schnase8444105.61
Tsengdar J. Lee9235.62