Parallel Processing Strategies for Geospatial Data in a Cloud Computing Infrastructure
Open Access
- 15 January 2022
- journal article
- research article
- Published by MDPI AG in Remote Sensing
- Vol. 14 (2) , 398
Abstract
This paper is on the optimization of computing resources to process geospatial image data in a cloud computing infrastructure. Parallelization was tested by combining two different strategies: image tiling and multi-threading. The objective here was to get insight on the optimal use of available processing resources in order to minimize the processing time. Maximum speedup was obtained when combining tiling and multi-threading techniques. Both techniques are complementary, but a trade-off also exists. Speedup is improved with tiling, as parts of the image can run in parallel. But reading part of the image introduces an overhead and increases the relative part of the program that can only run in serial. This limits speedup that can be achieved via multi-threading. The optimal strategy of tiling and multi-threading that maximizes speedup depends on the scale of the application (global or local processing area), the implementation of the algorithm (processing libraries), and on the available computing resources (amount of memory and cores). A medium-sized virtual server that has been obtained from a cloud service provider has rather limited computing resources. Tiling will not only improve speedup but can be necessary to reduce the memory footprint. However, a tiling scheme with many small tiles increases overhead and can introduce extra latency due to queued tiles that are waiting to be processed. In a high-throughput computing cluster with hundreds of physical processing cores, more tiles can be processed in parallel, and the optimal strategy will be different. A quantitative assessment of the speedup was performed in this study, based on a number of experiments for different computing environments. The potential and limitations of parallel processing by tiling and multi-threading were hereby assessed. Experiments were based on an implementation that relies on an application programming interface (API) abstracting any platform-specific details, such as those related to data access.Keywords
This publication has 21 references indexed in Scilit:
- Google Earth Engine: Planetary-scale geospatial analysis for everyoneRemote Sensing of Environment, 2017
- Understanding the temporal behavior of crops using Sentinel-1 and Sentinel-2-like data for agricultural applicationsRemote Sensing of Environment, 2017
- xarray: N-D labeled Arrays and Datasets in PythonJournal of Open Research Software, 2017
- Latest evolution of EOS filesystemJournal of Physics: Conference Series, 2015
- Recent Developments in High Performance Computing for Remote Sensing: A ReviewIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2011
- Amdahl's Law in the Multicore EraComputer, 2008
- High performance JPEG 2000 and MPEG-4 VTC on SMPs using OpenMPParallel Computing, 2005
- The multidimensional database system RasDaManACM SIGMOD Record, 1998
- OpenMP: an industry standard API for shared-memory programmingIEEE Computational Science and Engineering, 1998
- Validity of the single processor approach to achieving large scale computing capabilities1967