A flexible I/O arbitration framework for netCDF‐based big data processing workflows on high‐end supercomputers
- 15 May 2017
- journal article
- research article
- Published by Wiley in Concurrency and Computation: Practice and Experience
- Vol. 29 (15)
Abstract
Summary: On the verge of the convergence between high‐performance computing and Big Data processing, it has become increasingly prevalent to deploy large‐scale data analytics workloads on high‐end supercomputers. Such applications often come in the form of complex workflows with various different components, assimilating data from scientific simulations as well as from measurements streamed from sensor networks, such as radars and satellites. For example, as part of the Flagship 2020 (post‐K) supercomputer project of Japan, RIKEN is investigating the feasibility of a highly accurate weather forecasting system that would provide a real‐time outlook for severe guerrilla rainstorms. One of the main performance bottlenecks of this application is the lack of efficient communication among workflow components, which currently takes place over the parallel file system.In this paper, we present an initial study of a direct communication framework designed for complex workflows that eliminates unnecessary file I/O among components. Specifically, we propose an I/O arbitration layer that provides direct parallel data transfer (both synchronous and asynchronous) among job components that rely on the netCDF interface for performing I/O operations. Our solution requires only minimal modifications to application code. Moreover, we propose a configuration file–based approach that allows users to specify the desired data transfer pattern among workflow components, offering a general solution for different application contexts. We present a preliminary evaluation of the proposed framework on the K Computer (running on up to 4800 compute nodes) using RIKEN's experimental weather forecasting workflow as a case study.Keywords
Funding Information
- National Natural Science Foundation of China (61303038)
This publication has 27 references indexed in Scilit:
- Big Ensemble Data Assimilation in Numerical Weather PredictionComputer, 2015
- Influence of grid aspect ratio on planetary boundary layer turbulence in large-eddy simulationsGeoscientific Model Development, 2015
- One-Day-Ahead Streamflow Forecasting Using Artificial Neural Networks and a Meteorological Mesoscale ModelJournal of Hydrologic Engineering, 2015
- Impacts of cloud microphysics on trade wind cumulus: which cloud microphysics processes contribute to the diversity in a large eddy simulation?Progress in Earth and Planetary Science, 2015
- Exascale computing and big dataCommunications of the ACM, 2015
- An integrated modelling framework of catchment‐scale ecohydrological processes: 1. Model description and tests over an energy‐limited watershedEcohydrology, 2013
- The integrated WRF/urban modelling system: development, evaluation, and applications to urban environmental problemsInternational Journal of Climatology, 2011
- Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filterPhysica D: Nonlinear Phenomena, 2007
- CPL6: The New Extensible, High Performance Parallel Coupler for the Community Climate System ModelThe International Journal of High Performance Computing Applications, 2005
- The Model Coupling Toolkit: A New Fortran90 Toolkit for Building Multiphysics Parallel Coupled ModelsThe International Journal of High Performance Computing Applications, 2005