Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows
Authors:
- Tong JinTJTong Jin
- Fan ZhangFZFan Zhang
- Qian SunQSQian Sun
- Hoang BuiHBHoang Bui
- Melissa RomanusMRMelissa Romanus
- Norbert PodhorszkiNPNorbert Podhorszki
- Scott KlaskySKScott Klasky
- Hemanth KollaHKHemanth Kolla
- Jacqueline ChenJCJacqueline Chen
- Robert HagerRHRobert Hager
- Choong-Seock ChangCCChoong-Seock Chang
- Manish ParasharMPManish Parashar
- 1 May 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15302075,p. 1033-1042
Abstract
As applications target extreme scales, data staging and in-situ/in-transit data processing have been proposed to address the data challenges and improve scientific discovery. However, further research is necessary in order to understand how growing data sizes from data intensive simulations coupled with the limited DRAM capacity in High End Computing systems will impact the effectiveness of this approach. In this paper, we explore how we can use deep memory levels for data staging, and develop a multi-tiered data staging method that spans bothDRAM and solid state disks (SSD). This approach allows us to support both code coupling and data management for data intensive simulation workflows. We also show how an adaptive application-aware data placement mechanism can dynamically manage and optimize data placement across the DRAM ands storage levels in this multi-tiered data staging method. We present an experimental evaluation of our approach using wolf resources: an Infiniband cluster (Sith) and a Cray XK7system (Titan), and using combustion (S3D) and fusion (XGC1) simulations.Keywords
This publication has 24 references indexed in Scilit:
- Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows2013
- A scalable messaging system for accelerating discovery from large scale scientific simulations2012
- Combining in-situ and in-transit processing to enable extreme-scale scientific analysis2012
- Six degrees of scientific data2011
- Provisioning a Multi-tiered Data Staging Area for Extreme-Scale Machines2011
- Dual space analysis of turbulent combustion particle data2011
- Computing Contingency Statistics in Parallel: Design Trade-Offs and Limiting Cases2010
- In Situ Visualization at Extreme Scale: Challenges and OpportunitiesIEEE Computer Graphics and Applications, 2009
- Importance-Driven Time-Varying Data VisualizationIEEE Transactions on Visualization and Computer Graphics, 2008
- A design retrospective of the DIII-D tokamakNuclear Fusion, 2002