Dmitry Pekurovsky
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Dmitry Pekurovsky has a total of 46 co-authors. Here are the top co-authors by number of shared publications:
G. Kilcup
4 shared publications- Philip E. Bourne
University of Virginia
3 shared publications - Hari Subramoni
The Ohio State University, Columbus, USA
2 shared publications - Karen Tomko
Ohio Supercomputer Center
2 shared publications - Adam Birnbaum
University of California, San Diego
1 shared publication - Akshay Venkatesh
The Ohio State University
1 shared publication
Latest Publications
- 20 June 2015
- Published by Springer Nature
- p. 434-453
Designing Non-blocking Personalized Collectives with Near Perfect Overlap for RDMA-Enabled Clusters
- 20 June 2015
- Published by Springer Nature
- p. 434-453
Designing Non-blocking Personalized Collectives with Near Perfect Overlap for RDMA-Enabled Clusters
- Open Access
- 1 January 2015
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Porting scientific libraries to PGAS in XSEDE resources
The next generation of supercomputers presents new and complex challenges that might require a change in the current paradigm of how parallel applications are developed. Hybrid programming is usually described as the best approach for exascale computers. PGAS programming models are considered an interesting alternative to work together with MPI in this hybrid model to achieve good performance in those machines. This is a very promising approach especially for one-sided and irregular communication patterns. However, this is still an emerging technology and there is not much previous experience on how to port existing MPI applications to the PGAS model. Due to the promising relevance of this approach for the next generation of devices, it is relevant to have early experience on porting applications as well as knowledge on the issues that might be faced in this new paradigm. In this paper we present two different scientific applications that are currently implemented in MPI and that are promising candidates for this PGAS paradigm. We describe how these applications have been ported, the challenges faced and some of the solutions that we found. We also show how PGAS models can achieve great performance when compared to MPI.
- 1 January 2015
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Open AccessPorting scientific libraries to PGAS in XSEDE resources
The next generation of supercomputers presents new and complex challenges that might require a change in the current paradigm of how parallel applications are developed. Hybrid programming is usually described as the best approach for exascale computers. PGAS programming models are considered an interesting alternative to work together with MPI in this hybrid model to achieve good performance in those machines. This is a very promising approach especially for one-sided and irregular communication patterns. However, this is still an emerging technology and there is not much previous experience on how to port existing MPI applications to the PGAS model. Due to the promising relevance of this approach for the next generation of devices, it is relevant to have early experience on porting applications as well as knowledge on the issues that might be faced in this new paradigm. In this paper we present two different scientific applications that are currently implemented in MPI and that are promising candidates for this PGAS paradigm. We describe how these applications have been ported, the challenges faced and some of the solutions that we found. We also show how PGAS models can achieve great performance when compared to MPI.
- 1 September 2014
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 01903918,p. 231-240
Designing Topology-Aware Communication Schedules for Alltoall Operations in Large InfiniBand Clusters
Network contention is a significant factor affecting the performance of communication intensive operations like All to all exchanges used for transpose operations of multi-dimensional FFTs on modern supercomputing systems. Over the last decade InfiniBand has become anincreasingly popular interconnect for deploying these systems. However, no practical schemes exist that allow the users of these systems to perform these communication operations in a network-to-pology-aware manner. In this paper we propose multiple schemes to create network topology-aware communication schedules for All to all FFT operations that reduce the volume of contention encountered by the operations. Through careful study and analysis of communication performance we derive critical factors that result in network contention in large scale InfiniBand clusters. We propose enhancements to our topology discovery service to generate the path matrix in a scalable and efficient manner. Through our techniques, we are able to significantly reduce the amount of network contention observed during the Alltoall / FFT operations. The results of our experimental evaluation indicate that our proposed technique is able to deliver up to a 12% improvement in the communication time of P3DFFT at 4,096 processes.
- 1 September 2014
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 01903918,p. 231-240
Designing Topology-Aware Communication Schedules for Alltoall Operations in Large InfiniBand Clusters
Network contention is a significant factor affecting the performance of communication intensive operations like All to all exchanges used for transpose operations of multi-dimensional FFTs on modern supercomputing systems. Over the last decade InfiniBand has become anincreasingly popular interconnect for deploying these systems. However, no practical schemes exist that allow the users of these systems to perform these communication operations in a network-to-pology-aware manner. In this paper we propose multiple schemes to create network topology-aware communication schedules for All to all FFT operations that reduce the volume of contention encountered by the operations. Through careful study and analysis of communication performance we derive critical factors that result in network contention in large scale InfiniBand clusters. We propose enhancements to our topology discovery service to generate the path matrix in a scalable and efficient manner. Through our techniques, we are able to significantly reduce the amount of network contention observed during the Alltoall / FFT operations. The results of our experimental evaluation indicate that our proposed technique is able to deliver up to a 12% improvement in the communication time of P3DFFT at 4,096 processes.
- 1 October 2013
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 01903918,p. 611-620
A Novel Functional Partitioning Approach to Design High-Performance MPI-3 Non-blocking Alltoallv Collective on Multi-core Systems
Non-blocking collectives have been recently standardized by the Message Passing Interface (MPI) Forum. However, intelligent designs offered by the MPI communication runtimes are likely to be the key factors that drive their adoption. While hardware based solutions for non-blocking collective operations have shown promise, they require specialized hardware support and currently have several performance and scalability limitations. Alternatively, researchers have proposed software-based, Functional Partitioning solutions for non-blocking collectives, that rely on spare cores in each node to progress non-blocking collectives. However, these designs also require additional memory resources, and involve expensive copy operations. Such factors limit the overall performance and scalability benefits associated with using non-blocking collectives in MPI. In this paper, we propose a high performance, shared-memory backed, user-level approach based on functional partitioning, to design MPI-3 non-blocking collectives. Our approach relies on using one ``Communication Servlet (CS)" thread per node to seamlessly execute the non-blocking collective operations on behalf of the application processes. Our design also eliminates the need for additional memory resources and expensive copy operations between the application processes and the CS. We demonstrate that our solution can deliver near-perfect computation/communication overlap with large message, dense collective operations, such as MPI_Ialltoallv, while using just one core per node. We also study the benefits of our approach with a popular parallel 3D-FFT kernel, which has been re-designed to use the MPI_Ialltoallv operation. We observe that our proposed designs can improve the performance of the P3DFFT kernel by up to 27%, with 2,048 processes on the TACC Stampede system.
- 1 October 2013
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 01903918,p. 611-620
A Novel Functional Partitioning Approach to Design High-Performance MPI-3 Non-blocking Alltoallv Collective on Multi-core Systems
Non-blocking collectives have been recently standardized by the Message Passing Interface (MPI) Forum. However, intelligent designs offered by the MPI communication runtimes are likely to be the key factors that drive their adoption. While hardware based solutions for non-blocking collective operations have shown promise, they require specialized hardware support and currently have several performance and scalability limitations. Alternatively, researchers have proposed software-based, Functional Partitioning solutions for non-blocking collectives, that rely on spare cores in each node to progress non-blocking collectives. However, these designs also require additional memory resources, and involve expensive copy operations. Such factors limit the overall performance and scalability benefits associated with using non-blocking collectives in MPI. In this paper, we propose a high performance, shared-memory backed, user-level approach based on functional partitioning, to design MPI-3 non-blocking collectives. Our approach relies on using one ``Communication Servlet (CS)" thread per node to seamlessly execute the non-blocking collective operations on behalf of the application processes. Our design also eliminates the need for additional memory resources and expensive copy operations between the application processes and the CS. We demonstrate that our solution can deliver near-perfect computation/communication overlap with large message, dense collective operations, such as MPI_Ialltoallv, while using just one core per node. We also study the benefits of our approach with a popular parallel 3D-FFT kernel, which has been re-designed to use the MPI_Ialltoallv operation. We observe that our proposed designs can improve the performance of the P3DFFT kernel by up to 27%, with 2,048 processes on the TACC Stampede system.
- Open Access
- 22 July 2013
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Scalable spectral transforms at petascale
In this paper, I describe a framework for spectral transforms called P3DFFT, and its extended features and applications. I discuss the scaling seen on petascale platforms, and directions and some results of the ongoing work on improving performance, such as overlap of communication with computation and hybrid MPI/OpenMP implementation.
- 22 July 2013
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessScalable spectral transforms at petascale
In this paper, I describe a framework for spectral transforms called P3DFFT, and its extended features and applications. I discuss the scaling seen on petascale platforms, and directions and some results of the ongoing work on improving performance, such as overlap of communication with computation and hybrid MPI/OpenMP implementation.
- Top Cited Papers
- 1 January 2012
- Published by Society for Industrial & Applied Mathematics (SIAM) in SIAM Journal on Scientific Computing
- Vol. 34 (4) , C192-C209
P3DFFT: A Framework for Parallel Computations of Fourier Transforms in Three Dimensions
Fourier and related transforms are a family of algorithms widely employed in diverse areas of computational science, notoriously difficult to scale on high-performance parallel computers with a large number of processing elements (cores). This paper introduces a popular software package called P3DFFT which implements fast Fourier transforms (FFTs) in three dimensions in a highly efficient and scalable way. It overcomes a well-known scalability bottleneck of three-dimensional (3D) FFT implementations by using two-dimensional domain decomposition. Designed for portable performance, P3DFFT achieves excellent timings for a number of systems and problem sizes. On a Cray XT5 system P3DFFT attains 45% efficiency in weak scaling from 128 to 65,536 computational cores. Library features include Fourier and Chebyshev transforms, Fortran and C interfaces, in- and out-of-place transforms, uneven data grids, and single and double precision. P3DFFT is available as open source at http://code.google.com/p/p3dfft/. This paper discusses P3DFFT implementation and performance in a way that helps guide the user in making optimal choices for parameters of their runs.
- 1 January 2012
- Published by Society for Industrial & Applied Mathematics (SIAM) in SIAM Journal on Scientific Computing
- Vol. 34 (4) , C192-C209
Top Cited PapersP3DFFT: A Framework for Parallel Computations of Fourier Transforms in Three Dimensions
Fourier and related transforms are a family of algorithms widely employed in diverse areas of computational science, notoriously difficult to scale on high-performance parallel computers with a large number of processing elements (cores). This paper introduces a popular software package called P3DFFT which implements fast Fourier transforms (FFTs) in three dimensions in a highly efficient and scalable way. It overcomes a well-known scalability bottleneck of three-dimensional (3D) FFT implementations by using two-dimensional domain decomposition. Designed for portable performance, P3DFFT achieves excellent timings for a number of systems and problem sizes. On a Cray XT5 system P3DFFT attains 45% efficiency in weak scaling from 128 to 65,536 computational cores. Library features include Fourier and Chebyshev transforms, Fortran and C interfaces, in- and out-of-place transforms, uneven data grids, and single and double precision. P3DFFT is available as open source at http://code.google.com/p/p3dfft/. This paper discusses P3DFFT implementation and performance in a way that helps guide the user in making optimal choices for parameters of their runs.
- Open Access
- 18 July 2011
- Published by Association for Computing Machinery (ACM)
- p. 1-2
Ultrascalable Fourier transfroms in three dimensions
Fourier and related types of transforms are widely used in scientific community. Three-dimensional Fast Fourier Transforms (3D FFT), for example, are used in many areas such as DNS turbulence, astrophysics, material science, chemistry, oceanography and X-ray crystallography. In many cases this is a very compute-intensive operation. Lately there has been a need for implementations of scalable 3D FFT and related algorithms on Petascale parallel machines [1-8]. Most existing implementations of 3D FFT use one-dimensional task decomposition, and therefore are subject to scaling limitation when the number of cores reaches domain size. P3DFFT library overcomes this limitation. It is an open-source, easy-to-use software package [9] providing general solution for 3D FFT based on two-dimensional decomposition. In this way it is different from majority of other libraries such as FFTW, PESSL, MKL and ACML. P3DFFT is written in Fortran90 and MPI, with C interface available. It uses FFTW as an underlying library for FFT computation in one dimension. P3DFFT has been demonstrated to scale quite well up to tens of thousands cores on several platforms, including Kraken at NICS/ORNL. Theoretically it is scalable up to N-squared cores, provided suitable hardware support, where N is the domain size. In practice all-to-all communication inherent in the algorithm is often the performance bottleneck at large core counts. This type of communication stresses bisection bandwidth of the interconnect and is a challenging operation for most High Performance Computing (HPC) systems. (In fact one of the three NSF Track 1 system application procurement requirements involves 3D FFT as a crucial software component.) As a consequence, communication time is typically a high fraction of overall time for the algorithm (80% is not uncommon). In spite of this, P3DFFT scales quite well since with the increase of core counts the volume of data to be exchanged decreases proportionately. A test benchmark P3DFFT program has shown about 50% efficiency in strong scaling from 4k to 64k cores on Cray XT5 (see Figure 1). This is consistent with the expectation of a power law scaling of an all-to-all exchange on a 3D torus (where bisection bandwidth scales as P2/3). Some performance tuning is recommended to get the maximum benefit, and it is carried out by simply varying the aspect ratio of the two-dimensional processor grid. More details will be included in presentation.
- 18 July 2011
- Published by Association for Computing Machinery (ACM)
- p. 1-2
Open AccessUltrascalable Fourier transfroms in three dimensions
Fourier and related types of transforms are widely used in scientific community. Three-dimensional Fast Fourier Transforms (3D FFT), for example, are used in many areas such as DNS turbulence, astrophysics, material science, chemistry, oceanography and X-ray crystallography. In many cases this is a very compute-intensive operation. Lately there has been a need for implementations of scalable 3D FFT and related algorithms on Petascale parallel machines [1-8]. Most existing implementations of 3D FFT use one-dimensional task decomposition, and therefore are subject to scaling limitation when the number of cores reaches domain size. P3DFFT library overcomes this limitation. It is an open-source, easy-to-use software package [9] providing general solution for 3D FFT based on two-dimensional decomposition. In this way it is different from majority of other libraries such as FFTW, PESSL, MKL and ACML. P3DFFT is written in Fortran90 and MPI, with C interface available. It uses FFTW as an underlying library for FFT computation in one dimension. P3DFFT has been demonstrated to scale quite well up to tens of thousands cores on several platforms, including Kraken at NICS/ORNL. Theoretically it is scalable up to N-squared cores, provided suitable hardware support, where N is the domain size. In practice all-to-all communication inherent in the algorithm is often the performance bottleneck at large core counts. This type of communication stresses bisection bandwidth of the interconnect and is a challenging operation for most High Performance Computing (HPC) systems. (In fact one of the three NSF Track 1 system application procurement requirements involves 3D FFT as a crucial software component.) As a consequence, communication time is typically a high fraction of overall time for the algorithm (80% is not uncommon). In spite of this, P3DFFT scales quite well since with the increase of core counts the volume of data to be exchanged decreases proportionately. A test benchmark P3DFFT program has shown about 50% efficiency in strong scaling from 4k to 64k cores on Cray XT5 (see Figure 1). This is consistent with the expectation of a power law scaling of an all-to-all exchange on a 3D torus (where bisection bandwidth scales as P2/3). Some performance tuning is recommended to get the maximum benefit, and it is carried out by simply varying the aspect ratio of the two-dimensional processor grid. More details will be included in presentation.
- 13 April 2011
- Published by Springer Nature in Computer Science - Research and Development
- Vol. 26 (3-4) , 237-246
High-performance and scalable non-blocking all-to-all with collective offload on InfiniBand clusters: a study with parallel 3D FFT
- 13 April 2011
- Published by Springer Nature in Computer Science - Research and Development
- Vol. 26 (3-4) , 237-246
High-performance and scalable non-blocking all-to-all with collective offload on InfiniBand clusters: a study with parallel 3D FFT
- 1 May 2009
- Published by Elsevier in Parallel Computing
- Vol. 35 (5) , 284-304
Parallel performance of the coarse space linear vertex solver and low energy basis preconditioner for spectral/hp elements
- 1 May 2009
- Published by Elsevier in Parallel Computing
- Vol. 35 (5) , 284-304
Parallel performance of the coarse space linear vertex solver and low energy basis preconditioner for spectral/hp elements
- 1 June 2004
- Published by Springer Nature in New Generation Computing
- Vol. 22 (2) , 127-136
The encyclopedia of life project: Grid software and deployment
- 1 June 2004
- Published by Springer Nature in New Generation Computing
- Vol. 22 (2) , 127-136
The encyclopedia of life project: Grid software and deployment