DPDmitry Pekurovsky

Dmitry Pekurovsky

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Dmitry Pekurovsky has a total of 46 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Designing Non-blocking Personalized Collectives with Near Perfect Overlap for RDMA-Enabled Clusters

    by Hari Subramoni, Ammar Ahmad Awan, Khaled Hamidouche, Dmitry Pekurovsky, Akshay Venkatesh, Sourav Chakraborty, Karen Tomko, Dhabaleswar K. Panda


  • Open Access

    Porting scientific libraries to PGAS in XSEDE resources

    by Antonio Gómez-Iglesias, Dmitry Pekurovsky, Khaled Hamidouche, Jie Zhang, Jérôme Vienne

    The next generation of supercomputers presents new and complex challenges that might require a change in the current paradigm of how parallel applications are developed. Hybrid programming is usually described as the best approach for exascale computers. PGAS programming models are considered an interesting alternative to work together with MPI in this hybrid model to achieve good performance in those machines. This is a very promising approach especially for one-sided and irregular communication patterns. However, this is still an emerging technology and there is not much previous experience on how to port existing MPI applications to the PGAS model. Due to the promising relevance of this approach for the next generation of devices, it is relevant to have early experience on porting applications as well as knowledge on the issues that might be faced in this new paradigm. In this paper we present two different scientific applications that are currently implemented in MPI and that are promising candidates for this PGAS paradigm. We describe how these applications have been ported, the challenges faced and some of the solutions that we found. We also show how PGAS models can achieve great performance when compared to MPI.


  • Designing Topology-Aware Communication Schedules for Alltoall Operations in Large InfiniBand Clusters

    by Hari Subramoni, Krishna. Kandalla, Jithin Jose, Karen Tomko, Karl Schulz, Dmitry Pekurovsky, Dhabaleswar K. Panda

    Network contention is a significant factor affecting the performance of communication intensive operations like All to all exchanges used for transpose operations of multi-dimensional FFTs on modern supercomputing systems. Over the last decade InfiniBand has become anincreasingly popular interconnect for deploying these systems. However, no practical schemes exist that allow the users of these systems to perform these communication operations in a network-to-pology-aware manner. In this paper we propose multiple schemes to create network topology-aware communication schedules for All to all FFT operations that reduce the volume of contention encountered by the operations. Through careful study and analysis of communication performance we derive critical factors that result in network contention in large scale InfiniBand clusters. We propose enhancements to our topology discovery service to generate the path matrix in a scalable and efficient manner. Through our techniques, we are able to significantly reduce the amount of network contention observed during the Alltoall / FFT operations. The results of our experimental evaluation indicate that our proposed technique is able to deliver up to a 12% improvement in the communication time of P3DFFT at 4,096 processes.


  • A Novel Functional Partitioning Approach to Design High-Performance MPI-3 Non-blocking Alltoallv Collective on Multi-core Systems

    by K. Kandalla, H. Subramoni, K. Tomko, D. Pekurovsky, D.K. Panda

    Non-blocking collectives have been recently standardized by the Message Passing Interface (MPI) Forum. However, intelligent designs offered by the MPI communication runtimes are likely to be the key factors that drive their adoption. While hardware based solutions for non-blocking collective operations have shown promise, they require specialized hardware support and currently have several performance and scalability limitations. Alternatively, researchers have proposed software-based, Functional Partitioning solutions for non-blocking collectives, that rely on spare cores in each node to progress non-blocking collectives. However, these designs also require additional memory resources, and involve expensive copy operations. Such factors limit the overall performance and scalability benefits associated with using non-blocking collectives in MPI. In this paper, we propose a high performance, shared-memory backed, user-level approach based on functional partitioning, to design MPI-3 non-blocking collectives. Our approach relies on using one ``Communication Servlet (CS)" thread per node to seamlessly execute the non-blocking collective operations on behalf of the application processes. Our design also eliminates the need for additional memory resources and expensive copy operations between the application processes and the CS. We demonstrate that our solution can deliver near-perfect computation/communication overlap with large message, dense collective operations, such as MPI_Ialltoallv, while using just one core per node. We also study the benefits of our approach with a popular parallel 3D-FFT kernel, which has been re-designed to use the MPI_Ialltoallv operation. We observe that our proposed designs can improve the performance of the P3DFFT kernel by up to 27%, with 2,048 processes on the TACC Stampede system.


  • Open Access

    Scalable spectral transforms at petascale

    by Dmitry Pekurovsky

    In this paper, I describe a framework for spectral transforms called P3DFFT, and its extended features and applications. I discuss the scaling seen on petascale platforms, and directions and some results of the ongoing work on improving performance, such as overlap of communication with computation and hybrid MPI/OpenMP implementation.

  • Top Cited Papers

    P3DFFT: A Framework for Parallel Computations of Fourier Transforms in Three Dimensions

    by Dmitry Pekurovsky

    Fourier and related transforms are a family of algorithms widely employed in diverse areas of computational science, notoriously difficult to scale on high-performance parallel computers with a large number of processing elements (cores). This paper introduces a popular software package called P3DFFT which implements fast Fourier transforms (FFTs) in three dimensions in a highly efficient and scalable way. It overcomes a well-known scalability bottleneck of three-dimensional (3D) FFT implementations by using two-dimensional domain decomposition. Designed for portable performance, P3DFFT achieves excellent timings for a number of systems and problem sizes. On a Cray XT5 system P3DFFT attains 45% efficiency in weak scaling from 128 to 65,536 computational cores. Library features include Fourier and Chebyshev transforms, Fortran and C interfaces, in- and out-of-place transforms, uneven data grids, and single and double precision. P3DFFT is available as open source at http://code.google.com/p/p3dfft/. This paper discusses P3DFFT implementation and performance in a way that helps guide the user in making optimal choices for parameters of their runs.


  • Open Access

    Ultrascalable Fourier transfroms in three dimensions

    by Dmitry Pekurovsky

    Fourier and related types of transforms are widely used in scientific community. Three-dimensional Fast Fourier Transforms (3D FFT), for example, are used in many areas such as DNS turbulence, astrophysics, material science, chemistry, oceanography and X-ray crystallography. In many cases this is a very compute-intensive operation. Lately there has been a need for implementations of scalable 3D FFT and related algorithms on Petascale parallel machines [1-8]. Most existing implementations of 3D FFT use one-dimensional task decomposition, and therefore are subject to scaling limitation when the number of cores reaches domain size. P3DFFT library overcomes this limitation. It is an open-source, easy-to-use software package [9] providing general solution for 3D FFT based on two-dimensional decomposition. In this way it is different from majority of other libraries such as FFTW, PESSL, MKL and ACML. P3DFFT is written in Fortran90 and MPI, with C interface available. It uses FFTW as an underlying library for FFT computation in one dimension. P3DFFT has been demonstrated to scale quite well up to tens of thousands cores on several platforms, including Kraken at NICS/ORNL. Theoretically it is scalable up to N-squared cores, provided suitable hardware support, where N is the domain size. In practice all-to-all communication inherent in the algorithm is often the performance bottleneck at large core counts. This type of communication stresses bisection bandwidth of the interconnect and is a challenging operation for most High Performance Computing (HPC) systems. (In fact one of the three NSF Track 1 system application procurement requirements involves 3D FFT as a crucial software component.) As a consequence, communication time is typically a high fraction of overall time for the algorithm (80% is not uncommon). In spite of this, P3DFFT scales quite well since with the increase of core counts the volume of data to be exchanged decreases proportionately. A test benchmark P3DFFT program has shown about 50% efficiency in strong scaling from 4k to 64k cores on Cray XT5 (see Figure 1). This is consistent with the expectation of a power law scaling of an all-to-all exchange on a 3D torus (where bisection bandwidth scales as P2/3). Some performance tuning is recommended to get the maximum benefit, and it is carried out by simply varying the aspect ratio of the two-dimensional processor grid. More details will be included in presentation.


  • High-performance and scalable non-blocking all-to-all with collective offload on InfiniBand clusters: a study with parallel 3D FFT

    by Krishna Kandalla, Hari Subramoni, Karen Tomko, Dmitry Pekurovsky, Sayantan Sur, Dhabaleswar K. Panda


  • The encyclopedia of life project: Grid software and deployment

    by Wilfred W. Li, Robert W. Byrnes, Jim Hayes, Adam Birnbaum, Vicente M. Reyes, Atif Shahab, Coleman Mosley, Dmitry Pekurovsky, Greg B. Quinn, Ilya N. Shindyalov, et al.