PPD: A Portable and Highly Parallel Dispatching System for Deep Learning
Open Access
- 17 December 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Design Automation of Electronic Systems
- Vol. 31 (2) , 1-24
Abstract
The acceleration of inference process for deep learning models is closely tied with the parallelization capability of computational graph operators and the parallel scheduling strategies. Most existing deep learning compilers focus on optimizing intra-operator parallelism, while neglecting inter-operator parallelism. Furthermore, most industrial inference engines, such as PyTorch and TensorFlow, utilize a dataflow-based model to describe tasks and schedule operators. They are computationally expensive and operate in a topological order and are parallelized to run within a single CUDA stream. However, they fail to fully exploit the parallelism capabilities of multiple CUDA streams. In this article, we propose PPD, a portable, highly parallel dispatching system. It boosts the inference performance by dividing the computational graph into multiple taskflow-based subgraphs. Additionally, PPD entails a dispatching algorithm on a single GPU with multiple CUDA streams to enhance the parallelism and performance of model inference. PPD offers users a lightweight model definition and an inference C++ interface, allowing for seamless integration into any context. We also verify the feasibility of PPD on AMD and other graphics cards. We validate our approach on widely adopted neural network models with varying degrees of parallelism, and compare it with industrial inference engines. Experiments demonstrate that PPD outperforms SOTA methods by up to 2.28× .Keywords
Funding Information
- Theme-Based Research Scheme (T45-701/22-R)
- AVNET-HKU Emerging Microelectronics and Ubiquitous Systems (EMUS) Lab
This publication has 24 references indexed in Scilit:
- Efficient CUDA stream management for multi-DNN real-time inference on embedded GPUsJournal of Systems Architecture, 2023
- Multi-swarm particle swarm optimization based on CUDA for sparse reconstructionSwarm and Evolutionary Computation, 2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Taskflow: A Lightweight Parallel and Heterogeneous Task Graph Computing SystemIEEE Transactions on Parallel and Distributed Systems, 2021
- Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and InsightsProceedings of the IEEE, 2021
- Deep-Learning-Empowered Breast Cancer Auxiliary Diagnosis for 5GB Remote E-HealthIEEE Wireless Communications, 2021
- Deep Learning-Based Vein Localization on Embedded SystemIEEE Access, 2021
- Deep High-Resolution Representation Learning for Visual RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Deep Residual Learning for Image Recognition2016
- Optimizing CUDA code by kernel fusion: application on BLASThe Journal of Supercomputing, 2015