Prediction-Aware Dynamic Scheduling Strategy for Long-Tail Remote Sensing Tasks in Heterogeneous Clusters
Open Access
- 20 March 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 677-682
Abstract
To address the challenges of computational power fragmentation and long-tail blocking encountered during the operational processing of remote sensing big data in heterogeneous clusters, this paper proposes a multi-dimensional dynamic scheduling strategy based on execution time prediction awareness. A Directed Acyclic Graph (DAG) is introduced to achieve fine-grained decoupling of complex remote sensing production workflows. Combined with advanced network storage protocols, a cross-layer zero-copy data architecture is constructed to fundamentally eliminate I/O congestion in concurrent environments. In response to the extreme right-skewed long-tail distribution of remote sensing atomic task execution times, a two-stage ensemble learning model, XLHP (XGBoost & LightGBM Hybrid Predictor), is proposed. This model utilizes XGBoost for high-dimensional monitoring feature refinement and integrates the Log1p transformation with LightGBM's Leaf-wise growth mechanism, achieving accurate microsecond-level prediction for both high-residual, long-duration tasks and massive short-duration tasks. Ultimately, the temporal prior knowledge is incorporated into a multi-objective combinatorial optimization model for heterogeneous clusters, formulating the PABF (Prediction-Aware Best Fit) scheduling algorithm, which features multi-stage funnel capacity pre-checking and dual-mode adaptive weighting. Experimental results demonstrate that, compared with traditional static heuristic scheduling, the proposed strategy leverages a dynamic backfilling mechanism for short tasks to reduce the maximum completion time (Makespan) by 42.2%, while maintaining the core resource utilization of the cluster at a stable, high-level equilibrium range of 80% to 95%.Keywords
Funding Information
- This study was funded by the National Key Research and Develop ment Program of China (2024YFB3908404)
This publication has 5 references indexed in Scilit:
- Intelligent Prediction Based on NRBO–LightGBM Model of Reservoir Slope Deformation and Interpretability AnalysisWater, 2025
- Latency-Sensitive Function Placement among Heterogeneous Nodes in Serverless ComputingSensors, 2024
- ScienceEarth: A Big Data Platform for Remote Sensing Data ProcessingRemote Sensing, 2020
- Introduction to MPI: The Message Passing Interface2016
- Utilization Prediction Aware VM Consolidation Approach for Green Cloud Computing2015