Abstract
To address the challenges of computational power fragmentation and long-tail blocking encountered during the operational processing of remote sensing big data in heterogeneous clusters, this paper proposes a multi-dimensional dynamic scheduling strategy based on execution time prediction awareness. A Directed Acyclic Graph (DAG) is introduced to achieve fine-grained decoupling of complex remote sensing production workflows. Combined with advanced network storage protocols, a cross-layer zero-copy data architecture is constructed to fundamentally eliminate I/O congestion in concurrent environments. In response to the extreme right-skewed long-tail distribution of remote sensing atomic task execution times, a two-stage ensemble learning model, XLHP (XGBoost & LightGBM Hybrid Predictor), is proposed. This model utilizes XGBoost for high-dimensional monitoring feature refinement and integrates the Log1p transformation with LightGBM's Leaf-wise growth mechanism, achieving accurate microsecond-level prediction for both high-residual, long-duration tasks and massive short-duration tasks. Ultimately, the temporal prior knowledge is incorporated into a multi-objective combinatorial optimization model for heterogeneous clusters, formulating the PABF (Prediction-Aware Best Fit) scheduling algorithm, which features multi-stage funnel capacity pre-checking and dual-mode adaptive weighting. Experimental results demonstrate that, compared with traditional static heuristic scheduling, the proposed strategy leverages a dynamic backfilling mechanism for short tasks to reduce the maximum completion time (Makespan) by 42.2%, while maintaining the core resource utilization of the cluster at a stable, high-level equilibrium range of 80% to 95%.
Funding Information
  • This study was funded by the National Key Research and Develop ment Program of China (2024YFB3908404)