Solving Dynamic Multi-Objective Flexible Job Shop Scheduling Problems Using a Dual-Level Integrated Deep Q-Network Approach
Open Access
Abstract
Economic performance in modern manufacturing enterprises is often influenced by random dynamic events, requiring real-time scheduling to manage multiple conflicting production objectives simultaneously. However, traditional scheduling methods often fall short due to their limited responsiveness in dynamic environments. To address this challenge, this paper proposes an innovative online rescheduling framework called the Dual-Level Integrated Deep Q-Network (DLIDQN). This framework is designed to solve the dynamic multi-objective flexible job shop scheduling problem (DMOFJSP), which is affected by six types of dynamic events: new job insertion, job operation modification, job deletion, machine addition, machine tool replacement, and machine breakdown. The optimization focuses on three key objectives: minimizing makespan, maximizing average machine utilization (), and minimizing average job tardiness rate (). The DLIDQN framework leverages a hierarchical reinforcement learning approach and consists of two integrated IDQN-based agents. The high-level IDQN serves as the decision-maker during rescheduling, implementing dual-level decision-making by dynamically selecting optimization objectives based on the current system state and guiding the low-level IDQN’s actions. To meet diverse optimization requirements, two reward mechanisms are designed, focusing on job tardiness and machine utilization, respectively. The low-level IDQN acts as the executor, selecting the best scheduling rules to achieve the optimization goals determined by the high-level agent. To improve scheduling adaptability, nine composite scheduling rules are introduced, enabling the low-level IDQN to flexibly choose strategies for job sequencing and machine assignment, effectively addressing both sub-tasks to achieve optimal scheduling performance. Additionally, a local search algorithm is incorporated to further enhance efficiency by optimizing idle time between jobs. The numerical experimental results show that in 27 test scenarios, the DLIDQN framework consistently outperforms all proposed composite scheduling rules in terms of makespan, surpasses the widely used single scheduling rules in 26 instances, and always exceeds other reinforcement learning-based methods. Regarding the metric, the framework demonstrates superiority in 21 instances over all composite scheduling rules and maintains a consistent advantage over single scheduling rules and other RL-based strategies. For the metric, DLIDQN outperforms composite and single scheduling rules in 20 instances and surpasses other RL-based methods in 25 instances. Specifically, compared to the baseline methods, our model achieves maximum performance improvements of approximately 37%, 34%, and 30% for the three objectives, respectively. These results validate the robustness and adaptability of the proposed framework in dynamic manufacturing environments and highlight its significant potential to enhance scheduling efficiency and economic benefits.
Keywords
This publication has 33 references indexed in Scilit:
- Multi-objective reinforcement learning framework for dynamic flexible job shop scheduling problem with uncertain eventsApplied Soft Computing, 2022
- Reinforcement learning for an intelligent and autonomous production control of complex job-shops under time constraintsProduction Engineering, 2020
- Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learningApplied Soft Computing, 2020
- Adaptive job shop scheduling strategy based on weighted Q-learning algorithmJournal of Intelligent Manufacturing, 2018
- Hybrid genetic algorithms for minimizing makespan in dynamic job shop scheduling problemComputers & Industrial Engineering, 2016
- A novel dynamic scheduling strategy for solving flexible job-shop problemsJournal of Ambient Intelligence and Humanized Computing, 2016
- A GEP-based reactive scheduling policies constructing approach for dynamic flexible job shop scheduling problem with job release datesJournal of Intelligent Manufacturing, 2012
- Multi-agent-based proactive–reactive scheduling for a job shopThe International Journal of Advanced Manufacturing Technology, 2011
- A survey of dynamic scheduling in manufacturing systemsJournal of Scheduling, 2008
- Dynamic parallel machine scheduling with mean weighted tardiness objective by Q-LearningThe International Journal of Advanced Manufacturing Technology, 2006