Combinatorial Q-Learning for Condition-Based Infrastructure Maintenance
Open Access
- 15 February 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 46788-46799
Abstract
Infrastructure maintenance planning is a large-scale optimization problem of planning when and on which components to carry out maintenance so as to keep the whole infrastructure in good condition with minimal maintenance cost. Recent advances in condition monitoring techniques have enabled timely maintenance in response to the condition of each part regardless of age. In addition to the condition, the spatial structure is also important for cost-efficiency in infrastructure maintenance since traveling costs and/or setup costs can be saved by simultaneous maintenance of neighboring components, which is called economic dependency. This optimization problem naively has a high computational complexity of O(2nH), where n is the number of components and H is the planning horizon, and the predictive modeling of degradation is also a big issue. To solve this problem efficiently at scale, our proposed method utilizes two kinds of dynamic programming for temporal and spatial scalability and consequently enjoys O(n) complexity at each time step. For temporal scalability, we utilize a direct modeling approach for the action value of maintenance instead of modeling degradation, namely, Q-learning. For spatial scalability, we exploit locality in economic dependency by means of a reasonable approximation of the Q-function. A typical baseline approach is to divide the whole infrastructure into fixed groups of neighboring components beforehand and determine if maintenance should be performed for all the components in each group at each time step. In contrast, our scalable method enables fully combinatorial optimization for each component at each time step. We demonstrate the advantage of our method in a simulated environment, and the resulting maintenance history intuitively illustrates the benefit of our dynamic grouping approach. We also show that our method has a kind of interpretability in the optimization at each time step.Keywords
This publication has 37 references indexed in Scilit:
- A dynamic predictive maintenance policy for complex multi-component systemsReliability Engineering & System Safety, 2013
- Condition based maintenance optimization for multi-component systems using proportional hazards modelReliability Engineering & System Safety, 2011
- Long-Range Pipeline Monitoring by Distributed Fiber Optic SensingJournal of Pressure Vessel Technology, 2009
- Dynamic scheduling of maintenance tasks in the petroleum industry: A reinforcement approachEngineering Applications of Artificial Intelligence, 2009
- Optimizing infrastructure network maintenance when benefits are interdependentOR Spectrum, 2005
- Recent applications of fiber optic sensors to health monitoring in civil engineeringEngineering Structures, 2004
- Stochastic Modeling of Pavement PerformanceInternational Journal of Pavement Engineering, 2003
- A review of multi-component maintenance models with economic dependenceMathematical Methods of Operations Research, 1997
- Reinforcement learning with replacing eligibility tracesMachine Learning, 1996
- Q-learningMachine Learning, 1992