The joint policy of production, maintenance, and product quality in a multi-machine production system by reinforcement learning and agent-based modeling

Nazabadi, Mohammad Reza; Najafi, Seyed Esmaeil; Mohaghar, Ali; Movahedi Sobhani, Farzad

doi:10.22105/riej.2023.298557.1240

Document Type : Research Paper

Authors

¹ Department of Industrial Engineering, Science and Research Branch, Islamic Azad University Tehran, Iran.

² Faculty of Industrial Management, University of Tehran, Tehran, Iran.

https://doi.org/10.22105/riej.2023.298557.1240

Abstract

Adopting an integrated production, maintenance, and quality policy in production systems is of great importance due to their interconnected influence. Consequently, investigating these aspects in isolation may yield an infeasible solution. This paper aims to address the joint optimal policy of production, maintenance, and quality in a two-machine-single-product production system with an intermediate buffer and final product storage. The production machines have degradation levels from as-good-as-new to the breakdown state. The failures increase the production machine's degradation level, and maintenance activities change the status to the initial state. Also, the quality of the final product depends on the level of degradation of the machines and the correlation between the degradation level of the production machines and the product's quality in the case that high degradation of the previous production machines leads to a high probability to produce wastage by the following machines is considered. The production system studied in this research has been modeled using the agent-based simulation, and the Reinforcement Learning (RL) algorithm has obtained the optimal integrated policy. The goal is to find an integrated optimal policy that minimizes production costs, maintenance costs, inventory costs, lost orders, breakdown of production machines, and low-quality production. The meta-heuristic technique evaluates the joint policy obtained by the decision-maker agent. The results show that the acquired joint policy by the RL algorithm offers acceptable performance and can be applied to the autonomous real-time decision-making process in manufacturing systems.

Keywords

Main Subjects

Machine Learning

References

[1] CEN, E. (2001). EN 13306: maintenance terminology. European Committee For Standardization. https://dl.mpedia.ir/e-books/18-[BSI]BS-EN-13306-2010-maintenance-terminology[mpedia.ir].pdf

[2] Liu, Q., Dong, M., & Chen, F. F. (2018). Single-machine-based joint optimization of predictive maintenance planning and production scheduling. Robotics and computer-integrated manufacturing, 51, 238–247.

[3] Rivera Gómez, H., Gharbi, A., Kenné, J. P., Montaño Arango, O., & Corona Armenta, J. R. (2020). Joint optimization of production and maintenance strategies considering a dynamic sampling strategy for a deteriorating system. Computers & industrial engineering, 140, 106273. https://doi.org/10.1016/j.cie.2020.106273

[4] Sutton, R. S., Barto, A. G., & others. (1998). Introduction to reinforcement learning (Vol. 135). MIT press Cambridge.

[5] Zheng, W., Lei, Y., & Chang, Q. (2017). Comparison study of two reinforcement learning based real-time control policies for two-machine-one-buffer production system. 2017 13th ieee conference on automation science and engineering (CASE) (pp. 1163–1168). IEEE.

[6] Kuhnle, A., Jakubik, J., & Lanza, G. (2019). Reinforcement learning for opportunistic maintenance optimization. Production engineering, 13, 33–41.

[7] Xanthopoulos, A. S., Kiatipis, A., Koulouriotis, D. E., & Stieger, S. (2017). Reinforcement learning-based and parametric production-maintenance control policies for a deteriorating manufacturing system. IEEE access, 6, 576–588.

[8] Paraschos, P. D., Koulinas, G. K., & Koulouriotis, D. E. (2020). Reinforcement learning for combined production-maintenance and quality control of a manufacturing system with deterioration failures. Journal of manufacturing systems, 56, 470–483.

[9] Yang, H., Li, W., & Wang, B. (2021). Joint optimization of preventive maintenance and production scheduling for multi-state production systems based on reinforcement learning. Reliability engineering & system safety, 214, 107713. https://doi.org/10.1016/j.ress.2021.107713

[10] Huang, J., Chang, Q., & Arinez, J. (2020). Deep reinforcement learning based preventive maintenance policy for serial production lines. Expert systems with applications, 160, 113701. https://doi.org/10.1016/j.eswa.2020.113701

[11] Su, J., Huang, J., Adams, S., Chang, Q., & Beling, P. A. (2022). Deep multi-agent reinforcement learning for multi-level preventive maintenance in manufacturing systems. Expert systems with applications, 192, 116323. https://doi.org/10.1016/j.eswa.2021.116323

[12] Zhao, Y., & Smidts, C. (2022). Reinforcement learning for adaptive maintenance policy optimization under imperfect knowledge of the system degradation model and partial observability of system states. Reliability engineering & system safety, 224, 108541. https://doi.org/10.1016/j.ress.2022.108541

[13] Ye, Z., Cai, Z., Yang, H., Si, S., & Zhou, F. (2023). Joint optimization of maintenance and quality inspection for manufacturing networks based on deep reinforcement learning. Reliability engineering & system safety, 236, 109290. https://doi.org/10.1016/j.ress.2023.109290

[14] Lavoie, P., Gharbi, A., & Kenne, J.-P. (2010). A comparative study of pull control mechanisms for unreliable homogenous transfer lines. International journal of production economics, 124(1), 241–251.

[15] Bouslah, B., Gharbi, A., & Pellerin, R. (2018). Joint production, quality and maintenance control of a two-machine line subject to operation-dependent and quality-dependent failures. International journal of production economics, 195, 210–226.

[16] Rivera Gomez, H., Gharbi, A., & Kenné, J. P. (2013). Joint production and major maintenance planning policy of a manufacturing system with deteriorating quality. International journal of production economics, 146(2), 575–587.

[17] Tambe, P. P., & Kulkarni, M. S. (2022). A reliability based integrated model of maintenance planning with quality control and production decision for improving operational performance. Reliability engineering & system safety, 226, 108681. https://doi.org/10.1016/j.ress.2022.108681

[18] Borshchev, A., & Filippov, A. (2004). From system dynamics and discrete event to practical agent based modeling: reasons, techniques, tools. Proceedings of the 22nd international conference of the system dynamics society (pp. 25–29). Oxfort England.

[19] Jennings, N. R. (2000). On agent-based software engineering. Artificial intelligence, 117(2), 277–296.

[20] Bonabeau, E. (2002). Agent-based modeling: Methods and techniques for simulating human systems. Proceedings of the national academy of sciences, 99(3), 7280–7287.

[21] Macal, C. M., & North, M. J. (2010). Tutorial on agent-based modelling and simulation. Journal of simulation, 4, 151–162.

[22] Schwartz, A. (1993). A reinforcement learning method for maximizing undiscounted rewards. Proceedings of the 10th international conference on machine learning (Vol. 298, pp. 298–305). Morgan Kaufmann Publishers. DOI: 10.1016/b978-1-55860-307-3.50045-9

[23] Gosavi, A., & Gosavi, A. (2015). Control optimization with reinforcement learning. In Simulation-based optimization: parametric optimization techniques and reinforcement learning (pp. 197–268). Springer.

International Journal of Research in Industrial Engineering

The joint policy of production, maintenance, and product quality in a multi-machine production system by reinforcement learning and agent-based modeling

References

References

Volume 13, Issue 1
March 2024
Pages 71-87

The joint policy of production, maintenance, and product quality in a multi-machine production system by reinforcement learning and agent-based modeling

References

References

Volume 13, Issue 1March 2024Pages 71-87

Volume 13, Issue 1
March 2024
Pages 71-87