Dynamic Pricing of Customer Classes in Rail Transportation Systems Using Deep Q Network Algorithm

Document Type : Research Paper

Authors

1 MSc. Student, Department of Industrial Engineering, Faculty of Industrial and Systems Engineering, Tarbiat Modares University, Tehran, Iran.

2 Assistant Prof., Department of Information Technology, Faculty of Industrial Engineering and Systems, Tarbiat Modares University, Tehran, Iran.

10.22059/imj.2024.377050.1008164

Abstract

Objective
This research investigates the problem of dynamic pricing in rail transportation systems using advanced deep reinforcement learning techniques. The main goal is to optimize the revenue of railway transport companies by developing a ticket sales policy that dynamically adjusts ticket prices based on service classes. This approach allows rail transport companies to enhance revenue and profitability by accurately aligning prices with passenger demand.
 
Methods
To solve the problem of dynamic pricing, this research utilizes the Q deep network algorithm, which combines deep neural networks with Q-learning. Deep neural networks approximate Q values instead of using a costly Q table. The Q deep network algorithm is widely used due to its ability to learn optimal policies in complex environments. As reinforcement learning models are often too complex to analyze, numerical experiments and simulations are used to analyze different pricing strategies.
 
Results
The simulations demonstrate that the Q deep network algorithm successfully converges to a stable pricing policy. Various performance indicators were investigated, including such as total revenue, remaining capacity, average prices offered to customers, and the number of tickets sold in each service class. The algorithm showed improvement in the early stages and gradually achieved stability. The average total revenue converges to 225,000 after 5,000 iterations, indicating that the company earns an average of 225,000 monetary units from each train. The average residual capacity approaches zero after approximately 3,000 iterations, indicating that the reinforcement learning agent learns to sell all available tickets to maximize total revenue. The average price index offered to customers stabilizes after approximately 7,500 iterations, indicating that the algorithm has converged to an optimal pricing policy. In this state, the average prices remain within the range of 680 to 700 monetary units, with no significant fluctuations observed. In other words, the reinforcement learning model has successfully converged based on the average proposed price index. Finally, after about 5,000 iterations, the average number of tickets sold for all service classes reaches a stable level. The average number of tickets sold for economy class is around 175 to 180 tickets, for business class is around 130 to 135 tickets, for special class is around 60 to 65 tickets, and for hotel class is around 23 to 25 tickets.
 
Conclusion
The findings of this study suggest that employing the Deep Q-Network algorithm in dynamic pricing can lead to substantial optimization in revenue management for railway transportation systems. The results of this research indicate that after approximately 7,500 iterations, the Q deep network algorithm reaches an optimal and stable policy with no significant changes in performance. It can be concluded that the use of the Q deep network algorithm in dynamic pricing can significantly improve the revenue management of rail transportation systems. This algorithm can learn and adapt to changing conditions, allowing for effective pricing policies to maximize revenue and determine the optimal number of tickets sold in each service class. The obtained findings can help rail transport companies improve pricing strategies and increase economic productivity.

Keywords

Main Subjects

References

 
Abdalrahman, A. & Zhuang, W. (2020). Dynamic pricing for differentiated PEV charging services using deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 23(2), 1415-1427.‏
Adelnia Najafabadi, H., Shekarchizadeh, A., Nabiollahi, A., Khani, N. & Rastgari, H. (2022). Dynamic pricing for information goods using revenue management and recommender systems. Journal of Revenue and Pricing Management, 21(2), 153–163.
Alexander, R. B. & Ling, J. S. (2019). Multi-segment dynamic pricing for airline tickets using model-free reinforcement learning.
Aljafari, B., Jeyaraj, P. R., Kathiresan, A. C. & Thanikanti, S. B. (2023). Electric vehicle optimum charging-discharging scheduling with dynamic pricing employing multi agent deep neural network. Computers and Electrical Engineering, 105, 108555.‏
Armstrong, A. & Meissner, J. (2010). Railway revenue management: Overview and models (operations research). Department of Management Science, Lancaster University Working Papers, (MRG/0019).‏
Avila, N., Hardan, S., Zhalieva, E., Aloqaily, M. & Guizani, M. (2022). Energy Pricing in P2P Energy Systems Using Reinforcement Learning. arXiv preprint arXiv:2210.13555.‏
Bagherpour, R., Mozayani, N. & Badnava, B. (2021). Improving demand-response scheme in smart grids using reinforcement learning. International Journal of Energy Research, 45(15), 21082-21095.‏
Bertsimas, D. & Kallus, N. (2020). From predictive to prescriptive analytics. Management Science, 66(3), 1025-1044.‏
Bondoux, N., Nguyen, A. Q., Fiig, T. & Acuna-Agost, R. (2020). Reinforcement learning applied to airline revenue management. Journal of Revenue and Pricing Management, 19(5), 332-348.‏
Burger, B. & Fuchs, M. (2005). Dynamic pricing—A future airline business model. Journal of Revenue and Pricing Management, 4(1), 39-53.‏
Chen, S., Li, L., Chen, Z. & Li, S. (2020). Dynamic pricing for smart mobile edge computing: A reinforcement learning approach. IEEE Wireless Communications Letters, 10(4), 700-704.‏
Collins, A. & Thomas, L. (2012). Comparing reinforcement learning approaches for solving game theoretic models: a dynamic airline pricing game example. Journal of the Operational Research Society, 63(8), 1165–1173.
Cong, P., Zhou, J., Chen, M. & Wei, T. (2020). Personality-guided cloud pricing via reinforcement learning. IEEE Transactions on Cloud Computing, 10(2), 925-943.‏
Den Boer, A. V. (2015). Dynamic pricing and learning: historical origins, current research, and new directions. Surveys in operations research and management science, 20(1), 1-18.‏
Du, J., Cheng, W., Lu, G., Cao, H., Chu, X., Zhang, Z. & Wang, J. (2021). Resource pricing and allocation in MEC enabled blockchain systems: An A3C deep reinforcement learning approach. IEEE Transactions on Network Science and Engineering, 9(1), 33-44.‏
Du, P. & Chen, Q. (2017). Skimming or penetration: optimal pricing of new fashion products in the presence of strategic consumers. Annals of Operations Research, 257, 275-295.‏
Fraija, A., Agbossou, K., Henao, N., Kelouwani, S., Fournier, M. & Hosseini, S. S. (2022). A discount-based time-of-use electricity pricing strategy for demand response with minimum information using reinforcement learning. IEEE Access, 10, 54018-54028.‏
Gao, J., Le, M. & Fang, Y. (2022). Dynamic air ticket pricing using reinforcement learning method. RAIRO-Operations Research, 56(4), 2475-2493.‏
Gosavi, A., Bandla, N. & Das, T. (2002). A reinforcement learning approach to a single leg airline revenue management problem with multiple fare classes and overbooking. IIE Transactions, 34, 729–742.
Jing, Y., Guo, S., Chen, F., Wang, X. & Li, K. (2021). Dynamic differential pricing of high-speed railway based on improved GBDT train classification and bootstrap time node determination. IEEE Transactions on Intelligent Transportation Systems, 23(9), 16854-16866.‏
Jung, H. (2022). An optimal charging and discharging scheduling algorithm of energy storage system to save electricity pricing using reinforcement learning in urban railway system. Journal of Electrical Engineering & Technology, 17(1), 727-735.‏
Kamandanipour, K., Haji Yakhchali, S. & Tavakkoli-Moghaddam, R. (2023). Dynamic revenue management in a passenger rail network under price and fleet management decisions. Annals of Operations Research, 1-25.‏
Kamandanipour, K., Yakhchali, S. H. & Tavakkoli-Moghaddam, R. (2023). Learning-based dynamic ticket pricing for passenger railway service providers. Engineering optimization, 55(4), 703-717.‏
Kim, B. G., Zhang, Y., Van Der Schaar, M. & Lee, J. W. (2015). Dynamic pricing and energy consumption scheduling with reinforcement learning. IEEE Transactions on smart grid, 7(5), 2187-2198.‏
Koc, I. & Arslan, E. (2021). Dynamic ticket pricing of airlines using variant batch size interpretable multi-variable long short-term memory. Expert Systems with Applications, 175, 114794.‏
Krasheninnikova, E., García, J., Maestre, R. & Fernández, F. (2019). Reinforcement learning for pricing strategy optimization in the insurance industry. Engineering applications of artificial intelligence, 80, 8-19.‏
Lei, Z. & Ukkusuri, S. V. (2023). Scalable reinforcement learning approaches for dynamic pricing in ride-hailing systems. Transportation Research Part B: Methodological, 178, 102848.‏
Liao, Y., Qiao, X., Yu, Q. & Liu, Q. (2021). Intelligent dynamic service pricing strategy for multi-user vehicle-aided MEC networks. Future Generation Computer Systems, 114, 15-22.‏
Liu, H., Chen, C., Li, Y., Duan, Z. & Li, Y. (2022). Chapter 1 - Introduction. In H. Liu, C. Chen, Y. Li, Z. Duan & Y. Li (Eds.), Smart Metro Station Systems (pp. 1–32). Elsevier.
Liu, Y., Zhang, D. & Gooi, H. B. (2020). Data-driven decision-making strategies for electricity retailers: A deep reinforcement learning approach. CSEE Journal of Power and Energy Systems, 7(2), 358-367.‏
Lu, R., Hong, S. H. & Zhang, X. (2018). A dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach. Applied energy, 220, 220-230.‏
Lu, T., Chen, X., McElroy, M. B., Nielsen, C. P., Wu, Q. & Ai, Q. (2020). A reinforcement learning-based decision system for electricity pricing plan selection by smart grid end users. IEEE Transactions on Smart Grid, 12(3), 2176-2187.‏
Mehrjoo, S., Amoozad Mahdirji, H., Heidary Dahoei, J., Razavi Haji Agha, S. H. & Hosseinzadeh, M. (2023). Providing a Robust Dynamic Pricing Model and Comparing It with Static Pricing in Multi-level Supply Chains Using a Game Theory Approach. Industrial Management Journal, 15(4), 534-565.‏ (in Persian)
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D. & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.‏
Moghaddam, V., Yazdani, A., Wang, H., Parlevliet, D. & Shahnia, F. (2020). An online reinforcement learning approach for dynamic pricing of electric vehicle charging stations. IEEE Access, 8, 130305-130313.‏
Narahari, Y., Raju, C. V. L., Ravikumar, K. & Shah, S. (2005). Dynamic pricing models for electronic business. Sadhana, 30(2), 231–256.
Nian, R., Liu, J. & Huang, B. (2020). A review on reinforcement learning: Introduction and applications in industrial process control. Computers & Chemical Engineering, 139, 106886.‏
Pandey, V. & Boyles, S. D. (2018). Dynamic pricing for managed lanes with multiple entrances and exits. Transportation Research Part C: Emerging Technologies, 96, 304-320.‏
Pandey, V., Wang, E. & Boyles, S. D. (2020). Deep reinforcement learning algorithm for dynamic pricing of express lanes with multiple access locations. Transportation Research Part C: Emerging Technologies, 119, 102715.‏
Poh, L. Z., Connie, T., Ong, T. S. & Goh, M. K. O. (2023). Deep reinforcement learning-based dynamic pricing for parking solutions. Algorithms, 16(1), 32.‏
Qiu, D., Ye, Y., Papadaskalopoulos, D. & Strbac, G. (2020). A deep reinforcement learning method for pricing electric vehicles with discrete charging levels. IEEE Transactions on Industry Applications, 56(5), 5901-5912.‏
Raju, C. V. L., Narahari, Y. & Ravikumar, K. (2006). Learning dynamic prices in electronic retail markets with customer segmentation. Annals of Operations Research, 143(1), 59–75.
Rana, R. & Oliveira, F. S. (2014). Real-time dynamic pricing in a non-stationary environment using model-free reinforcement learning. Omega, 47, 116–126.
Russell, S. & Norvig, P. (2010). Artificial Intelligence: A Modern Approach (3th ed.). Prentice-Hall, Upper Saddle River.
Saharan, S., Bawa, S. & Kumar, N. (2020). Dynamic pricing techniques for Intelligent Transportation System in smart cities: A systematic review. Computer Communications, 150, 603-625.‏
Sato, K., Seo, T. & Fuse, T. (2021). A reinforcement learning-based dynamic congestion pricing method for the morning commute problems. Transportation Research Procedia, 52, 347-355.‏
Shan, X., Lv, X., Wu, J., Zhao, S. & Zhang, J. (2024). Revenue management method and critical techniques of railway passenger transport. Railway Sciences, 3(5), 636-649.
Stavinova, E., Chunaev, P. & Bochenina, K. (2021). Forecasting railway ticket dynamic price with Google Trends open data. Procedia Computer Science, 193, 333–342.
Strauss, A. K., Klein, R. & Steinhardt, C. (2018). A review of choice-based revenue management: Theory and methods. European Journal of Operational Research, 271(2), 375–387.
Sutton, R. S. & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.‏
Tan, M. (2018). Optimal Pricing for Tickets with Myopic and Strategic Passengers. Ind. Eng. Manag, 23, 107-115.‏
Wan, Y., Qin, J., Yu, X., Yang, T. & Kang, Y. (2021). Price-based residential demand response management in smart grids: A reinforcement learning-based approach. IEEE/CAA Journal of Automatica Sinica, 9(1), 123-134.‏
Wang, S., Bi, S. & Zhang, Y. A. (2019). Reinforcement learning for real-time pricing and scheduling control in EV charging stations. IEEE Transactions on Industrial Informatics, 17(2), 849-859.‏
Wittman, M. D. & Belobaba, P. P. (2019). Dynamic pricing mechanisms for the airline industry: a definitional framework. Journal of Revenue and Pricing Management, 18(2), 100–106.
Wu, X., Qin, J., Qu, W., Zeng, Y. & Yang, X. (2019). Collaborative optimization of dynamic pricing and seat allocation for high-speed railways: An empirical study from China. IEEE Access, 7, 139409-139419.‏
Xiaoqiang, Z., Lang, M. & Jin, Z. (2017). Dynamic pricing for passenger groups of high-speed rail transportation. Journal of Rail Transport Planning & Management, 6(4), 346-356.‏
Xu, H., Wen, J., Hu, Q., Shu, J., Lu, J. & Yang, Z. (2022). Energy Procurement and Retail Pricing for Electricity Retailers via Deep Reinforcement Learning with Long Short-term Memory. CSEE Journal of Power and Energy Systems, 8(5), 1338-1351.‏
Xu, Z., Guo, Y., Sun, H., Tang, W. & Huang, W. (2023). Deep reinforcement learning for competitive DER pricing problem of virtual power plants. CSEE Journal of Power and Energy Systems.‏
Yan, Z., Zhang, P., Zhang, Y., Liu, H., Feng, C. & Li, X. (2019). Joint decision model of group ticket booking limits and individual passenger dynamic pricing for the high-speed railway. Symmetry, 11(9), 1128.‏
Yang, Q. Q., Xu, L. P. & Yang, Y. (2012). Dynamic Pricing for Multiple-Class High-Speed Railway on the Internet. Applied Mechanics and Materials, 253–255, 1263–1267.
Yousefi, A. & Pishvaee, M. S. (2022). A hybrid machine learning-optimization approach to pricing and train formation problem under demand uncertainty. RAIRO-Operations Research, 56(3), 1429-1451.‏
Zeng, H. & Zhang, Y. (2015). Intertemporal Pricing of Substitutes under the Coexistence of Myopic and Strategic Consumers. Syst. Eng, 65, 33-39.‏
Zhang, P., Wang, C., Aujla, G. S. & Batth, R. S. (2021). ReLeDP: Reinforcement-learning-assisted dynamic pricing for wireless smart grid. IEEE Wireless Communications, 28(6), 62-69.‏
Zheng, J., Gan, Y., Liang, Y., Jiang, Q. & Chang, J. (2021). Joint Strategy of Dynamic Ordering and Pricing for Competing Perishables with Q-Learning Algorithm. Wireless Communications and Mobile Computing, (1), 6643195.‏
Zhu, Y. T., Wang, F. Z., Lv, X. Y. & Pan, Y. (2014, August). Dynamic pricing for railway tickets with demand-shifted passenger groups. In 2014 International Conference on Management Science & Engineering 21th Annual Conference Proceedings (pp. 256-262). IEEE.‏
Industrial Management Journal
Volume 16, Issue 4
2024
Pages 597-630

Files

Share

How to cite

Statistics