Enhancing Predictive Business Process Monitoring in Call Centers through Multimodal Data Fusion and Heterogeneous Time-Aware LSTM-Based Multi-Task Learning

Farhat Mehmood; Afshan Ishaq

Authors

Farhat Mehmood Department of computer science, University of Gujrat Hafiz Hayat campus, Gujrat Pakistan
Afshan Ishaq Department of Electronics, University of Engineering and Technology Abbottabad Campus

Keywords:

H-MMoE, Deep Learning, Optimization, Multitask Learning, Call Center

Abstract

The optimization of call center operations and the enhancement of customer service are greatly supported by predictive business process monitoring. Traditional methods often overlook valuable multimodal data, such as conversations occurring in contact centers, because they typically rely on sequence data from business IT systems. This limitation hinders a complete understanding of business processes. In this study, we introduce a unique time-aware LSTM-based framework for predictive business process monitoring, which leverages both IT system data and dialogue data from contact centers. Our approach combines multiple data sources to improve the accuracy of forecasting ongoing business activities. To address challenges related to multi-task learning and to better utilize the rich information embedded in various data types, we propose a heterogeneous multi-task learning architecture called Heterogeneous Multi-gate Mixture-of-Experts (H-MMoE). Experimental results show that our method outperforms established baseline models such as Transformer, CNN, and standard LSTM. These findings demonstrate the potential of time-aware LSTM models to improve process monitoring, optimize workflows, and drive operational success in call center environments.

References

M. Abbasi et al., “A Review of AI and Machine Learning Contribution in Predictive Business Process Management (Process Enhancement and Process Improvement Approaches),” Bus. Process Manag. J., vol. ahead-of-print, no. ahead-of-print, Jul. 2024, doi: 10.1108/BPMJ-07-2024-0555/FULL/XML.

L. X. Ruixuan Zheng, Yanping Baoa, Lihua Zhaob, “Prediction of steelmaking process variables using K-medoids and a time-aware LSTM network,” Heliyon, vol. 10, no. 12, p. e32901, 2024, [Online]. Available: https://www.cell.com/heliyon/fulltext/S2405-8440(24)08932-1?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS2405844024089321%3Fshowall%3Dtrue

S. B. Mohammad Al Olaimat, “TA-RNN: an attention-based time-aware recurrent neural network architecture for electronic health records,” Bioinformatics, vol. 40, no. 1, pp. i169–i179, 2024, doi: https://doi.org/10.1093/bioinformatics/btae264.

Z. G. Ying Liu, Cai Xu, Long Chen, Meng Yan, Wei Zhao, “TABLE: Time-aware Balanced Multi-view Learning for stock ranking,” Knowledge-Based Syst., vol. 303, p. 112424, 2024, doi: https://doi.org/10.1016/j.knosys.2024.112424.

J. Cui, J. Ji, T. Zhang, Q. Ni, L. Cao, and Z. Chen, “A novel sparse Gaussian process regression with time-aware spatiotemporal kernel for remaining useful life prediction and uncertainty quantification of bearings,” Struct. Heal. Monit., 2024, doi: 10.1177/14759217241282876;PAGE:STRING:ARTICLE/CHAPTER.

L. Deng et al., “Time-Aware Attention-Based Transformer (TAAT) for Cloud Computing System Failure Prediction,” Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., pp. 4906–4917, Aug. 2024, doi: 10.1145/3637528.3671547/SUPPL_FILE/TAAT-.

M. W. and P. X. X. Liu, Y. Liu, “LMGD: Log-Metric Combined Microservice Anomaly Detection Through Graph-Based Deep Learning,” IEEE Access, vol. 12, pp. 186510–186519, 2024, doi: 10.1109/ACCESS.2024.3481676.

E.-P. L. Gary Ang, “Temporal Implicit Multimodal Networks for Investment and Risk Management,” ACM Trans. Intell. Syst. Technol., vol. 15, no. 2, pp. 1–25, 2024, doi: https://doi.org/10.1145/3643855.

H. Z. Chang Zong, “Stock Movement Prediction with Multimodal Stable Fusion via Gated Cross-Attention Mechanism,” arXiv:2406.06594, 2024, doi: https://doi.org/10.48550/arXiv.2406.06594.

S. W. Wuzhida Bao, Yuting Cao, Yin Yang, Hangjun Che, Junjian Huang, “Data-driven stock forecasting models based on neural networks: A review,” Inf. Fusion, vol. 113, p. 102616, 2025, doi: https://doi.org/10.1016/j.inffus.2024.102616.

Y. C. Xuan, Yang, Pang, Chengxin, Yu, Haimeng, Zeng, Xinhua, “An Enhanced Bidirectional Transformer Model with Temporal-Aware Self-Attention for Short-Term Load Forecasting,” IEEE Access, vol. 12, pp. 75625–75639, 2024, doi: 10.1109/access.2024.3373801.

P. Wang, X. Zhang, Z. Cao, and Z. Chen, “MADMM: microservice system anomaly detection via multi-modal data and multi-feature extraction,” Neural Comput. Appl., vol. 36, no. 25, pp. 15739–15757, Sep. 2024, doi: 10.1007/S00521-024-09918-1/METRICS.

E. A. M. & M. S. M. Nurul Athirah Nasarudin, Fatma Al Jasmi, Richard O. Sinnott, Nazar Zaki, Hany Al Ashwal, “A review of deep learning models and online healthcare databases for electronic health records and their use for health prediction,” Artif. Intell. Rev., vol. 57, no. 249, 2024, doi: https://doi.org/10.1007/s10462-024-10876-2.

C. S.-R. Sergio Martínez-Agüero, Antonio G. Marques, Inmaculada Mora-Jiménez, Joaquín Alvárez-Rodríguez, “Multimodal Interpretable Data-Driven Models for Early Prediction of Antimicrobial Multidrug Resistance Using Multivariate Time-Series,” arXiv:2402.06295, 2024, doi: https://doi.org/10.48550/arXiv.2402.06295.

B. A. M.-H. Ulises Manuel Ramirez-Alcocer, Edgar Tello-Leal, Gerardo Romero, “A Deep Learning Approach for Predictive Healthcare Process Monitoring,” Information, vol. 14, no. 9, p. 508, 2023, doi: https://doi.org/10.3390/info14090508.

X. X. Weijian Ni, Gang Zhao, Tong Liu, Qingtian Zeng, “Predictive Business Process Monitoring Approach Based on Hierarchical Transformer,” Electronics, vol. 12, no. 6, p. 1273, 2023, doi: https://doi.org/10.3390/electronics12061273.

Y. W. and D. Y. J. Wang, J. Huang, X. Ma, Z. Li, “MTLFormer: Multi-Task Learning Guided Transformer Network for Business Process Prediction,” IEEE Access, vol. 11, pp. 76722–76738, 2023, doi: 10.1109/ACCESS.2023.3298305.

C. Liu, Y. Wang, L. Wen, J. Cheng, L. Cheng, and Q. Zeng, “Discovering Hierarchical Multi-Instance Business Processes from Event Logs,” IEEE Trans. Serv. Comput., vol. 17, no. 1, pp. 142–155, Jan. 2024, doi: 10.1109/TSC.2023.3335360.

S. Y. and M. Z. G. Duan, “A Hybrid Neural Network Model for Sentiment Analysis of Financial Texts Using Topic Extraction, Pre-Trained Model, and Enhanced Attention Mechanism Methods,” IEEE Access, vol. 12, pp. 98207–98224, 2024, doi: 10.1109/ACCESS.2024.3429150.

F. A. D. Suraj, S. Dinesh, R. Balaji, P. Deepika, “Deciphering Product Review Sentiments Using BERT and TensorFlow,” FMDB Transactions on Sustainable Computing Systems. Accessed: Apr. 29, 2025. [Online]. Available: https://www.fmdbpub.com/uploads/articles/170054179782534. FTSCS-55-2023.pdf

C. Y. Wen jun Gu, Yi hao Zhong, Shi zun Li, Chang song Wei, Li ting Dong, Zhuo yue Wang, “Predicting Stock Prices with FinBERT-LSTM: Integrating News Sentiment Analysis,” ICCBDC ’24 Proc. 2024 8th Int. Conf. Cloud Big Data Comput., pp. 67–72, 2024, doi: https://doi.org/10.1145/3694860.3694870.

S.-H. K. and H.-J. Y. A. -Q. Duong, N. -H. Ho, S. Pant, S. Kim, “Residual Relation-Aware Attention Deep Graph-Recurrent Model for Emotion Recognition in Conversation,” IEEE Access, vol. 12, pp. 2349–2360, 2024, doi: 10.1109/ACCESS.2023.3348518.

Y. Zhao, P. Barnaghi, and H. Haddadi, “Multimodal Federated Learning on IoT Data,” Proc. - 7th ACM/IEEE Conf. Internet Things Des. Implementation, IoTDI 2022, pp. 43–54, 2022, doi: 10.1109/IOTDI54339.2022.00011.

Y. Y. G. and A. A. I. U. Haq, M. Ahmed, M. Assam, “Unveiling the Future of Oral Squamous Cell Carcinoma Diagnosis: An Innovative Hybrid AI Approach for Accurate Histopathological Image Analysis,” IEEE Access, vol. 11, pp. 118281–118290, 2023, doi: 10.1109/ACCESS.2023.3326152.

Y. Wang, Y. Wang, C. Shi, L. Cheng, H. Li, and X. Li, “An Edge 3D CNN Accelerator for Low-Power Activity Recognition,” IEEE Trans. Comput. Des. Integr. Circuits Syst., vol. 40, no. 5, pp. 918–930, May 2021, doi: 10.1109/TCAD.2020.3011042.

M. P. Negin Ashrafi, Armin Abdollahi, Greg Placencia, “Effect of a Process Mining based Pre-processing Step in Prediction of the Critical Health Outcomes,” arXiv:2407.02821, 2024, doi: https://doi.org/10.48550/arXiv.2407.02821.

Jasmin Praful Bharadiya, “Machine Learning and AI in Business Intelligence: Trends and Opportunities,” Int. J. Comput., vol. 48, no. 1, pp. 123–134, 2023, [Online]. Available: https://ijcjournal.org/index.php/InternationalJournalOfComputer/article/view/2087

H. F. Hang Chen, Xianwen Fang, “Multi-task prediction method of business process based on BERT and Transfer Learning,” Knowledge-Based Syst., vol. 254, p. 109603, 2022, doi: https://doi.org/10.1016/j.knosys.2022.109603.

N. Mehdiyev, J. Evermann, and P. Fettke, “A Novel Business Process Prediction Model Using a Deep Learning Method,” Bus. Inf. Syst. Eng., vol. 62, no. 2, pp. 143–157, Apr. 2020, doi: 10.1007/S12599-018-0551-3/METRICS.