Harnessing Unconventional Malware Detection Techniques to Equip Proactive and Resilient Cyber Defense Strategies Against the Constantly Changing Landscape of Sophisticated Cyber Threats

Raad Ali Hashmi; Muhammad Majid Adeel; Awais Khan

Authors

Raad Ali Hashmi Department of Computer Science, Bahria University, Islamabad, Pakistan
Muhammad Majid Adeel Department of Computer Science, Bahria University, Islamabad, Pakistan
Awais Khan Department of Computer Science, Bahria University, Islamabad, Pakistan

Keywords:

Deep Learning, Malware Detection, API Call Patterns, Attention Mechanism, CNN-LSTM, SMOTE, Cybersecurity

Abstract

The rapidly changing intricacy of malware, specifically in the case of highly secured air-gapped networks, necessitates proactive and resilient detection mechanisms that are highly capable of detecting sophisticated, modern, and obfuscated malware. Instant work focuses on a model for malware detection that is vibrant and resilient and uses deep learning models to look at Windows API call patterns that come from executable files. The original dataset from Kaggle had a big class imbalance (malicious: 42,797; benign: 1,079), but the SMOTE approach helped balance the training data. In this regard, a comparison of seven deep learning models, including Simple ANN, MLP, DropConnect Improved ANN, Residual ANN, DenseNet ANN, RBF Network, and hybrid CNN-LSTM, has been conducted over both 50 and 150 training epochs on various metrics such as recall, accuracy, F1-score, precision, and ROC-AUC. As a result, the CNN-LSTM model, enhanced by an attention mechanism, exhibited superior efficacy in differentiating between benign and malicious samples. In this context, the accuracy improvement is minimal at +0.08%, but the most substantial increase in Class 0 recall is +4.1%, and the F1-score shows an enhancement of +2.7%. The most significant contribution of this study is the attention-augmented architecture that apparently diminishes interpretability and enhances focus on significant behavioral attributes.

References

X. M. Chaoxian Wei, Qiang Li, Dong Guo, “Toward Identifying APT Malware through API System Calls,” Secur. Commun. Networks, vol. 1, no. 1, 2021, doi: https://doi.org/10.1155/2021/8077220.

Rabia Tahir, “A Study on Malware and Malware Detection Techniques,” Int. J. Educ. Manag. Eng., vol. 8, no. 2, pp. 20–30, 2018, doi: 10.5815/ijeme.2018.02.03.

D. Gibert, C. Mateu, and J. Planes, “The rise of machine learning for detection and classification of malware: Research developments, trends and challenges,” J. Netw. Comput. Appl., vol. 153, p. 102526, 2020, doi: https://doi.org/10.1016/j.jnca.2019.102526.

T. Alsmadi and N. Alqudah, “A Survey on malware detection techniques,” 2021 Int. Conf. Inf. Technol. ICIT 2021 - Proc., pp. 371–376, Jul. 2021, doi: 10.1109/ICIT52682.2021.9491765.

S. H. Smita Ranveer, “A Survey of Malware Detection Techniques based on Machine Learning,” Int. J. Comput. Appl., vol. 120, no. 5, pp. 1–7, 2015, doi: 10.5120/21220-3960.

K. C. Jundong Li, “Feature Selection: A Data Perspective,” ACM Comput. Surv., vol. 50, no. 6, pp. 1–45, 2017, doi: https://doi.org/10.1145/3136625.

Ö. A. A. and R. Samet, “A Comprehensive Review on Malware Detection Approaches,” IEEE Access, vol. 8, pp. 6249–6271, 2020, doi: 10.1109/ACCESS.2019.2963724.

S. Dong, L. Shu, and S. Nie, “Android Malware Detection Method Based on CNN and DNN Bybrid Mechanism,” IEEE Trans. Ind. Informatics, vol. 20, no. 5, pp. 7744–7753, May 2024, doi: 10.1109/TII.2024.3363016.

Y. Z. Ye Yao, “Research on Malware Detection Technology for Mobile Terminals Based on API Call Sequence,” Mathematics, vol. 12, no. 1, p. 20, 2024, doi: https://doi.org/10.3390/math12010020.

J. C. Zhiguo Chen, “VMCTE: Visualization-Based Malware Classification Using Transfer and Ensemble Learning,” Comput. Mater. Contin., vol. 75, no. 2, pp. 4445–4465, 2023, doi: https://doi.org/10.32604/cmc.2023.038639.

S. V. Pooja Yadav, Neeraj Menon , Vinayakumar Ravi, “EfficientNet convolutional neural networks-based Android malware detection,” Comput. Secur., vol. 115, p. 102622, 2022, doi: https://doi.org/10.1016/j.cose.2022.102622.

J. A. Muhammad Ajmal Azad, Farhan Riaz, Anum Aftab, Syed Khurram Jah Rizvi, “DEEPSEL: A novel feature selection for early identification of malware in mobile applications,” Futur. Gener. Comput. Syst., vol. 129, pp. 54–63, 2022, doi: https://doi.org/10.1016/j.future.2021.10.029.

M. Kinkead, S. Millar, N. McLaughlin, and P. O’Kane, “Towards Explainable CNNs for Android Malware Detection,” Procedia Comput. Sci., vol. 184, pp. 959–965, 2021, doi: https://doi.org/10.1016/j.procs.2021.03.118.

D. K. and D. H. S. Euh, H. Lee, “Comparative Analysis of Low-Dimensional Features and Tree-Based Ensembles for Malware Detection Systems,” IEEE Access, vol. 8, pp. 76796–76808, 2020, doi: 10.1109/ACCESS.2020.2986014.

A. Z. Tristan Bilot, Nour El Madhoun, Khaldoun Al Agha, “A Survey on Malware Detection with Graph Representation Learning,” ACM Comput. Surv., vol. 56, no. 11, pp. 1–36, 2024, doi: https://doi.org/10.1145/3664649.

M. A. Matthew G. Gaber, “Malware Detection with Artificial Intelligence: A Systematic Literature Review,” ACM Comput. Surv., vol. 56, no. 6, pp. 1–33, 2024, doi: https://doi.org/10.1145/3638552.

A. Bensaoud, J. Kalita, and M. Bensaoud, “A survey of malware detection using deep learning,” Mach. Learn. with Appl., vol. 16, p. 100546, 2024, doi: https://doi.org/10.1016/j.mlwa.2024.100546.

F. U. Siraj Uddin Qureshi, Jingsha He, Saima Tunio, Nafei Zhu, Ahsan Nazir, Ahsan Wajahat, “Systematic review of deep learning solutions for malware detection and forensic analysis in IoT,” J. King Saud Univ. - Comput. Inf. Sci., vol. 36, no. 8, p. 102164, 2024, doi: https://doi.org/10.1016/j.jksuci.2024.102164.

A. N. M. Pascal Maniriho, “A Survey of Recent Advances in Deep Learning Models for Detecting Malware in Desktop and Mobile Platforms,” ACM Comput. Surv., vol. 56, no. 6, pp. 1–41, 2024, doi: https://doi.org/10.1145/3638240.

W. and F. M. R. A. Yunmar, S. S. Kusumawardani, “Hybrid Android Malware Detection: A Review of Heuristic-Based Approach,” IEEE Access, vol. 12, pp. 41255–41286, 2024, doi: 10.1109/ACCESS.2024.3377658.

C. Y. Y. and S. N. S. -K. Kim, X. Feng, H. A. Hamadi, E. Damiani, “Advanced Machine Learning Based Malware Detection Systems,” IEEE Access, vol. 12, pp. 115296–115305, 2024, doi: 10.1109/ACCESS.2024.3434629.

Swapnaja Hiray, “Comparative Analysis of Feature Extraction Methods of Malware Detection,” Int. J. Comput. Appl., vol. 120, no. 5, pp. 1–7, 2015, doi: 10.5120/21220-3960.