Developing Robust Defense Mechanisms against Adversarial Evasion Attacks in Machine Learning-Based Malware Classifiers

Muhammad Zikria Tahir; Saba Firdous; Talha Ahsan; Muhammad Asad; Muhammad Ali Hasnaat; Ibn e Hassan; Abdul Haseeb Malik

Authors

Muhammad Zikria Tahir Department of computing Riphah International Univeristy Faisalabad, Pakistan
Saba Firdous Department of computing Riphah International Univeristy Faisalabad, Pakistan
Talha Ahsan Department of Computing Riphah International University Faisalabad, Pakistan
Muhammad Asad Department of computing Riphah International Univeristy Faisalabad, Pakistan
Muhammad Ali Hasnaat Department of computing Riphah International Univeristy Faisalabad, Pakistan
Ibn e Hassan Department of computing Riphah International Univeristy Faisalabad, Pakistan
Abdul Haseeb Malik Department of Computer Science, University of Peshawar 25120, Pakistan

Keywords:

Adversarial Machine Learning, Adversarial Training, Evasion Attacks, Feature Squeezing, Malware Detection, Robustness, EMBER Dataset

Abstract

The machine learning (ML) classifiers are commonly employed in antivirus and endpoint detection and response (EDR) solutions to differentiate between malicious and legitimate software. But these classifiers can be defeated by an attacker who simply modifies the features of a malware file by a small amount without affecting the functionality of the file. This paper investigates this issue in the case of static Windows Portable Executable (PE) malware. We adopt the EMBER benchmark and a LightGBM classifier, and evaluate 2 attacks, FGSM and PGD, under white-box and black-box scenarios. The clean accuracy drops to 43.1% with PGD with a budget of 20 added features and to 31.4% with a budget of 50 added features without a defense. We investigate a layered defense, using adversarial training (AT) and feature squeezing (FS). With a budget of 20, AT+FS increased the robust accuracy to 91.2%, while reducing the evasion rate from 56.9% to 8.8%. It also reduces the black-box evasion rate from 31.4% to 6.1%. The cost is a 2.2 percentage-point drop in clean accuracy (97.8% to 95.6%) and a drop in F1-score from 0.976 to 0.954. An ablation study reveals that both AT and FS are helpful and that they are most beneficial when combined. All settings are reported so that others can replicate this study.

References

H. S. Anderson and P. Roth, “EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models,” Apr. 2018, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/1804.04637

R. Harang and E. M. Rudd, “SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection,” CEUR Workshop Proc., vol. 3095, pp. 13–21, Dec. 2020, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/2012.07634

T.-T. Huynh, D.-T. Huynh, and V.-Q. Trinh, “Toward Robust Malware Detection: A Survey of Datasets, Techniques, and Practical Challenges,” Eng. Technol. Appl. Sci. Res., vol. 16, no. 3, pp. 35064–35070, Jun. 2026, doi: 10.48084/ETASR.17500.

D. Li, Q. Li, Y. Ye, and S. Xu, “Arms Race in Adversarial Malware Detection: A Survey,” ACM Comput. Surv., vol. 55, no. 1, Aug. 2021, doi: 10.1145/3484491.

X. Ling et al., “Adversarial attacks against Windows PE malware detection: A survey of the state-of-the-art,” Comput. Secur., vol. 128, May 2023, doi: 10.1016/j.cose.2023.103134.

I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” 3rd Int. Conf. Learn. Represent. ICLR 2015 - Conf. Track Proc., 2015, Accessed: Apr. 22, 2026. [Online]. Available: https://www.researchgate.net/publication/403853041_Adversarial_Attacks_on_AI-Based_Intrusion_Detection_Systems

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards Deep Learning Models Resistant to Adversarial Attacks,” 6th Int. Conf. Learn. Represent. ICLR 2018 - Conf. Track Proc., Jun. 2017, Accessed: Sep. 23, 2026. [Online]. Available: https://arxiv.org/pdf/1706.06083

K. Grosse, N. Papernot, P. Manoharan, M. Backes, and P. McDaniel, “Adversarial examples for malware detection,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 10493 LNCS, pp. 62–79, 2017, doi: 10.1007/978-3-319-66399-9_4/FIGURES/5.

H. S. Anderson, A. Kharkar, B. Filar, D. Evans, and P. Roth, “Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning,” Jan. 2018, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/1801.08917

F. Kreuk, A. Barak, S. Aviv-Reuven, M. Baruch, B. Pinkas, and J. Keshet, “Deceiving End-to-End Deep Learning Malware Detectors using Adversarial Examples,” Feb. 2018, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/1802.04528

W. Song, X. Li, S. Afroz, D. Garg, D. Kuznetsov, and H. Yin, “MAB-Malware: A Reinforcement Learning Framework for Attacking Static Malware Classifiers,” Mar. 2020, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/2003.03100

T. Simonetto, S. Dyrmishi, S. Ghamizi, M. Cordy, and Y. Le Traon, “A Unified Framework for Adversarial Attack and Defense in Constrained Feature Space,” IJCAI Int. Jt. Conf. Artif. Intell., pp. 1313–1319, Dec. 2021, doi: 10.24963/ijcai.2022/183.

M. Kozák and M. Jureček, “Combining Generators of Adversarial Malware Examples to Increase Evasion Rate,” Proc. Int. Conf. Secur. Cryptogr., vol. 1, pp. 778–786, Apr. 2023, doi: 10.5220/0012127700003555.

K. Ghanem, Z. Kherbache, and O. Ourdighi, “Enhancing Adversarial Examples for Evading Malware Detection Systems: A Memetic Algorithm Approach,” Int. J. Comput. Netw. Inf. Secur., vol. 17, no. 1, pp. 1–16, Feb. 2025, doi: 10.5815/IJCNIS.2025.01.01.

M. M. Arifin, B. Abrew, D. Hubbs, and J. H. Yeh, “Adversarial Assembly: Leveraging LLMs to Evade CNN-Based Windows Malware Detection,” Lect. Notes Comput. Sci., vol. 16465 LNCS, pp. 305–318, 2026, doi: 10.1007/978-981-95-9846-5_26/SAVE-RESEARCH.

T. Lan and F. Naït-Abdesselam, “LLM-Driven Feature-Level Adversarial Attacks on Android Malware Detectors,” Dec. 2025, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/2512.21404

L. Breiman et al., Adversarial Training for {Raw-Binary} Malware Classifiers, vol. 45, no. 1. in Request for Comments, vol. 45. IEEE, 2023. Accessed: Nov. 05, 2026. [Online]. Available: https://www.usenix.org/conference/usenixsecurity23/presentation/shen-meng

B. G. Doan et al., “Feature-Space Bayesian Adversarial Learning Improved Malware Detector Robustness,” Proc. 37th AAAI Conf. Artif. Intell. AAAI 2023, vol. 37, pp. 14783–14791, Jan. 2023, doi: 10.1609/aaai.v37i12.26727.

H. Bostani, J. Cortellazzi, D. Arp, F. Pierazzi, V. Moonsamy, and L. Cavallaro, “On the Effectiveness of Adversarial Training on Malware Classifiers,” Dec. 2024, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/2412.18218

S. Dyrmishi, S. Ghamizi, T. Simonetto, Y. Le Traon, and M. Cordy, “On the Empirical Effectiveness of Unrealistic Adversarial Hardening Against Realistic Adversarial Attacks,” Proc. - IEEE Symp. Secur. Priv., vol. 2023-May, pp. 1384–1400, 2023, doi: 10.1109/SP46215.2023.10179316.

S. Saha, W. Wang, Y. Kaya, S. Feizi, and T. Dumitras, “DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified Robustness,” 12th Int. Conf. Learn. Represent. ICLR 2024, Mar. 2023, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/2303.13372

S. Calzavara, L. Cazzaro, G. E. Pibiri, and N. Prezza, “Verifiable Learning for Robust Tree Ensembles,” CCS 2023 - Proc. 2023 ACM SIGSAC Conf. Comput. Commun. Secur., vol. 1, pp. 1850–1864, May 2023, doi: 10.1145/3576915.3623100.

N. Giri, A. K. A., S. Nicolazzo, A. Nocera, and V. P, “Towards Certified Malware Detection: Provable Guarantees Against Evasion Attacks,” Apr. 2026, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/2604.20495

D. Gibert, G. Zizzo, and Q. Le, “Towards a Practical Defense against Adversarial Attacks on Deep Learning-based Malware Detectors via Randomized Smoothing,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 14399 LNCS, pp. 683–699, Aug. 2023, doi: 10.1007/978-3-031-54129-2_40.

D. Gibert and F. Manyà, “ByteShield: Adversarially Robust End-to-End Malware Detection through Byte Masking,” Dec. 2025, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/2512.09883

“Density Boosts Everything: A One-stop Strategy for Improving Performance, Robustness, and Sustainability of Malware Detectors - NDSS Symposium.” Accessed: Nov. 05, 2026. [Online]. Available: https://www.ndss-symposium.org/ndss-paper/density-boosts-everything-a-one-stop-strategy-for-improving-performance-robustness-and-sustainability-of-malware-detectors/

A. Al Siam, M. Alazab, A. Obeidat, N. Faruqui, and I. Mahmud, “An explainable and robust ensemble-based framework for malware detection,” Peer-to-Peer Netw. Appl. 2026 193, vol. 19, no. 3, pp. 81-, Apr. 2026, doi: 10.1007/S12083-026-02231-9.

D. Arp et al., “Dos and Don’ts of Machine Learning in Computer Security,” Proc. 31st USENIX Secur. Symp. Secur. 2022, pp. 3971–3988, Oct. 2020, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/2010.09470

M. Jafari and A. Shameli-Sendi, “Evaluating the robustness of adversarial defenses in malware detection systems,” Comput. Electr. Eng., vol. 130, Feb. 2026, doi: 10.1016/j.compeleceng.2025.110845.

A. Vassilev, A. Oprea, A. Fordyce, H. Anderson, X. Davies, and M. Hamin, “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations,” Mar. 2025, doi: 10.6028/NIST.AI.100-2E2025.

R. J. Joyce et al., “EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers,” Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., vol. 2, pp. 5516–5526, Jun. 2025, doi: 10.1145/3711896.3737431.

P. Mahesh, D. Sai, L. Amasala, T. Sree Bhimavarapu, and G. C. Garikipati, “A Malware Classification Survey on Adversarial Attacks and Defences,” Dec. 2023, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/2312.09636

W. Xu, D. Evans, and Y. Qi, “Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks,” 25th Annu. Netw. Distrib. Syst. Secur. Symp. NDSS 2018, Dec. 2017, doi: 10.14722/ndss.2018.23198.

A. Abusnaina, Y. Wang, S. Arora, K. Wang, M. Christodorescu, and D. Mohaisen, “Burning the Adversarial Bridges: Robust Windows Malware Detection Against Binary-level Mutations,” Proc. ACM Conf., vol. 1, Oct. 2023, doi: 10.NNNN/NNNNNNN.NNNNNNN.

J. Saxe and K. Berlin, “Deep neural network based malware detection using two dimensional binary program features,” 2015 10th Int. Conf. Malicious Unwanted Software, MALWARE 2015, pp. 11–20, Feb. 2016, doi: 10.1109/MALWARE.2015.7413680.

E. Raff, J. Barker, J. Sylvester, R. Brandon, B. Catanzaro, and C. Nicholas, “Malware Detection by Eating a Whole EXE,” Oct. 2017, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/1710.09435

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, “LightGBM: A Highly Efficient Gradient Boosting Decision Tree,” 31st Conf. Neural Inf. Process. Syst., 2017, [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf

L. Grinsztajn, E. Oyallon, and G. Varoquaux, “Why do tree-based models still outperform deep learning on tabular data?,” Jul. 2022, Accessed: Sep. 10, 2026. [Online]. Available: https://arxiv.org/pdf/2207.08815

A. McCarthy, E. Ghadafi, P. Andriotis, and P. Legg, “Functionality-Preserving Adversarial Machine Learning for Robust Classification in Cybersecurity and Intrusion Detection Domains: A Survey,” J. Cybersecurity Priv. 2022, Vol. 2, Pages 154-190, vol. 2, no. 1, pp. 154–190, Mar. 2022, doi: 10.3390/JCP2010010.

C. Szegedy et al., “Intriguing properties of neural networks,” 2nd Int. Conf. Learn. Represent. ICLR 2014 - Conf. Track Proc., Dec. 2013, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/1312.6199

N. Carlini and D. Wagner, “Towards Evaluating the Robustness of Neural Networks,” Proc. - IEEE Symp. Secur. Priv., pp. 39–57, Aug. 2016, doi: 10.1109/SP.2017.49.

F. Croce et al., “RobustBench: a standardized adversarial robustness benchmark,” Adv. Neural Inf. Process. Syst., Oct. 2020, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/2010.09670

K. Lucas, M. Sharif, L. Bauer, M. K. Reiter, and S. Shintre, “Malware Makeover: Breaking ML-based Static Analysis by Modifying Executable Bytes,” ASIA CCS 2021 - Proc. 2021 ACM Asia Conf. Comput. Commun. Secur., pp. 744–758, Oct. 2021, doi: 10.1145/3433210.3453086.

L. Demetrio, S. E. Coull, B. Biggio, G. Lagorio, A. Armando, and F. Roli, “Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection,” ACM Trans. Priv. Secur., vol. 24, no. 4, Mar. 2021, doi: 10.1145/3473039.

M. Kozák and M. Jureček, “Effectiveness of Adversarial Benign and Malware Examples in Evasion and Poisoning Attacks,” Mach. Learn. Deep Learn. AI Cybersecurity, pp. 267–290, Jan. 2025, doi: 10.1007/978-3-031-83157-7_10.

D. Košťál and M. Jureček, “Building an Adversarial Malware Dataset by Family and Type: Generation, Evasion, and Poisoning Evaluation,” May 2026, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/2605.25937

K. Aryal, M. Gupta, M. Abdelsalam, P. Kunwar, and B. Thuraisingham, “A Survey on Adversarial Attacks for Malware Analysis,” IEEE Access, vol. 13, pp. 428–459, 2025, doi: 10.1109/ACCESS.2024.3519524.

Y. Chen, S. Wang, Y. Qin, X. Liao, S. Jana, and D. Wagner, “Learning Security Classifiers with Verified Global Robustness Properties,” Proc. ACM Conf. Comput. Commun. Secur., pp. 477–494, Dec. 2021, doi: 10.1145/3460120.3484776.

K. Kireev, B. Kulynych, and C. Troncoso, “Adversarial Robustness for Tabular Data through Cost and Utility Awareness,” 30th Annu. Netw. Distrib. Syst. Secur. Symp. NDSS 2023, Aug. 2022, doi: 10.14722/ndss.2023.24924.

O. Jurečková, M. Jureček, M. Kozák, and R. Lórencz, “Adversarial Co-Evolution of Malware and Detection Models: A Bilevel Optimization Perspective,” Proc. Int. Conf. Secur. Cryptogr., pp. 938–947, Apr. 2026, doi: 10.5220/0015191500004103.

M.-I. Nicolae et al., “Adversarial Robustness Toolbox v1.0.0,” Jul. 2018, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/1807.01069

D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, “Robustness May Be at Odds with Accuracy,” 7th Int. Conf. Learn. Represent. ICLR 2019, May 2018, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/1805.12152

Y. Chen, Z. Ding, and D. Wagner, “Continuous Learning for Android Malware Detection,” Jun. 2023, Accessed: Nov. 05, 2026. [Online]. Available: https://arxiv.org/pdf/2302.04332v2

L. Yang, A. Ciptadi, I. Laziuk, A. Ahmadzadeh, and G. Wang, “BODMAS: An Open Dataset for Learning based Temporal Analysis of PE Malware,” Proc. - 2021 IEEE Symp. Secur. Priv. Work. SPW 2021, pp. 78–84, May 2021, doi: 10.1109/SPW53761.2021.00020.

A. Nadeem et al, “SoK: Explainable Machine Learning for Computer Security Applications,” 2023 IEEE 8th Eur. Symp. Secur. Priv. (EuroS&P), Delft, Netherlands, pp. 221–240, 2023, doi: 10.1109/EuroSP57164.2023.00022.