A Comparative Study of Automated Approaches for Detecting Subjectivity and Unverifiability in Natural Language Software Requirements

Muhammad Arsalan Iltaf; Aamer Nadeem

Authors

Muhammad Arsalan Iltaf Capital University of Science and Technology, Islamabad
Aamer Nadeem Capital University of Science and Technology, Islamabad

Keywords:

Requirement Smells, Subjectivity Detection, Unverifiability Detection, Transformer-Based Models, Hybrid Machine-Learning, DistilBERT

Abstract

Natural language software requirements are prone to subjectivity and unverifiability, which reduces clarity and testability. This study provides a controlled comparative evaluation of four automated detection approaches: rule-based lexicons, classical machine-learning models with TF-IDF features, a fine-tuned DistilBERT transformer, and a feature-level hybrid model that concatenates rule-based linguistic indicators with DistilBERT contextual embeddings on the same manually annotated dataset of 985 requirements, including 259 subjective and 726 objective items, and 165 unverifiable and 820 verifiable items. All models used identical preprocessing, an 80:20 stratified train-test split, and the same evaluation metrics. The rule-based approach achieved a precision of 0.67 and a recall of 0.04 for subjectivity and zero recall for unverifiability. Classical ML models reached F1-scores ranging from 0.47 to 0.58 (positive class). DistilBERT obtained weighted F1-scores of 0.74 (subjectivity) and 0.86 (unverifiability). The feature-level hybrid model improved subjectivity detection to a weighted F1-score of 0.79 while matching DistilBERT at 0.86 for unverifiability. These results demonstrate that combining explicit linguistic cues with contextual embeddings improves detection performance under class imbalance.

References

“Natural Language Processing In Requirements Engineering And Its Challenges For Requirements Modelling In The Engineering Design Domain.” Accessed: Apr. 21, 2026. [Online]. Available: https://www.researchgate.net/publication/371694672_NATURAL_LANGUAGE_PROCESSING_IN_REQUIREMENTS_ENGINEERING_AND_ITS_CHALLENGES_FOR_REQUIREMENTS_MODELLING_IN_THE_ENGINEERING_DESIGN_DOMAIN

M. Tukur, S. Umar, and J. Hassine, “Requirement Engineering Challenges: A Systematic Mapping Study on the Academic and the Industrial Perspective,” Arab. J. Sci. Eng. 2021 464, vol. 46, no. 4, pp. 3723–3748, Jan. 2021, doi: 10.1007/s13369-020-05159-1.

H. Femmer, D. Méndez Fernández, S. Wagner, S. Eder, “Rapid quality assurance with Requirements Smells,” arXiv:1611.08847, 2016, [Online]. Available: https://arxiv.org/abs/1611.08847

D. M. Fernández et al., “Naming the pain in requirements engineering: Contemporary problems, causes, and effects in practice,” Empir. Softw. Eng., vol. 22, no. 5, pp. 2298–2338, Oct. 2017, doi: 10.1007/S10664-016-9451-7.

H. Villamizar, T. Escovedo, and M. Kalinowski, “Requirements Engineering for Machine Learning: A Systematic Mapping Study,” Proc. - 2021 47th Euromicro Conf. Softw. Eng. Adv. Appl. SEAA 2021, pp. 29–36, Sep. 2021, doi: 10.1109/SEAA53835.2021.00013.

Alvaro Veizaga, Seung Yeob Shin, Lionel C. Briand, “Automated Smell Detection and Recommendation in Natural Language Requirements,” arXiv:2305.07097, 2023, [Online]. Available: https://arxiv.org/abs/2305.07097

V. Gervasi, A. Ferrari, D. Zowghi, and P. Spoletini, “Ambiguity in Requirements Engineering: Towards a Unifying Framework,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 11865 LNCS, pp. 191–210, 2019, doi: 10.1007/978-3-030-30985-5_12.

“ISO/IEC/IEEE International Standard - Systems and software engineering -- Life cycle processes -- Requirements engineering,” Art. no. 29148–2011, Oct. 2018, doi: 10.1109/IEEESTD.2018.8559686.

“(PDF) An Automatic Quality Evaluation for Natural Language Requirements.” Accessed: Mar. 30, 2026. [Online]. Available: https://www.researchgate.net/publication/244206643_An_Automatic_Quality_Evaluation_for_Natural_Language_Requirements

A. Ferrari et al., “Detecting requirements defects with NLP patterns: an industrial experience in the railway domain,” Empir. Softw. Eng. 2018 236, vol. 23, no. 6, pp. 3684–3733, Feb. 2018, doi: 10.1007/s10664-018-9596-7.

M. Q. Riaz, W. H. Butt, and S. Rehman, “Automatic Detection of Ambiguous Software Requirements: An Insight,” 5th Int. Conf. Inf. Manag. ICIM 2019, pp. 1–6, May 2019, doi: 10.1109/INFOMAN.2019.8714682.

V. Patel, P. Mehta, and K. Lavingia, “Software Requirement Classification Using Machine Learning Algorithms,” 2023 Int. Conf. Artif. Intell. Appl. ICAIA 2023 Alliance Technol. Conf. ATCON-1 2023 - Proceeding, 2023, doi: 10.1109/ICAIA57370.2023.10169588.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv:1810.04805, 2018, [Online]. Available: https://arxiv.org/abs/1810.04805

Israr Ali, Syed Sajjad Hussain Rizvi, “Enhancing Software Quality with AI: A Transformer-Based Approach for Code Smell Detection,” Appl. Sci., vol. 15, no. 8, p. 4559, 2025, doi: 10.3390/app15084559.

A. Rahali and M. A. Akhloufi, “End-to-End Transformer-Based Models in Textual-Based NLP,” AI 2023, Vol. 4, Pages 54-110, vol. 4, no. 1, pp. 54–110, Jan. 2023, doi: 10.3390/AI4010004.

J. Peer, Y. Mordecai, and Y. Reich, “NLP4ReF: Requirements Classification and Forecasting: From Model-Based Design to Large Language Models,” IEEE Aerosp. Conf. Proc., 2024, doi: 10.1109/AERO58975.2024.10521022.

Sallam Abualhaija, Chetan Arora, Mehrdad Sabetzadeh, Lionel C. Briand & Michael Traynor, “Automated demarcation of requirements in textual specifications: a machine learning-based approach,” Empir. Softw. Eng., vol. 25, pp. 5454–5497, 2020, [Online]. Available: https://link.springer.com/article/10.1007/s10664-020-09864-1

I. P. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, “Attention Is All You Need,” arXiv:1706.03762, 2017, doi: https://doi.org/10.48550/arXiv.1706.03762.

Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, Sivanesan Sangeetha, “AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing,” arXiv:2108.05542, 2021, [Online]. Available: https://arxiv.org/abs/2108.05542

Nadia Mushtaq Gardazi, Ali Daud, Muhammad Kamran Malik, Amal Bukhari, Tariq Alsahfi, “BERT applications in natural language processing: a review,” Artif. Intell. Rev., vol. 58, no. 166, 2025, [Online]. Available: https://link.springer.com/article/10.1007/s10462-025-11162-5

Anna Rogers, Olga Kovaleva, Anna Rumshisky, “A Primer in BERTology: What we know about how BERT works,” arXiv:2002.12327, 2020, [Online]. Available: https://arxiv.org/abs/2002.12327

Ashagrew Liyih Alem, Ketema Keflie Gebretsadik, Shegaw Anagaw Mengistie & Muluye Fentie Admas, “Multi-label software requirement smells classification using deep learning,” Sci. Rep., 2025, [Online]. Available: https://www.nature.com/articles/s41598-025-86673-w

M. K. Habib, S. Wagner, and D. Graziotin, “Detecting Requirements Smells with Deep Learning: Experiences, Challenges and Future Work,” Proc. IEEE Int. Conf. Requir. Eng., vol. 2021-September, pp. 153–156, Sep. 2021, doi: 10.1109/REW53955.2021.00027.

Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, Jianfeng Gao, “Deep Learning Based Text Classification: A Comprehensive Review,” arXiv:2004.03705, 2021, [Online]. Available: https://arxiv.org/abs/2004.03705

T. B. Brown et al., “Language Models are Few-Shot Learners,” Adv. Neural Inf. Process. Syst., vol. 2020-December, May 2020, Accessed: Sep. 26, 2024. [Online]. Available: https://arxiv.org/abs/2005.14165v4

S. Ezzini, S. Abualhaija, C. Arora, M. Sabetzadeh, and L. C. Briand, “Using domain-specific corpora for improved handling of ambiguity in requirements,” Proc. - Int. Conf. Softw. Eng., pp. 1485–1497, Nov. 2021, doi: 10.1109/ICSE43902.2021.00133.

Saad Ezzini, Sallam Abualhaija, Chetan Arora, Mehrdad Sabetzadeh, “TAPHSIR: Towards AnaPHoric Ambiguity Detection and ReSolution In Requirements,” arXiv:2206.10227, 2022, [Online]. Available: https://arxiv.org/abs/2206.10227

A. Fantechi, S. Gnesi, and L. Semini, “Rule-based NLP vs ChatGPT in ambiguity detection, a preliminary study,” REFSQ Work., 2023.

Qixiang Zhou, Tong Li, “Assisting in requirements goal modeling: a hybrid approach based on machine learning and logical reasoning,” Proc. - 25th ACM/IEEE Int. Conf. Model Driven Eng. Lang. Syst. Model. 2022, 2022, [Online]. Available: https://dl.acm.org/doi/10.1145/3550355.3552415

“ERTMS/ETCS System Requirements Specification”, [Online]. Available: https://www.era.europa.eu/system/files/2023-01/sos1_index001_-_era_ertms_003204_v500.pdf