Automating Severity Prediction of Bug Reports Using Large Language Models (LLM)

Abstract

Accurate prediction of bug report severity is essential for prioritizing software-maintenance activities, but manual severity assignment is costly, time-consuming, and potentially inconsistent. This study investigates prompt-based open-weight Large Language Models for automatically classifying bug reports as severe or non-severe without task-specific fine-tuning. The experiments used 3,535 bug reports from the CDT, JDT, Bugzilla, and Thunderbird projects. The original severity labels were mapped into two categories: blocker, critical, and major as severe, and normal, minor, and trivial as non-severe. LLaMA 3.1:8B and Qwen3:8B were evaluated under zero-shot and few-shot prompting and compared with Support Vector Machine, Multinomial Naive Bayes, Random Forest, and Long Short-Term Memory models using TF-IDF and Word2Vec representations. On the held-out test set, Qwen3:8B with few-shot prompting achieved 98.6% accuracy, 100% precision, 97.5% recall, and a 98.7% F1-score. Compared with the strongest conventional baseline, Random Forest with TF-IDF, it improved accuracy and F1-score by 26.6 and 24.2 percentage points, respectively. On an additional dataset of 200 synthetically generated bug reports, LLaMA 3.1:8B with zero-shot prompting achieved 97.99% accuracy and a 98.51% F1-score, while Qwen3:8B with few-shot prompting achieved 97.9% accuracy and a 97.5% F1-score. These results indicate that prompt-based LLMs can substantially reduce task-specific preprocessing and training requirements while providing high predictive performance. However, further evaluation on independent, human-authored, cross-project datasets is required before making broad claims regarding real-world generalization.