Challenging the Transformer: A Comparative Study of CNN and RoBERTa for Hate Speech Detection on Imbalanced Data

Samreen Yousaf; Sabeen Masood; Maryam Nageen; Farah Haneef; Afsheen Masood

Authors

Samreen Yousaf Dept. of Computer Science, Capital University of Science and Technology, Islamabad, Pakistan
Sabeen Masood Dept. of Software Engineering, Capital University of Science and Technology, Islamabad, Pakistan
Maryam Nageen Dept. of Computer Science, Capital University of Science and Technology, Islamabad, Pakistan
Farah Haneef Dept. of Software Engineering, Capital University of Science and Technology, Islamabad, Pakistan
Afsheen Masood Dept. of Software Engineering, Capital University of Science and Technology, Islamabad, Pakistan

Keywords:

Hate Speech Detection, Transformer Models, Social Media Analysis, Convolutional Neural Network, Natural Language Processing, Imbalanced Classification

Abstract

Introduction/Importance of Study: Hate speech on social media is becoming more prevalent; this highlights the need for effective automated detection technologies, which are essential to creating safer online communities.

Novelty Statement: This paper reflects original comparative research showing that a Convolutional Neural Network (CNN) performs more robustly than a fine-tuned RoBERTa model for hate speech classification on imbalanced Twitter data.

Material and Method: This study was conducted through a controlled comparison of two deep learning architectures: a fine-tuned RoBERTa Transformer and a CNN. Both models were trained and evaluated on a dataset of 24,783 tweets characterized by severe class imbalance, using identical class-weighted loss functions to ensure a fair evaluation of their inherent architectural characteristics.

Result and Discussion: Experimental results revealed that the CNN achieved superior accuracy (87.23%) and F1-score (0.92), demonstrating that it is more resistant to class imbalance than the RoBERTa model, which exhibited significant classification bias toward the majority class. Qualitative and computational analysis indicates that while Transformers offer deeper contextual understanding, CNNs provide a more stable and efficient baseline for real-time moderation on skewed datasets, with Transformer performance improvable via weighted loss functions and synthetic embedding generation.

Concluding Remarks: For hate speech detection in short-text, imbalanced social media data, simpler CNN architectures can offer greater robustness and practical efficiency than advanced Transformer models, though targeted techniques can mitigate Transformer limitations.

References

Li Zheng, Hao Fei, Ting Dai, Zuquan Peng, Fei Li, Huisheng Ma, Chong Teng, Donghong Ji, “Multi-Granular Multimodal Clue Fusion for Meme Understanding,” arXiv:2503.12560, 2025, [Online]. Available: https://arxiv.org/abs/2503.12560

G. Arya, “Multimodal Hate Speech Detection in Memes Using Contrastive Language-Image Pre-Training,” IEEE Access, vol. 12, pp. 22359–22375, 2024, doi: 10.1109/ACCESS.2024.3361322.

Eftekhar Hossain, Omar Sharif, Mohammed Moshiul Hoque, Sarah M. Preum, “Deciphering Hate: Identifying Hateful Memes and Their Targets,” ACL Anthol., 2024, [Online]. Available: https://aclanthology.org/2024.acl-long.454/

Shraman Pramanick, Dimitar Dimitrov, Rituparna Mukherjee, Shivam Sharma, Md. Shad Akhtar, Preslav Nakov, Tanmoy Chakraborty, “Detecting Harmful Memes and Their Targets,” ACL Anthol., 2021, [Online]. Available: https://aclanthology.org/2021.findings-acl.246/

A. Duggal, A. Singh, and D. Garg, “Improving Safety on the Internet: A New Multimodal Framework for Hateful Meme Classification,” ICDT 2025 - 3rd Int. Conf. Disruptive Technol., pp. 195–200, 2025, doi: 10.1109/ICDT63985.2025.10986544.

Jesus Armenta-Segura, César Jesús Núñez-Prado, Grigori Olegovich Sidorov, Alexander Gelbukh, Rodrigo Francisco Román-Godínez, “Ometeotl@Multimodal Hate Speech Event Detection 2023: Hate Speech and Text-Image Correlation Detection in Real Life Memes Using Pre-Trained BERT Models over Text,” ACL Anthol., 2023, [Online]. Available: https://aclanthology.org/2023.case-1.7/

Shivam Sharma, Firoj Alam, Md. Shad Akhtar, Dimitar Dimitrov, Giovanni Da San Martino, “Detecting and Understanding Harmful Memes: A Survey,” arXiv:2205.04274, 2022, [Online]. Available: https://arxiv.org/abs/2205.04274

Marzieh Mozafari , Reza Farahbakhsh, Noël Crespi, “Hate speech detection and racial bias mitigation in social media based on BERT model,” PLoS One, 2020, doi: https://doi.org/10.1371/journal.pone.0237861.

A. C. Mazari, N. Boudoukhani, and A. Djeffal, “BERT-based ensemble learning for multi-aspect hate speech detection,” Clust. Comput. 2023 271, vol. 27, no. 1, pp. 325–339, Jan. 2023, doi: 10.1007/s10586-022-03956-x.

Umera Wajeed Pasha, “Multilingual Sexism Detection in Memes, A CLIP - Enhanced Machine Learning Approach,” Conf. Labs Eval. Forum, 2024, [Online]. Available: https://ceur-ws.org/Vol-3740/paper-107.pdf

Njung’e Fredrick. Ng’ang’a, Aaron M. Oirere, “A Comparative Study of Transformer-based Models for Hate-Speech Detection in English-Kiswahili Code-Switched Social Media Text,” Int. J. Adv. Trends Comput. Sci. Eng., vol. 13, no. 5, 2024, [Online]. Available: http://repository.mut.ac.ke:8080/xmlui/handle/123456789/6483

M. Chakarverti, A. Goswami, and A. Yadav, “Comparative Evaluation of Pre-Trained Models for Hate Speech Detection on Social Media,” 2024 15th Int. Conf. Comput. Commun. Netw. Technol. ICCCNT 2024, 2024, doi: 10.1109/ICCCNT61001.2024.10723959.

U. Mittal, “Detecting Hate Speech Utilizing Deep Convolutional Network and Transformer Models,” IEEE Int. Conf. Electr. Electron. Commun. Comput. ELEXCOM 2023, 2023, doi: 10.1109/ELEXCOM58812.2023.10370502.

Zainab Mansur, Nazlia Omar, “Twitter Hate Speech Detection: A Systematic Review of Methods, Taxonomy Analysis, Challenges, and Opportunities,” IEEE Access, vol. 11, 2023, [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10025718

M. U. Arshad, R. Ali, M. O. Beg, and W. Shahzad, “UHated: hate speech detection in Urdu language using transfer learning,” Lang. Resour. Eval. 2023 572, vol. 57, no. 2, pp. 713–732, Feb. 2023, doi: 10.1007/s10579-023-09642-7.

Swapnanil Mukherjee, Sujit Das, “Application of Transformer-Based Language Models to Detect Hate Speech in Social Media,” J. Comput. Cogn. Eng., vol. 2, no. 4, pp. 278–286, 2021, doi: 10.47852/bonviewJCCE2022010102.

J. A. Siddiqui, S. S. Yuhaniz, G. M. Shaikh, S. A. Soomro and Z. A. Mahar, “Fine-Grained Multilingual Hate Speech Detection Using Explainable AI and Transformers,” IEEE Access, vol. 12, pp. 143177–143192, 2024, doi: 10.1109/ACCESS.2024.3470901.

Adhe Akram Azhari, Yuliant Sibaroni, “Detection of Indonesian Hate Speech in the Comments Column of Indone-sian Artists’ Instagram Using the RoBERTa Method,” JIPI (Jurnal Ilm. Penelit. dan Pembelajaran Inform., vol. 8, no. 3, pp. 764–773, 2023, doi: 10.29100/jipi.v8i3.3898.

Deepawali Sharma, Vivek Kumar Singh & Vedika Gupta, “TABHATE: A Target-based hate speech detection dataset in Hindi,” Soc. Netw. Anal. Min., vol. 14, no. 190, 2024, [Online]. Available: https://link.springer.com/article/10.1007/s13278-024-01355-1

Shivang Agarwal, Ankur Sonawane, “Accelerating automatic hate speech detection using parallelized ensemble learning models,” Expert Syst. Appl., vol. 230, p. 120564, 2023, doi: https://doi.org/10.1016/j.eswa.2023.120564.

S. Kamal, A. Yadav, and V. Singh, “PRISM: Profiling Hate Speech Spreaders using SVM and RoBERTa Models,” 2nd IEEE Int. Conf. Innov. High-Speed Commun. Signal Process. IHCSP 2024, 2024, doi: 10.1109/IHCSP63227.2024.10960186.

M. S. Mohamed, H. Elzayady, K. M. Badran, and G. I. Salama, “An efficient approach for data-imbalanced hate speech detection in Arabic social media,” J. Intell. Fuzzy Syst., vol. 45, no. 4, pp. 6381–6390, Oct. 2023, doi: 10.3233/JIFS-231151.

A. Madhukar, A. Madhukar, Anubhav, Ishan, and S. Nagpal, “An Ensemble Based Approach to Detect Hate Speech,” 2024 IEEE Reg. 10 Symp. TENSYMP 2024, 2024, doi: 10.1109/TENSYMP61132.2024.10752152.

Endrit Fetahi, Arsim Susuri, Mentor Hamiti, Zenun Kastrati, Ercan Canhasi, “Enhancing social media hate speech detection in low-resource languages using transformers and explainable AI,” Soc. Netw. Anal. Min., vol. 15, no. 82, 2025, [Online]. Available: https://link.springer.com/article/10.1007/s13278-025-01497-w

Deepawali Sharma, Vedika Gupta, “Stop the Hate, Spread the Hope: An Ensemble Model for Hope Speech Detection in English and Dravidian Languages,” ACM Trans. Asian Low-Resource Lang. Inf. Process., 2025, [Online]. Available: https://dl.acm.org/doi/abs/10.1145/3716383

Anna Glazkova, “A comparison of text preprocessing techniques for hate and offensive speech detection in Twitter,” Soc. Netw. Anal. Min., vol. 13, no. 155, 2023, [Online]. Available: https://link.springer.com/article/10.1007/s13278-023-01156-y

A. Wicaksana, K. Sorensen, and F. Dinarta, “Enhancing Hate Speech Detection in Mixed-Language Texts: A Comparative Study of BLOOM and XLM-RoBERTa Models,” 2025 17th Int. Conf. Comput. Autom. Eng. ICCAE 2025, pp. 419–425, 2025, doi: 10.1109/ICCAE64891.2025.10980554.