Enhanced Emotion Recognition on the FER-2013 Dataset by Training VGG from Scratch

Muhammad Talha Jahangir; Syed Yameen Ali; Ali Hasnain; Muhammad Faizan; Mohsin Jabbar; Muhammad Kashan Basit; Abdul Rafey

Authors

Muhammad Talha Jahangir Department of Computer Science, MNS-University of Engineering and Technology, Multan, Pakistan
Syed Yameen Ali Department of Computer Science, MNS-University of Engineering and Technology, Multan, Pakistan
Ali Hasnain College of Electrical and Mechanical Engineering, NUST, Pakistan
Muhammad Faizan Department of Computer Science, MNS-University of Engineering and Technology, Multan, Pakistan
Mohsin Jabbar Department of Artificial Intelligence NASTP Institute of Information Technology, Lahore, Pakistan
Muhammad Kashan Basit Department of Computer Science, MNS-University of Engineering and Technology, Multan, Pakistan
Abdul Rafey Department of Computer Science, MNS-University of Engineering and Technology, Multan, Pakistan

Keywords:

Facial Emotion Recognition, Convolutional, Neural Network, FER-2013 Dataset, Pre-Trained Models, Fine Tuning, VGG16, Res Net 50.

Abstract

Recognizing facial emotions is still a major obstacle in computer vision, particularly when dealing with complex datasets such as FER-2013. Advancements in deep learning have simplified the process of achieving high accuracy, yet obtaining high accuracy on the FER-2013 dataset with traditional methods remains challenging. The aim of this research is to analyze the effectiveness of Convolutional Neural Networks (CNNs) utilizing VGG16 and ResNet50 architectures through three different training methods: training from scratch, with transfer learning, and fine-tuning. Our research demonstrates that while training VGG16 from scratch achieved a validation accuracy of 67.23%, fine-tuning produced a slight reduction in performance at 64.80%. Conversely, ResNet50 struggled across all approaches, with the highest validation accuracy being only 54.69% when trained from scratch. We offer an in-depth analysis of these methodologies by utilizing confusion matrices, training durations, and accuracy measures to showcase the balance between computational expenses and model effectiveness. Our results indicate that, although transfer learning and fine-tuning offer rapid convergence, training from scratch may still be necessary for specialized feature learning in complex FER tasks. These results help in the continuous work of improving emotion recognition systems by maintaining a balance between accuracy and computational efficiency.

References

M. Bansal, M. Kumar, M. Sachdeva, and A. Mittal, “Transfer learning for image classification using VGG19: Caltech-101 image data set,” J. Ambient Intell. Humaniz. Comput., vol. 14, no. 4, pp. 3609–3620, Apr. 2023, doi: 10.1007/S12652-021-03488-Z/TABLES/8.

A. Voulodimos, N. Doulamis, A. Doulamis, and E. Protopapadakis, “Deep Learning for Computer Vision: A Brief Review,” Comput. Intell. Neurosci., vol. 2018, 2018, doi: 10.1155/2018/7068349.

M. R. Appasaheb Borgalli and D. S. Surve, “Deep learning for facial emotion recognition using custom CNN architecture,” J. Phys. Conf. Ser., vol. 2236, no. 1, p. 012004, Mar. 2022, doi: 10.1088/1742-6596/2236/1/012004.

“Deep Learning.” Accessed: Oct. 23, 2024. [Online]. Available: https://www.deeplearningbook.org/

S. Yousefi, M. P. Nguyen, N. Kehtarnavaz, and Y. Cao, “Facial expression recognition based on diffeomorphic matching,” Proc. - Int. Conf. Image Process. ICIP, pp. 4549–4552, 2010, doi: 10.1109/ICIP.2010.5650670.

H. Huang, “A facial expression recognition method based on Convolutional Neural Network,” Front. Comput. Intell. Syst., vol. 2, no. 1, pp. 116–119, Nov. 2022, doi: 10.54097/FCIS.V2I1.3178.

K. Sarvakar, R. Senkamalavalli, S. Raghavendra, J. Santosh Kumar, R. Manjunath, and S. Jaiswal, “Facial emotion recognition using convolutional neural networks,” Mater. Today Proc., vol. 80, pp. 3560–3564, Jan. 2023, doi: 10.1016/J.MATPR.2021.07.297.

L. Zahara, P. Musa, E. Prasetyo Wibowo, I. Karim, and S. Bahri Musa, “The Facial Emotion Recognition (FER-2013) Dataset for Prediction System of Micro-Expressions Face Using the Convolutional Neural Network (CNN) Algorithm based Raspberry Pi,” 2020 5th Int. Conf. Informatics Comput. ICIC 2020, Nov. 2020, doi: 10.1109/ICIC50835.2020.9288560.

J. Xiang and G. Zhu, “Joint face detection and facial expression recognition with MTCNN,” Proc. - 2017 4th Int. Conf. Inf. Sci. Control Eng. ICISCE 2017, pp. 424–427, Nov. 2017, doi: 10.1109/ICISCE.2017.95.

“Facial Emotion Detection Using Neural Network.” Accessed: Oct. 23, 2024. [Online]. Available: https://www.researchgate.net/publication/344331972_Facial_Emotion_Detection_Using_Neural_Network

M. Mukhopadhyay, S. Pal, A. Nayyar, P. K. D. Pramanik, N. Dasgupta, and P. Choudhury, “Facial Emotion Detection to Assess Learner’s State of Mind in an Online Learning System,” ACM Int. Conf. Proceeding Ser., pp. 107–115, Feb. 2020, doi: 10.1145/3385209.3385231.

D. K. R. Gaddam, M. D. Ansari, S. Vuppala, V. K. Gunjan, and M. M. Sati, “Human Facial Emotion Detection Using Deep Learning,” Lect. Notes Electr. Eng., vol. 783, pp. 1417–1427, 2022, doi: 10.1007/978-981-16-3690-5_136.

S. Singh and F. Nasoz, “Facial Expression Recognition with Convolutional Neural Networks,” 2020 10th Annu. Comput. Commun. Work. Conf. CCWC 2020, pp. 324–328, Jan. 2020, doi: 10.1109/CCWC47524.2020.9031283.

R. M. D, H. H. Kenchannavar, and U. P. Kulkarni, “Facial Emotion Recognition using Three-Layer ConvNet with Diversity in Data and Minimum Epochs,” Int. J. Intell. Syst. Appl. Eng., vol. 10, no. 4, pp. 264–268, Dec. 2022, Accessed: Oct. 23, 2024. [Online]. Available: https://ijisae.org/index.php/IJISAE/article/view/2225

M. Coşkun Irmak, M. Bilge Han Taş, S. Turan, and A. Haşıloğlu, “Emotion Analysis from Facial Expressions Using Convolutional Neural Networks,” Proc. - 6th Int. Conf. Comput. Sci. Eng. UBMK 2021, pp. 570–574, 2021, doi: 10.1109/UBMK52708.2021.9558917.

D. Bhagat, A. Vakil, R. K. Gupta, and A. Kumar, “Facial Emotion Recognition (FER) using Convolutional Neural Network (CNN),” Procedia Comput. Sci., vol. 235, pp. 2079–2089, Jan. 2024, doi: 10.1016/J.PROCS.2024.04.197.

K. Lalli and M. Senbagavalli, “Enhancing Deep Learning for Autism Spectrum Disorder Detection with Dual-Encoder GAN-based Augmentation of Electroencephalogram Data,” Salud, Cienc. y Tecnol. - Ser. Conf., vol. 3, pp. 958–958, Jan. 2024, doi: 10.56294/SCTCONF2024958.

“FER-2013.” Accessed: Oct. 23, 2024. [Online]. Available: https://www.kaggle.com/datasets/msambare/fer2013

H. D. Nguyen, S. Yeom, G. S. Lee, H. J. Yang, I. S. Na, and S. H. Kim, “Facial Emotion Recognition Using an Ensemble of Multi-Level Convolutional Neural Networks,” Int. J. Pattern Recognit. Artif. Intell., vol. 33, no. 11, Oct. 2019, doi: 10.1142/S0218001419400159.

I. J. Goodfellow et al., “Challenges in representation learning: A report on three machine learning contests,” Neural Networks, vol. 64, pp. 59–63, 2015, doi: 10.1016/j.neunet.2014.09.005.

“Exploring ResNet50: An In-Depth Look at the Model Architecture and Code Implementation | by Nitish Kundu | Medium.” Accessed: Oct. 12, 2024. [Online]. Available: https://medium.com/@nitishkundu1993/exploring-resnet50-an-in-depth-look-at-the-model-architecture-and-code-implementation-d8d8fa67e46f