Modified Convolutional Neural Networks for Facial Emotion Classification
Keywords:
Facial expression recognition, Facial Action Coding System, machine learning, CNN (Convolutional Neural Network), Artificial Intelligence, Deep Learning, Extended Cohn-Kanade (CK+)Abstract
Facial expression analysis is a fascinating yet challenging problem in the realm of artificial intelligence. The vast variability in human expressions poses a significant hurdle for machine learning methods to detect them accurately. Recently, machine learning and deep learning approaches have made notable strides in this area, leveraging Deep Neural Networks (DNNs) to identify human emotions. Convolutional Neural Networks (CNNs), in particular, have proven effective in resolving the complexities involved in human facial expressions, making them a preferred choice for these tasks. In this study, we proposed a modified CNN architecture by introducing a new layer to enhance accuracy. The CNN network is trained on both frontal face images and images with varying poses. We utilized three distinct datasets FER 2013, CK+ and our own dataset to achieve the desired results. The evaluation results obtained using the proposed network surpass those achieved by conventional CNN networks. Notably, our proposed network achieves an average accuracy of 97.5% on our collected dataset.
References
Y. Wang, “Research on the Construction of Human-Computer Interaction System Based on a Machine Learning Algorithm,” J. Sensors, 2022, doi: https://doi.org/10.1155/2022/3817226.
Y. G. and Y. S. Y. -J. Liu, M. Yu, G. Zhao, J. Song, “Real-Time Movie-Induced Discrete Emotion Recognition from EEG Signals,” IEEE Trans. Affect. Comput., vol. 9, no. 4, pp. 550–562, 2018, doi: 10.1109/TAFFC.2017.2660485.
S. L. Dingus, Thomas A, Feng Guo, “Driver crash risk factors and prevalence evaluation using naturalistic driving data,” Proc. Natl. Acade, vol. 13, no. 10, pp. 2636–2641, 2016, doi: 10.1073/pnas.1513271113.
S. M. H. Garcia, “CK+,” DataCite Commons, 2024, [Online]. Available: https://commons.datacite.org/doi.org/10.5281/zenodo.11221350
G. H. et Al, “Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups,” IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82–97, 2012, doi: 10.1109/MSP.2012.2205597.
M. J. M. P S Bellet, “The Importance of Empathy as an Interviewing Skill in Medicine,” JAMA, vol. 266, no. 13, pp. 1831–2, 1991, [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/1909761/
and I. T. K. Han, D. Yu, “Speech Emotion Recognition Using Deep Neural Network and Extreme Learning Machine,” Interspeech, 2014, doi: 0.21437/Interspeech.2014-57.
H. L. and E. M. P. Y. Kim, “"Deep learning for robust feature generation in audiovisual emotion recognition,” IEEE Int. Conf. Acoust. Speech Signal Process. Vancouver, BC, Canada, pp. 3687–3691, 2013, doi: 10.1109/ICASSP.2013.6638346.
B. C. Ko, “A Brief Review of Facial Emotion Recognition Based on Visual Information,” Sensors (Basel), vol. 18, no. 2, p. 401, 2018, doi: 10.3390/s18020401.
and L. C. H. Li, J. Sun, Z. Xu, “Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network,” IEEE Trans. Multimed., vol. 19, no. 12, pp. 2816–2831, 2017, doi: 10.1109/TMM.2017.2713408.
and W. B. H. Bejaoui, H. Ghazouani, “Fully Automated Facial Expression Recognition Using 3D Morphable Model and Mesh-Local Binary Pattern,” Adv. Concepts Intell. Vis. Syst., pp. 39–50, 2017, doi: 10.1007/978-3-319-70353-4_4.
P. E. and W. V. Friesen, “Constants across cultures in the face and emotion,” J. Personal. Soc. Ps, vol. 17, no. 2, pp. 124–9, 1971, doi: 10.1037/h0030377.
and T. J. J. R. Venkatesan, S. Shirly, M. Selvarathi, “Human Emotion Detection Using DeepFace and Artificial Intelligence,” Eng. Proc., 2023, doi: 10.3390/engproc2023059037.
X. X. Yu-Gang Jiang, Baohan Xu, “Predicting emotions in user-generated videos”, [Online]. Available: https://cdn.aaai.org/ojs/8724/8724-13-12252-1-2-20201228.pdf
M. Aslan, “CNN based efficient approach for emotion recognition,” J. King Saud Univ. - Comput. Inf. Sci., vol. 34, no. 9, pp. 7335–7346, 2022, doi: https://doi.org/10.1016/j.jksuci.2021.08.021.
B. O. Alieh Hajizadeh Saffar , Tiffany Katharine Mann, “Textual emotion detection in health: Advances and applications,” J. Biomed. Inform., 2023, doi: 10.1016/j.jbi.2022.104258.
J. H. Yuwei Chen, “Deep Learning-Based Emotion Detection,” J. Comput. Commun., vol. 10, no. 2, 2022, doi: 10.4236/jcc.2022.102005.
and R. Z. Q. Qi, L. Lin, “Feature Extraction Network with Attention Mechanism for Data Enhancement and Recombination Fusion for Multimodal Sentiment Analysis,” Information, vol. 12, no. 9, p. 342, 2021, doi: https://doi.org/10.3390/info12090342.
S. Du, Y. Tao, and A. M. Martinez, “Compound facial expressions of emotion,” Proc. Natl. Acad. Sci. U. S. A., vol. 111, no. 15, pp. E1454–E1462, Apr. 2014, doi: 10.1073/PNAS.1322355111/ASSET/C2BBF816-8771-46B5-9871-0AA84A513697/ASSETS/GRAPHIC/PNAS.1322355111I91.GIF.
T. K. and J. F. C. Y. . -I. Tian, “Recognizing action units for facial expression analysis,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 2, pp. 97–115, 2001, doi: 10.1109/34.908962.
W. V Ekman, P., & Friesen, “Facial Action Coding System,” APA PsycNet Direct, 1978, doi: https://doi.org/10.1037/t27734-000.
P. P. R. and D.-M. J. J. -H. Kim, B. -G. Kim, “Efficient Facial Expression Recognition Algorithm Based on Hierarchical Deep Neural Network Structure,” IEEE Access, vol. 7, pp. 41273–41285, 2019, doi: 10.1109/ACCESS.2019.2907327.
E. C. D. and R. I. B. Ramdhani, “Convolutional Neural Networks Models for Facial Expression Recognition,” Int. Symp. Adv. Intell. Informatics (SAIN), Yogyakarta, Indones., pp. 96–101, 2018, doi: 10.1109/SAIN.2018.8673352.
and Z. T. Zhang, Tao, “Survey of deep emotion recognition in dynamic data using facial, speech and textual cues,” Multimed. Tools Appl., vol. 83, pp. 66223–66262, 2024, [Online]. Available: https://link.springer.com/article/10.1007/s11042-023-17944-9
G. Muhammad Sajjad, Fath U Min Ullah, Christodoulou and J. J. P. C. R. Ullah, Mohib, Faouzi Alaya Cheikh, Mohammad Hijji, Khan Muhammad, “A comprehensive survey on deep facial expression recognition: challenges, applications, and future guidelines,” Alexandria Eng. J., vol. 68, pp. 817–840, 2023, doi: https://doi.org/10.1016/j.aej.2023.01.017.
S. A. Jain, Deepak Kumar, Ashit Kumar Dutta, Elena Verdú and A. R. W. Sait, “An automated hyperparameter tuned deep learning model enabled facial emotion recognition for autonomous vehicle drivers,” Image Vis. Comput., vol. 133, p. 104659, 2023, doi: https://doi.org/10.1016/j.imavis.2023.104659.
H. W. & C. Z. Fan Zhang, Gongguan Chen, “CF-DAN: Facial-expression recognition based on cross-fusion dual-attention network,” Comput. Vis. Media, vol. 10, pp. 593–608, 2024, doi: https://doi.org/10.1007/s41095-023-0369-x.
and Q. D. Tao, Huanjie, “Hierarchical attention network with progressive feature fusion for facial expression recognition,” Neural Networks, vol. 170, pp. 337–348, 2024, doi: https://doi.org/10.1016/j.neunet.2023.11.033.
A. A. Ali Ezati, Mohammadreza Dezyani, Rajib Rana, Roozbeh Rajabi, “A Lightweight Attention-based Deep Network via Multi-Scale Feature Fusion for Multi-View Facial Expression Recognition,” Comput. Vis. Pattern Recognit., 2024, doi: https://doi.org/10.48550/arXiv.2403.14318.
R. K. and M. G. O. Khajuria, “Facial Emotion Recognition using CNN and VGG-16,” Int. Conf. Inven. Comput. Technol. (ICICT), Lalitpur, Nepal, pp. 472–477, 2023, doi: 10.1109/ICICT57646.2023.10133972.
T. W. Erlangga Satrio Agung , Achmad Pratama Rifai, “Image-based facial emotion recognition using convolutional neural network on emognition dataset,” Sci. Rep., vol. 13, no. 1, p. 14429, 2024, doi: 10.1038/s41598-024-65276-x.
S. S. Fatimatuzzahra, Lindawati, “Development of Convolutional Neural Network Models to Improve Facial Expression Recognition Accuracy,” J. Ilm. Tek. Elektro Komput. dan Inform., vol. 10, no. 2, pp. 279–289, 2024, doi: 10.26555/jiteki.v10i2.28863.
amna kosar hafiz burhan ul haq, waseem akram, muhammad nauman irshad and M. Abid, “Enhanced real-time facial expression recognition using deep learning,” Acadlore Trans. AI Mach. Learn., vol. 3, no. 1, pp. 24–35, 2024, doi: https://doi.org/10.56578/ataiml030103.
H. A. & Y. Ben Ayed, “Deep facial expression detection using Viola-Jones algorithm, CNN-MLP and CNN-SVM,” Soc. Netw. Anal. Min., vol. 14, p. 65, 2024, doi: https://doi.org/10.1007/s13278-024-01231-y.
R. R. & K. K. M. Brijesh Bakariya, Arshdeep Singh, Harmanpreet Singh, Pankaj Raju, “Facial emotion recognition and music recommendation system using CNN-based deep learning techniques,” Evol. Syst., vol. 15, pp. 641–658, 2024, doi: https://doi.org/10.1007/s12530-023-09506-z.
K. P. S. and Y. B. S. M. Mehrotra, “Facial Emotion Recognition and Detection Using Convolutional Neural Networks with Low Computation Cost,” 2nd Int. Conf. Disruptive Technol. (ICDT), Gt. Noida, India, pp. 1349–1354, 2024, doi: 10.1109/ICDT61202.2024.10489678.
and D. J. H. M. K. Chowdary, T. N. Nguyen, “Deep learning-based facial emotion recognition for human–computer interaction applications,” Neural Comput. Appl., vol. 35, no. 4, pp. 1–18, 2021, doi: 10.1007/s00521-021-06012-8.
Z. Z. and S. W. J. Pan, W. Fang, Z. Zhang, B. Chen, “Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG,” IEEE Open J. Eng. Med. Biol., vol. 5, pp. 396–403, 2024, doi: 10.1109/OJEMB.2023.3240280.
A. V. & S. S. Rajesh Singh, Sumeet Saurav, Tarun Kumar, Ravi Saini, “Facial expression recognition in videos using hybrid CNN & ConvLSTM,” Int. J. Inf. Technol., vol. 15, pp. 1819–1830, 2023, doi: https://doi.org/10.1007/s41870-023-01183-0.
N. Shivalila Hangaragi, Tripty Singh, N, “Face Detection and Recognition Using Face Mesh and Deep Neural Network,” Procedia Comput. Sci., vol. 218, pp. 741–749, 2023, doi: https://doi.org/10.1016/j.procs.2023.01.054.
“Challenges in Representation Learning: Facial Expression Recognition Challenge,” Res. Predict. Compet., [Online]. Available: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data
and C.-S. L. J.-C. Kim, M.-H. Kim, H.-E. Suh, M. T. Naseem, “Hybrid Approach for Facial Expression Recognition Using Convolutional Neural Networks and SVM,” Appl. Sci., vol. 12, no. 11, p. 5493, 2022, doi: 10.3390/app12115493.
K. Sarvakar, R. Senkamalavalli, S. Raghavendra, J. Santosh Kumar, R. Manjunath, and S. Jaiswal, “Facial emotion recognition using convolutional neural networks,” Mater. Today Proc., vol. 80, pp. 3560–3564, Jan. 2023, doi: 10.1016/J.MATPR.2021.07.297.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 50SEA
This work is licensed under a Creative Commons Attribution 4.0 International License.
