Interpretation of Expressions through Hand Signs Using Deep Learning Techniques

Sameena Javaid; Safdar Rizvi; Muhammad Talha Ubaid; Abdou Darboe; Shakir Mahmood Mayo

Authors

Sameena Javaid Department of Computer Sciences, School of Engineering and Applied Sciences, Bahria University Karachi Campus, Karachi 75290, Pakistan.
Safdar Rizvi Department of Computer Sciences, School of Engineering and Applied Sciences, Bahria University Karachi Campus, Karachi 75290, Pakistan.
Muhammad Talha Ubaid National Center of Artificial Intelligence, KICS, University of Engineering and Technology, Lahore 39161, Pakistan.
Abdou Darboe University of The Gambia, Serrekunda, The Gambia
Shakir Mahmood Mayo University of Engineering and Technology Lahore

Keywords:

Pakistan Sign Language, Hand Gestures, Convolutional Neural Network, VGG16, Transfer Learning

Abstract

It is a challenging task to interpret sign language automatically, as it comprises high-level vision features to accurately understand and interpret the meaning of the signer or vice versa. In the current study, we automatically distinguish hand signs and classify seven basic gestures representing symbolic emotions or expressions like happy, sad, neutral, disgust, scared, anger, and surprise. Convolutional Neural Network is a famous method for classifications using vision-based deep learning; here in the current study, proposed transfer learning using a well-known architecture of VGG16 to speed up the convergence and improve accuracy by using pre-trained weights. We obtained a high accuracy of 99.98% of the proposed architecture with a minimal and low-quality data set of 455 images collected by 65 individuals for seven hand gesture classes. Further, compared the performance of VGG16 architecture with two different optimizers, SGD, and Adam, along with some more architectures of AlexNet, LeNet05, and ResNet50.

References

R. Vaccaro, D. Zaccaria, M. Colombo, S. Abbondanza, and A. Guaita, “Adverse effect of self-reported hearing disability in elderly Italians: Results from the InveCe.Ab study,” Maturitas, vol. 121, pp. 35–40, Mar. 2019, doi: 10.1016/J.MATURITAS.2018.12.009.

R. Rastgoo, K. Kiani, and S. Escalera, “Real-time isolated hand sign language recognition using deep networks and SVD,” J. Ambient Intell. Humaniz. Comput. 2021 131, vol. 13, no. 1, pp. 591–611, Feb. 2021, doi: 10.1007/S12652-021-02920-8.

R. Rastgoo, K. Kiani, and S. Escalera, “Hand sign language recognition using multi-view hand skeleton,” Expert Syst. Appl., vol. 150, p. 113336, Jul. 2020, doi: 10.1016/J.ESWA.2020.113336.

R. Rastgoo, K. Kiani, and S. Escalera, “Sign Language Recognition: A Deep Survey,” Expert Syst. Appl., vol. 164, p. 113794, Feb. 2021, doi: 10.1016/J.ESWA.2020.113794.

H. Zahid, M. Rashid, S. Hussain, F. Azim, S. A. Syed, and A. Saad, “Recognition of Urdu sign language: a systematic review of the machine learning classification,” PeerJ. Comput. Sci., vol. 8, 2022, doi: 10.7717/PEERJ-CS.883.

E. De Stefani and D. De Marco, “Language, gesture, and emotional communication: An embodied view of social interaction,” Front. Psychol., vol. 10, no. SEP, pp. 1–8, 2019, doi: 10.3389/fpsyg.2019.02063.

T. R. Gadekallu et al., “Hand gesture classification using a novel CNN-crow search algorithm,” Complex Intell. Syst., vol. 7, no. 4, pp. 1855–1868, 2021, doi: 10.1007/s40747-021-00324-x.

S. Javaid, S. Rizvi, M. T. Ubaid, and A. Tariq, “VVC/H.266 Intra Mode QTMT Based CU Partition Using CNN,” IEEE Access, vol. 10, pp. 37246–37256, 2022, doi: 10.1109/ACCESS.2022.3164421.

C. M. Sharma, K. Tomar, R. K. Mishra, and V. M. Chariar, “Indian Sign Language Recognition Using Fine-tuned Deep Transfer Learning Model,” SSRN Electron. J., Sep. 2021, doi: 10.2139/SSRN.3932929.

S. J. Goyal, A. K. Upadhyay, and R. S. Jadon, “Combined Approach to Classify Human Emotions Based on the Hand Gesture,” pp. 301–309, 2020, doi: 10.1007/978-981-15-0633-8_29.

Y. Zou and L. Cheng, “A Transfer Learning Model for Gesture Recognition Based on the Deep Features Extracted by CNN,” IEEE Trans. Artif. Intell., vol. 2, no. 5, pp. 447–458, Jul. 2021, doi: 10.1109/TAI.2021.3098253.

A. Ranjan, C. Kumar, R. K. Gupta, and R. Misra, “Transfer Learning Based Approach for Pneumonia Detection Using Customized VGG16 Deep Learning Model,” Lect. Notes Networks Syst., vol. 340 LNNS, pp. 17–28, 2022, doi: 10.1007/978-3-030-94507-7_2/COVER/.

M. Shaha and M. Pawar, “Transfer Learning for Image Classification,” Proc. 2nd Int. Conf. Electron. Commun. Aerosp. Technol. ICECA 2018, pp. 656–660, Sep. 2018, doi: 10.1109/ICECA.2018.8474802.

F. Noroozi, C. A. Corneanu, D. Kaminska, T. Sapinski, S. Escalera, and G. Anbarjafari, “Survey on Emotional Body Gesture Recognition,” IEEE Trans. Affect. Comput., vol. 12, no. 2, pp. 505–523, Jan. 2018, doi: 10.48550/arxiv.1801.07481.

M. A. Arjun, S. Sreehari, and R. Nandakumar, “The Interplay of Hand Gestures and Facial Expressions in Conveying Emotions A CNN-BASED APPROACH,” Proc. 4th Int. Conf. Comput. Methodol. Commun. ICCMC 2020, pp. 833–837, Mar. 2020, doi: 10.1109/ICCMC48092.2020.ICCMC-000154.

R. Bhadra and S. Kar, “Sign language detection from hand gesture images using deep multi-layered convolution neural network,” 2021 IEEE 2nd Int. Conf. Control. Meas. Instrumentation, C. 2021 - Proc., pp. 196–200, Jan. 2021, doi: 10.1109/CMI50323.2021.9362897.

P. Parvathy, K. Subramaniam, G. K. D. Prasanna Venkatesan, P. Karthikaikumar, J. Varghese, and T. Jayasankar, “Development of hand gesture recognition system using machine learning,” J. Ambient Intell. Humaniz. Comput. 2020 126, vol. 12, no. 6, pp. 6793–6800, Jul. 2020, doi: 10.1007/S12652-020-02314-2.

S. Bhushan, M. Alshehri, I. Keshta, A. K. Chakraverti, J. Rajpurohit, and A. Abugabah, “An Experimental Analysis of Various Machine Learning Algorithms for Hand Gesture Recognition,” Electron. 2022, Vol. 11, Page 968, vol. 11, no. 6, p. 968, Mar. 2022, doi: 10.3390/ELECTRONICS11060968.

S. Ahmed et al., “Hand Sign to Bangla Speech: A Deep Learning in Vision Based System for Recognizing Hand Sign Digits and Generating Bangla Speech,” SSRN Electron. J., pp. 1–6, 2019, doi: 10.2139/ssrn.3358187.

R. Rastgoo, K. Kiani, and S. Escalera, “Multi-Modal Deep Hand Sign Language Recognition in Still Images Using Restricted Boltzmann Machine,” Entropy 2018, Vol. 20, Page 809, vol. 20, no. 11, p. 809, Oct. 2018, doi: 10.3390/E20110809.

J. Pardede, B. Sitohang, S. Akbar, and M. L. Khodra, “Implementation of Transfer Learning Using VGG16 on Fruit Ripeness Detection,” Int. J. Intell. Syst. Appl., vol. 13, no. 2, pp. 52–61, 2021, doi: 10.5815/ijisa.2021.02.04.

S. Arora, A. Gupta, R. Jain, and A. Nayyar, “Optimization of the CNN Model for Hand Sign Language Recognition Using Adam Optimization Technique,” Lect. Notes Networks Syst., vol. 179 LNNS, pp. 89–104, 2021, doi: 10.1007/978-981-33-4687-1_10/COVER/.

L. and W. Ministry of Health, “The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches,” 2012, [Online]. Available: http://www.mhlw.go.jp/new-info/kobetu/roudou/gyousei/anzen/dl/101004-3.pdf.

A. Jain, A. Sethi, D. K. Vishwakarma, and A. Jain, “Ensembled Neural Network for Static Hand Gesture Recognition,” 2021 12th Int. Conf. Comput. Commun. Netw. Technol. ICCCNT 2021, 2021, doi: 10.1109/ICCCNT51525.2021.9579633.

S. Mascarenhas and M. Agarwal, “A comparison between VGG16, VGG19 and ResNet50 architecture frameworks for Image Classification,” Proc. IEEE Int. Conf. Disruptive Technol. Multi-Disciplinary Res. Appl. CENTCON 2021, pp. 96–99, 2021, doi: 10.1109/CENTCON52345.2021.9687944.

M. L. George, T. Govindarajan, K. Angamuthu Rajasekaran, and S. R. Bandi, “A robust similarity based deep siamese convolutional neural network for gait recognition across views,” Comput. Intell., vol. 36, no. 3, pp. 1290–1319, Aug. 2020, doi: 10.1111/COIN.12361.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2323, 1998, doi: 10.1109/5.726791.

A. Chavan, J. Ghorpade-Aher, A. Bhat, A. Raj, and S. Mishra, “Interpretation of Hand Spelled Banking Helpdesk Terms for Deaf and Dumb Using Deep Learning,” 2021 IEEE Pune Sect. Int. Conf. PuneCon 2021, 2021, doi: 10.1109/PUNECON52575.2021.9686514.

T. Kattenborn, J. Leitloff, F. Schiefer, and S. Hinz, “Review on Convolutional Neural Networks (CNN) in vegetation remote sensing,” ISPRS J. Photogramm. Remote Sens., vol. 173, pp. 24–49, Mar. 2021, doi: 10.1016/J.ISPRSJPRS.2020.12.010.

S. Ghaffarian, J. Valente, M. Van Der Voort, and B. Tekinerdogan, “Effect of Attention Mechanism in Deep Learning-Based Remote Sensing Image Processing: A Systematic Literature Review,” Remote Sens. 2021, Vol. 13, Page 2965, vol. 13, no. 15, p. 2965, Jul. 2021, doi: 10.3390/RS13152965.

N. Nguyen Tu, S. Sako, and B. Kwolek, “Fingerspelling recognition using synthetic images and deep transfer learning,” no. March, p. 70, 2021, doi: 10.1117/12.2587592.

A. P. Parameshwaran, H. P. Desai, R. Sunderraman, and M. Weeks, “Transfer learning for classifying single hand gestures on comprehensive bharatanatyam mudra dataset,” IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Work., vol. 2019-June, pp. 508–510, Jun. 2019, doi: 10.1109/CVPRW.2019.00074.

K. Suri and R. Gupta, “Transfer learning for sEMG-based hand gesture classification using deep learning in a master- slave architecture,” Proc. 3rd Int. Conf. Contemp. Comput. Informatics, IC3I 2018, pp. 178–183, Oct. 2018, doi: 10.1109/IC3I44769.2018.9007304.

A. P. Parameshwaran, H. P. Desai, M. Weeks, and R. Sunderraman, “Unravelling of Convolutional Neural Networks through Bharatanatyam Mudra Classification with Limited Data,” 2020 10th Annu. Comput. Commun. Work. Conf. CCWC 2020, pp. 342–347, Jan. 2020, doi: 10.1109/CCWC47524.2020.9031185.

M. A. Arshed, H. Ghassan, M. Hussain, M. Hassan, A. Kanwal, and R. Fayyaz, “A Light Weight Deep Learning Model for Real World Plant Identification,” 2022 2nd Int. Conf. Distrib. Comput. High Perform. Comput. DCHPC 2022, pp. 40–45, 2022, doi: 10.1109/DCHPC55044.2022.9731841.

M. T. Ubaid, M. Z. Khan, M. Rumaan, M. A. Arshed, M. U. G. Khan, and A. Darboe, “COVID-19 SOP’s Violations Detection in Terms of Face Mask Using Deep Learning,” 4th Int. Conf. Innov. Comput. ICIC 2021, 2021, doi: 10.1109/ICIC53490.2021.9692999.

B. Liu et al., “Comparison of Machine Learning Classifiers for Breast Cancer Diagnosis Based on Feature Selection,” Proc. - 2018 IEEE Int. Conf. Syst. Man, Cybern. SMC 2018, pp. 4399–4404, Jan. 2019, doi: 10.1109/SMC.2018.00743.

M. T. Ubaid, A. Kiran, M. T. Raja, U. A. Asim, A. Darboe, and M. A. Arshed, “Automatic Helmet Detection using EfficientDet,” 4th Int. Conf. Innov. Comput. ICIC 2021, 2021, doi: 10.1109/ICIC53490.2021.9693093.

O. C. Uner, H. I. Kuru, R. G. Cinbis, O. Tastan, and E. Cicek, “DeepSide: A Deep Learning Approach for Drug Side Effect Prediction,” IEEE/ACM Trans. Comput. Biol. Bioinforma., no. 2016, 2022, doi: 10.1109/TCBB.2022.3141103.

D. Chaudhary, D. Karim, H. Alam, and S. Mumtaz, “Machine Learning with Data Balancing Technique for IoT Attack and Anomalies Detection Original Article,” Int. J. Innov. Sci. Technol., vol. 4, no. 2, pp. 490–498, 2022.