IoT-Enabled Assistive Glove for Real-Time Sign Language Translation Using Machine Learning

Muhammad Uzair Shahid; Muhammad Mahi Mahessar; Muhammad Salaar; Haris Bin Amir; Muhammad Zabil Mehboob

Authors

Muhammad Uzair Shahid Department of Software Engineering, Military College of Signals, National University of Science and Technology, Rawalpindi, Pakistan
Muhammad Mahi Mahessar Department of Software Engineering, Military College of Signals, National University of Science and Technology, Rawalpindi, Pakistan
Muhammad Salaar Department of Software Engineering, Military College of Signals, National University of Science and Technology, Rawalpindi, Pakistan
Haris Bin Amir Department of Software Engineering, Military College of Signals, National University of Science and Technology, Rawalpindi, Pakistan
Muhammad Zabil Mehboob Department of Software Engineering, Military College of Signals, National University of Science and Technology, Rawalpindi, Pakistan

Keywords:

Sign Language (SL), American Sign Language (ASL), Machine Learning (ML)

Abstract

This paper presents a real-time system for translating gestures from American Sign Language (ASL) using an IoT-enabled smart glove. The glove is equipped with five flex sensors and an MPU-6050 gyroscope to capture finger movements and wrist orientation, processed by an Arduino Nano. Sensor data is transmitted via a Bluetooth module to a mobile application, where a Random Forest machine learning model with 97% accuracy classifies the gestures. The recognized gestures are displayed as text and vocalized through a speaker. Moreover, the app has a feature that allows users to see ASL signs with its corresponding vocabulary, thus enabling accessibility and making language more accessible to learn. It enhances the communication between the deaf and the hearing community since it offers an accurate, portable, and interactive sign recognition application.

References

W. H. Organisation, “Deafness and hearing loss,” World Heal. Organ., 2024, [Online]. Available: https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss

“DISABLED POPULATION BY NATURE OF DISABILITY,” 2021, [Online]. Available: https://www.pbs.gov.pk/sites/default/files/tables/population/DISABLED POPULATION BY NATURE OF DISABILITY.pdf

M. M. bin L. Mohamed Aktham Ahmed, Bilal Bahaa Zaidan, Aws Alaa Zaidan, Mahmood Maher Salih, “A Review on Systems-Based Sensory Gloves for Sign Language Recognition State of the Art between 2007 and 2017,” Sensors, vol. 18, no. 7, p. 2208, 2018, doi: https://doi.org/10.3390/s18072208.

L. L. Ang Ji, Yongzhen Wang, Xin Miao, Tianqi Fan, Bo Ru, “Dataglove for Sign Language Recognition of People with Hearing and Speech Impairment via Wearable Inertial Sensors,” Sensors, vol. 23, no. 15, p. 6693, 2023, doi: https://doi.org/10.3390/s23156693.

S. B. Rakesh Mali, “IOT-BASED SMART AGRICULTURE USING CLOUD BASED STORAGE,” Int. Res. J. Mod. Eng. Technol. Sci., vol. 4, no. 11, 2022, [Online]. Available: https://www.irjmets.com/uploadedfiles/paper//issue_11_november_2022/31282/final/fin_irjmets1668688700.pdf

G. F. F. Maria Papatsimouli, Panos Sarigiannidis, “A Survey of Advancements in Real-Time Sign Language Translators: Integration with IoT Technology,” Technologies, vol. 11, no. 4, p. 83, 2023, doi: https://doi.org/10.3390/technologies11040083.

R. Shah, “American Sign Language Recognition using Hand Gloves,” Int. J. Res. Appl. Sci. Eng. Technol., vol. 7, no. 5, pp. 876–882, May 2019, doi: 10.22214/IJRASET.2019.5147.

D. M. R. Matiwade, S. V., & Dixit, “Electronic Device for Deaf and Dumb to Interpret Sign Language of Communication,” Int. J. Innov. Res. Comput. Commun. Eng. Int. J. Innov. Res. Comput. Commun. Eng., vol. 4, no. 11, 2016, [Online]. Available: https://ijircce.com/admin/main/storage/app/pdf/EzQt5nA5VIisdgxcpcxXDUCngbKdsI00Y4mzR0WL.pdf

N. P. Urav Dalal, Aasmi Thadhani, Mahek Upadhye, Shreya Shah, Meera Narvekar and Abstract, “Deep Learning-Enabled Smart Glove for Real-Time Sign Language Translation,” J. Electr. Syst., vol. 20, 2024, [Online]. Available: https://journal.esrgroups.org/jes/article/view/6153

M. M. A. J. and A. A. M. Radzi Ambar, Chan Kar Fai, Mohd Helmy Abd Wahab, “Development of a Wearable Device for Sign Language Recognition,” J. Phys. Conf. Ser., 2018, doi: 10.1088/1742-6596/1019/1/012017.

L. A. Muhammad Saad Amin, Syed Tahir Hussain Rizvi, Alessandro Mazzei, “Assistive Data Glove for Isolated Static Postures Recognition in American Sign Language Using Neural Network,” Electronics, vol. 12, no. 8, p. 1904, 2023, doi: https://doi.org/10.3390/electronics12081904.

M. de F. and D. R. J. DelPreto, J. Hughes, M. D’Aria, “A Wearable Smart Glove and Its Application of Pose and Gesture Detection to Sign Language Classification,” IEEE Robot. Autom. Lett., vol. 7, no. 4, pp. 10589–10596, 2022, doi: 10.1109/LRA.2022.3191232.

M. M. H. Muhammad Saad Amin, Syed Tahir Hussain Rizvi, “A Comparative Review on Applications of Different Sensors for Sign Language Recognition,” J. Imaging, vol. 8, no. 4, p. 98, 2022, doi: https://doi.org/10.3390/jimaging8040098.