A Dynamic Architecture to Control Multi-Rotors Using Hand Gestures
Keywords:
UAV, Hand Gestures, Human Drone Interaction, Deep Learning, Tensor Flow., CNNs, Control ArchitectureAbstract
Traditional methods for controlling multi-rotors typically involve joysticks, radio controllers, and mobile applications. However, these methods pose significant challenges, particularly for novice users like farmers, due to the extensive training and understanding required to effectively operate a copter. This paper introduces a highly adaptable architecture designed to offer an end-to-end solution for controlling a copter using hand gestures. The proposed system leverages a depth sensor and Convolutional Neural Network (CNN) to recognize hand gestures, utilizing a custom dataset collected from both indoor and outdoor environments. Through a series of simulations with novice users, the system has demonstrated successful operation in real-world scenarios. Currently, the architecture can accurately recognize six distinct gestures with an average accuracy of 90.5% across three different test environments with varying lighting conditions. Key features of this proposed solution include its adaptability, reliable performance, especially in low-light conditions, and its user-friendly design, making it particularly well-suited for farmers and other inexperienced users.
References
X. Wang, A. Chowdhery, and M. Chiang, “Networked Drone Cameras for Sports Streaming,” Proc. - Int. Conf. Distrib. Comput. Syst., pp. 308–318, Jul. 2017, doi. 10.1109/ICDCS.2017.200.
A. Sehrawat, T. A. Choudhury, and G. Raj, “Surveillance drone for disaster management and military security,” Proceeding - IEEE Int. Conf. Comput. Commun. Autom. ICCCA 2017, vol. 2017-January, pp. 470–475, Dec. 2017, doi. 10.1109/CCAA.2017.8229846.
V. Gatteschi et al., “New Frontiers of Delivery Services Using Drones. A Prototype System Exploiting a Quadcopter for Autonomous Drug Shipments,” Proc. - Int. Comput. Softw. Appl. Conf., vol. 2, pp. 920–927, Sep. 2015, doi. 10.1109/COMPSAC.2015.52.
R. Agrawal and N. Gupta, “Real Time Hand Gesture Recognition for Human Computer Interaction,” Proc. - 6th Int. Adv. Comput. Conf. IACC 2016, pp. 470–475, Aug. 2016, doi. 10.1109/IACC.2016.93.
Y. S. Tan, K. M. Lim, and C. P. Lee, “Hand gesture recognition via enhanced densely connected convolutional neural network,” Expert Syst. Appl., vol. 175, p. 114797, Aug. 2021, doi. 10.1016/J.ESWA.2021.114797.
S. Bhushan, M. Alshehri, I. Keshta, A. K. Chakraverti, J. Rajpurohit, and A. Abugabah, “An Experimental Analysis of Various Machine Learning Algorithms for Hand Gesture Recognition,” Electron. 2022, Vol. 11, Page 968, vol. 11, no. 6, p. 968, Mar. 2022, doi. 10.3390/ELECTRONICS11060968.
T. Begum, I. Haque, and V. Keselj, “Deep Learning Models for Gesture-controlled Drone Operation,” 16th Int. Conf. Netw. Serv. Manag. CNSM 2020, 2nd Int. Work. Anal. Serv. Appl. Manag. AnServApp 2020 1st Int. Work. Futur. Evol. Internet Protoc. IPFutu, 2020, [Online]. Available. https.//ieeexplore.ieee.org/document/9269056
Y. Liu, M. Dong, S. Bi, D. Gao, Y. Jing, and L. Li, “Gesture recognition based on Kinect,” 6th Annu. IEEE Int. Conf. Cyber Technol. Autom. Control Intell. Syst. IEEE-CYBER 2016, pp. 343–347, Sep. 2016, doi. 10.1109/CYBER.2016.7574847.
A. Sarkar, K. A. Patel, R. K. G. Ram, and G. K. Capoor, “Gesture control of drone using a motion controller,” 2016 Int. Conf. Ind. Informatics Comput. Syst. CIICS 2016, Apr. 2016, doi. 10.1109/ICCSII.2016.7462401.
K. Natarajan, T. H. D. Nguyen, and M. Mete, “Hand gesture controlled drones. An open source library,” Proc. - 2018 1st Int. Conf. Data Intell. Secur. ICDIS 2018, pp. 168–175, May 2018, doi. 10.1109/ICDIS.2018.00035.
T. B. Dinh, V. B. Dang, D. A. Duong, T. T. Nguyen, and D. D. Le, “Hand gesture classification using boosted cascade of classifiers,” Proc. 4th IEEE Int. Conf. Res. Innov. Vis. Futur. RIVF’06, pp. 139–144, 2006, doi. 10.1109/RIVF.2006.1696430.
S. A. Mehdi and Y. N. Khan, “Sign language recognition using sensor gloves,” ICONIP 2002 - Proc. 9th Int. Conf. Neural Inf. Process. Comput. Intell. E-Age, vol. 5, pp. 2204–2206, 2002, doi. 10.1109/ICONIP.2002.1201884.
A. Shimada, T. Yamashita, and R. I. Taniguchi, “Hand gesture based TV control system - Towards both user - & Machine-friendly gesture applications,” FCV 2013 - Proc. 19th Korea-Japan Jt. Work. Front. Comput. Vis., pp. 121–126, 2013, doi. 10.1109/FCV.2013.6485473.
X. Liu and K. Fujimura, “Hand gesture recognition using depth data,” Proc. - Sixth IEEE Int. Conf. Autom. Face Gesture Recognit., pp. 529–534, 2004, doi. 10.1109/AFGR.2004.1301587.
S. Supimros and S. Wongthanavasu, “Speech recognition - Based control system for Drone,” Proc. 2014 3rd ICT Int. Sr. Proj. Conf. ICT-ISPC 2014, pp. 107–110, Oct. 2014, doi. 10.1109/ICT-ISPC.2014.6923229.
X. H. Wu, M. C. Su, and P. C. Wang, “A hand-gesture-based control interface for a car-robot,” IEEE/RSJ 2010 Int. Conf. Intell. Robot. Syst. IROS 2010 - Conf. Proc., pp. 4644–4648, 2010, doi. 10.1109/IROS.2010.5650294.
K. Root and R. Urniezius, “Research and development of a gesture-controlled robot manipulator system,” IEEE Int. Conf. Multisens. Fusion Integr. Intell. Syst., vol. 0, pp. 353–358, Jul. 2016, doi. 10.1109/MFI.2016.7849513.
Y. Ma et al., “Hand gesture recognition with convolutional neural networks for the multimodal UAV control,” 2017 Work. Res. Educ. Dev. Unmanned Aer. Syst. RED-UAS 2017, pp. 198–203, Nov. 2017, doi. 10.1109/RED-UAS.2017.8101666.
O. M. Sincan and H. Y. Keles, “AUTSL. A large scale multi-modal Turkish sign language dataset and baseline methods,” IEEE Access, vol. 8, pp. 181340–181355, 2020, doi. 10.1109/ACCESS.2020.3028072.
M. Chandarana, E. L. Meszaros, A. Tmjillo, and B. Danette Allen, “Analysis of a gesture-based interface for UAV flight path generation,” 2017 Int. Conf. Unmanned Aircr. Syst. ICUAS 2017, pp. 36–45, Jul. 2017, doi. 10.1109/ICUAS.2017.7991390.
M. Zobl, M. Geiger, B. Schuller, M. Lang, and G. Rigoll, “A real-time system for hand gesture controlled operation of in-car devices,” pp. III–541, May 2004, doi. 10.1109/ICME.2003.1221368.
F. Parada-Loira, E. Gonzalez-Agulla, and J. L. Alba-Castro, “Hand gestures to control infotainment equipment in cars,” IEEE Intell. Veh. Symp. Proc., pp. 1–6, 2014, doi. 10.1109/IVS.2014.6856614.
P. J. Gonzalo and A. Holgado-Terriza Juan, “Control of home devices based on hand gestures,” 5th IEEE Int. Conf. Consum. Electron. - Berlin, ICCE-Berlin 2015, pp. 510–514, Jan. 2016, doi. 10.1109/ICCE-BERLIN.2015.7391325.
A. Budiyanto, M. I. Ramadhan, I. Burhanudin, H. H. Triharminto, and B. Santoso, “Navigation control of Drone using Hand Gesture based on Complementary Filter Algorithm,” J. Phys. Conf. Ser., vol. 1912, no. 1, p. 012034, May 2021, doi. 10.1088/1742-6596/1912/1/012034.
R. Mirsu, G. Simion, C. D. Caleanu, and I. M. Pop-Calimanu, “A PointNet-Based Solution for 3D Hand Gesture Recognition,” Sensors 2020, Vol. 20, Page 3226, vol. 20, no. 11, p. 3226, Jun. 2020, doi. 10.3390/S20113226.
B. Latif, N. Buckley, and E. L. Secco, “Hand Gesture and Human-Drone Interaction,” Lect. Notes Networks Syst., vol. 544 LNNS, pp. 299–308, 2023, doi. 10.1007/978-3-031-16075-2_20.
I. Kagirov, D. Ivanko, D. Ryumin, A. Axyonov, and A. Karpov, “TheRuSLan. Database of Russian Sign Language.” pp. 6079–6085, 2020. Accessed. Sep. 12, 2024. [Online]. Available. https.//aclanthology.org/2020.lrec-1.746
J. P. Sahoo, A. J. Prakash, P. Pławiak, and S. Samantray, “Real-Time Hand Gesture Recognition Using Fine-Tuned Convolutional Neural Network,” Sensors 2022, Vol. 22, Page 706, vol. 22, no. 3, p. 706, Jan. 2022, doi. 10.3390/S22030706.
M. Oudah, A. Al-Naji, and J. Chahl, “Elderly Care Based on Hand Gestures Using Kinect Sensor,” Comput. 2021, Vol. 10, Page 5, vol. 10, no. 1, p. 5, Dec. 2020, doi. 10.3390/COMPUTERS10010005.
D. Sarma and M. K. Bhuyan, “Methods, Databases and Recent Advancement of Vision-Based Hand Gesture Recognition for HCI Systems. A Review,” SN Comput. Sci., vol. 2, no. 6, pp. 1–40, Nov. 2021, doi. 10.1007/S42979-021-00827-X/FIGURES/18.
R. A. El-Laithy, J. Huang, and M. Yeh, “Study on the use of Microsoft Kinect for robotics applications,” Rec. - IEEE PLANS, Position Locat. Navig. Symp., pp. 1280–1288, 2012, doi. 10.1109/PLANS.2012.6236985.
G. Moon, S. I. Yu, H. Wen, T. Shiratori, and K. M. Lee, “InterHand2.6M. A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 12365 LNCS, pp. 548–564, 2020, doi. 10.1007/978-3-030-58565-5_33.
“LSA64. A Dataset for Argentinian Sign Language · Facundo Quiroga.” Accessed. Sep. 12, 2024. [Online]. Available. https.//facundoq.github.io/datasets/lsa64/
C. P. Papageorgiou, M. Oren, and T. Poggio, “General framework for object detection,” Proc. IEEE Int. Conf. Comput. Vis., pp. 555–562, 1998, doi. 10.1109/ICCV.1998.710772.
C. Quan and J. Liang, “A Simple and Effective Method for Hand Gesture Recognition,” pp. 302–305, Jun. 2017, doi. 10.1109/ICNISC.2016.072.
M. Panwar, “Hand gesture recognition based on shape parameters,” 2012 Int. Conf. Comput. Commun. Appl. ICCCA 2012, 2012, doi. 10.1109/ICCCA.2012.6179213.
N. H. Dardas and E. M. Petriu, “Hand gesture detection and recognition using principal component analysis,” IEEE Int. Conf. Comput. Intell. Meas. Syst. Appl. Proc., pp. 11–16, 2011, doi. 10.1109/CIMSA.2011.6059935.
A. Parvat, J. Chavan, S. Kadam, S. Dev, and V. Pathak, “A survey of deep-learning frameworks,” Proc. Int. Conf. Inven. Syst. Control. ICISC 2017, Oct. 2017, doi. 10.1109/ICISC.2017.8068684.
K. K. Biswas and S. K. Basu, “Gesture recognition using Microsoft Kinect,” ICARA 2011 - Proc. 5th Int. Conf. Autom. Robot. Appl., pp. 100–103, 2011, doi. 10.1109/ICARA.2011.6144864.
I. Coonjah, P. C. Catherine, and K. M. S. Soyjaudah, “Experimental performance comparison between TCP vs UDP tunnel using OpenVPN,” 2015 Int. Conf. Comput. Commun. Secur. ICCCS 2015, Jan. 2016, doi. 10.1109/CCCS.2015.7374133.
T. Le et al., “Reliable user datagram protocol for airborne network,” Proc. - IEEE Mil. Commun. Conf. MILCOM, 2009, doi. 10.1109/MILCOM.2009.5380007.
M. Petrovic and M. Aboelaze, “Performance of TCP/UDP under ad hoc IEEE802.11,” 10th Int. Conf. Telecommun. ICT 2003, vol. 1, pp. 700–708, 2003, doi. 10.1109/ICTEL.2003.1191496.
S. R. Haque, R. Kormokar, and A. U. Zaman, “Drone ground control station with enhanced safety features,” 2017 2nd Int. Conf. Converg. Technol. I2CT 2017, vol. 2017-January, pp. 1207–1210, Dec. 2017, doi. 10.1109/I2CT.2017.8226318.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 50SEA
This work is licensed under a Creative Commons Attribution 4.0 International License.
