Codebook-Based Feature Engineering for Human Activity Recognition Using Multimodal Sensory Data

Seerat Fatima; Laiba Zahid; Tazeem Haider; Muhammad Hassan Khan; Muhammad Shahid Farid

Authors

Seerat Fatima Department of Software Engineering, University of the Punjab, Lahore, Pakistan
Laiba Zahid Department of Software Engineering, University of the Punjab, Lahore, Pakistan
Tazeem Haider Department of Computer Science, University of the Punjab, Lahore, Pakistan
Muhammad Hassan Khan Department of Computer Science, University of the Punjab, Lahore, Pakistan
Muhammad Shahid Farid Department of Computer Science, University of the Punjab, Lahore, Pakistan

Keywords:

Multimodal Sensory Data, Codebook, Bag of Features, Mini Batch K-Means, Soft Assignment.

Abstract

Recently, Human Activity Recognition (HAR) using sensory data from various devices has become increasingly vital in fields like healthcare, elderly care, and smart home systems. However, many existing HAR systems face challenges such as high computational demands or the need for large datasets. This paper introduces a codebook-based approach designed to overcome these challenges by offering a more efficient method for HAR with reduced computational costs. Initially, the raw time series data is segmented into smaller subsequences, and codebooks are constructed using the Bag of Features (BOF) approach. Each subsequence is then assigned softly based on the center of each cluster (codeword), resulting in a histogram-based feature vector. These encoded feature vectors are subsequently classified using a Support Vector Machine (SVM). The proposed method was evaluated using the OPPORTUNITY dataset, comprising data from 72 sensors, achieving a classification accuracy of 90.7%. In comparison to other advanced techniques, our approach not only demonstrated superior accuracy in recognizing human activities but also significantly reduced computational costs. The use of soft assignments for mapping codewords to subsequences efficiently captured the key patterns within the activity data. The findings validate that the proposed codebook-based method provides substantial improvements in both accuracy and efficiency for HAR systems.

References

Nazish Ashfaq, Muhammad Hassan Khan, and Muhammad Adeel Nisar. Identification of optimal data augmentation techniques for multimodal time-series sensory data: A framework. Information, 15(6):343, 2024.

Muhammad Hassan Khan. Human activity analysis in visual surveillance and healthcare, volume 45. Logos Verlag Berlin GmbH, 2018.

Lukas Köping, Kimiaki Shirahama, and Marcin Grzegorzek. A general framework for sensor-based human activity recognition. Computers in biology and medicine, 95:248–260, 2018.

L Minh Dang, Kyungbok Min, Hanxiang Wang, Md Jalil Piran, Cheol Hee Lee, and Hyeonjoon Moon. Sensor- based and vision-based human activity recognition: A comprehensive survey. Pattern Recognition, 108:107561, 2020.

Rimsha Fatima, Muhammad Hassan Khan, Muhammad Adeel Nisar, Rafał Doniec, Muhammad Shahid Farid, and Marcin Grzegorzek. A systematic evaluation of feature encoding techniques for gait analysis using multimodal sensory data. Sensors, 24(1), 2024.

Hong Liang, Xiao Sun, Yunlei Sun, and Yuan Gao. Text feature extraction based on deep learning: a review. EURASIP journal on wireless communications and networking, 2017:1–12, 2017.

Muhammad Hassan Khan, Muhammad Shahid Farid, and Marcin Grzegorzek. A comprehensive study on codebook-based feature fusion for gait recognition. Information Fusion, 92:216–230, 2023.

Bendong Zhao, Huanzhang Lu, Shangfeng Chen, Junliang Liu, and Dongya Wu. Convolutional neural networks for time series classification. Journal of systems engineering and electronics, 28(1):162–169, 2017.

Lam Tran, Thang Hoang, Thuc Nguyen, Hyunil Kim, and Deokjai Choi. Multi-model long short-term memory network for gait recognition using window-based data segment. IEEE Access, 9:23826–23839, 2021.

Fatima Amjad, Muhammad Hassan Khan, Muhammad Adeel Nisar, Muhammad Shahid Farid, and Marcin Grze- gorzek. A comparative study of feature selection approaches for human activity recognition using multimodal sensory data. Sensors, 21(7):2368, 2021.

Zhenghua Chen, Le Zhang, Zhiguang Cao, and Jing Guo. Distilling the knowledge from handcrafted features for human activity recognition. IEEE Transactions on Industrial Informatics, 14(10):4334–4342, 2018.

Ervin Sejdić, Kristin A Lowry, Jennica Bellanca, Mark S Redfern, and Jennifer S Brach. A comprehensive assessment of gait accelerometry signals in time, frequency and time-frequency domains. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22(3):603–612, 2013.

Mahmudul Hasan Abid and Abdullah-Al Nahid. Two unorthodox aspects in handcrafted-feature extraction for human activity recognition datasets. In 2021 International Conference on Electronics, Communications and Information Technology (ICECIT), pages 1–4. IEEE, 2021.

Allah Bux Sargano, Plamen Angelov, and Zulfiqar Habib. A comprehensive review on handcrafted and learning- based action representation approaches for human activity recognition. Applied Sciences, 7(1), 2017.

Muhammad Attique Khan, Muhammad Sharif, Tallha Akram, Mudassar Raza, Tanzila Saba, and Amjad Rehman. Hand-crafted and deep convolutional neural network features fusion and selection strategy: An application to intelligent human action recognition. Applied Soft Computing, 87:105986, 2020.

Mingtao Dong, Jindong Han, Yuan He, and Xiaojun Jing. Har-net: Fusing deep representation and hand-crafted features for human activity recognition. In Signal and Information Processing, Networking and Computers: Proceedings of the 5th International Conference on Signal and Information Processing, Networking and Computers (ICSINC), pages 32–40. Springer, 2019.

Tazeem Haider, Muhammad Hassan Khan, and Muhammad Shahid Farid. An optimal feature selection method for human activity recognition using multimodal sensory data. Information, 15(10), 2024.

Muhammad Hassan Khan, Muhammad Shahid Farid, and Marcin Grzegorzek. A comprehensive study on codebook-based feature fusion for gait recognition. Information Fusion, 92:216–230, 2023.

Muhammad Hassan Khan, Muhammad Shahid Farid, and Marcin Grzegorzek. Using a generic model for codebook-based gait recognition algorithms. In 2018 International Workshop on Biometrics and Forensics (IWBF), pages 1–7. IEEE, 2018.

Kimiaki Shirahama and Marcin Grzegorzek. On the generality of codebook approach for sensor-based human activity recognition. Electronics, 6(2), 2017.

Muhammad Haseeb Arshad, Muhammad Bilal, and Abdullah Gani. Human activity recognition: Review, taxon- omy and open challenges. Sensors, 22(17), 2022.

Frédéric Li, Kimiaki Shirahama, Muhammad Adeel Nisar, Lukas Köping, and Marcin Grzegorzek. Comparison of feature learning methods for human activity recognition using wearable sensors. Sensors, 18(2), 2018.

Sheeza Batool, Muhammah Hassan Khan, and Muhammah Shahid Farid. An ensemble deep learning model for human activity analysis using wearable sensory data. Applied Soft Computing, page 111599, 2024.

Shibo Zhang, Yaxuan Li, Shen Zhang, Farzad Shahabi, Stephen Xia, Yu Deng, and Nabil Alshurafa. Deep learning in human activity recognition with wearable sensors: A review on advances. Sensors, 22(4), 2022.

Morsheda Akter, Shafew Ansary, Md. Al-Masrur Khan, and Dongwan Kim. Human activity recognition using attention-mechanism-based deep learning feature combination. Sensors, 23(12), 2023.

Md Milon Islam, Sheikh Nooruddin, Fakhri Karray, and Ghulam Muhammad. Human activity recognition using tools of convolutional neural networks: A state of the art review, data sets, challenges, and future prospects. Computers in Biology and Medicine, 149:106060, 2022.

Frédéric Li, Kimiaki Shirahama, Muhammad Adeel Nisar, Lukas Köping, and Marcin Grzegorzek. Comparison of feature learning methods for human activity recognition using wearable sensors. Sensors, 18(2):679, 2018.

Andreas Bulling, Ulf Blanke, and Bernt Schiele. A tutorial on human activity recognition using body-worn inertial sensors. 46(3), 2014.

Chuanlin Zhang, Kai Cao, Limeng Lu, and Tao Deng. A multi-scale feature extraction fusion model for human activity recognition. Scientific Reports, 12(1):20620, 2022.

Ivan Miguel Pires, Faisal Hussain, Nuno M. Garcia, Petre Lameski, and Eftim Zdravevski. Homogeneous data normalization and deep learning: A case study in human activity classification. Future Internet, 12(11), 2020.

Tahmina Zebin, Matthew Sperrin, Niels Peek, and Alexander J Casson. Human activity recognition from inertial sensor time-series using batch normalized deep lstm recurrent networks. In 2018 40th annual international conference of the IEEE engineering in medicine and biology society (EMBC), pages 1–4. IEEE, 2018.

Sujan Ray, Khaldoon Alshouiliy, and Dharma P Agrawal. Dimensionality reduction for human activity recognition using google colab. Information, 12(1):6, 2020.

Mustafa Gokce Baydogan, George Runger, and Eugene Tuv. A bag-of-features framework to classify time series. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(11):2796–2802, 2013.

Sanjay Hanji and Savita Hanji. Towards performance overview of mini batch k-means and k-means: case of four-wheeler market segmentation. In International Conference on Smart Trends in Computing and Communications, pages 801–813. Springer, 2023.

Nan Xu. Applications of support vector machines in electromagnetic problems. PhD thesis, The University of New Mexico, 2011.

Lyudmyla Kirichenko, Tamara Radivilova, and Vitalii Bulakh. Machine learning in classification time series with fractal properties. Data, 4(1):5, 2018.

Himani Bhavsar and Mahesh H Panchal. A review on support vector machine for data classification. International Journal of Advanced Research in Computer Engineering & Technology (IJARCET), 1(10):185–189, 2012.

Isna Alfi Bustoni, I Hidayatulloh, AM Ningtyas, A Purwaningsih, and SN Azhari. Classification methods per- formance on human activity recognition. In Journal of Physics: Conference Series, volume 1456, page 012027. IOP Publishing, 2020.

Lalita Mishra, Shekhar Verma, and Shirshu Varma. Brain tumor classification using feature extraction and non- linear svm hybrid model. In International Conference on Communication, Networks and Computing, pages 3–14. Springer, 2022.

Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian Förster, Gerhard Tröster, Paul Lukow- icz, David Bannach, Gerald Pirkl, Alois Ferscha, et al. Collecting complex activity datasets in highly rich networked sensor environments. In 2010 Seventh international conference on networked sensing systems (INSS), pages 233–240. IEEE, 2010.

Muhammad Adeel Nisar, Kimiaki Shirahama, Muhammad Tausif Irshad, Xinyu Huang, and Marcin Grzegorzek. A hierarchical multitask learning approach for the recognition of activities of daily living using data from wearable sensors. Sensors, 23(19):8234, 2023.

Anna Ferrari, Daniela Micucci, Marco Mobilio, and Paolo Napoletano. Deep learning and model personalization in sensor-based human activity recognition. Journal of Reliable Intelligent Environments, 9(1):27–39, 2023.

Azar Mahmoodzadeh. Human activity recognition based on deep belief network classifier and combination of local and global features. J. Inf. Syst. Telecommun, 9:33, 2021.

Farhad Nazari, Darius Nahavandi, Navid Mohajer, and Abbas Khosravi. Comparison of deep learning techniques on human activity recognition using ankle inertial signals. In 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2251–2256. IEEE, 2022.

Chaolei Han, Lei Zhang, Yin Tang, Wenbo Huang, Fuhong Min, and Jun He. Human activity recognition using wearable sensors by heterogeneous convolutional neural networks. Expert Systems with Applications, 198:116764, 2022.