Complex Human Activities Recognition Using Smartphone Sensors: A Deep Learning Approach

Mubashar Saeed; Dr. Arshad Awan; Saira Hameed

Authors

Mubashar Saeed Allama Iqbal Open University
Dr. Arshad Awan Allama Iqbal Open University
Saira Hameed Allama Iqbal Open University

Keywords:

Human Activity Recognition, Complex Human Activity Recognition, Long Short-Term Memory, Mobile Sensors, Machine Learning.

Abstract

Human Activity Recognition (HAR) plays a critical role in understanding human behavior, with mobile phone sensors offering a promising approach for practical applications. This research uniquely addresses the challenge of Complex Human Activity Recognition (CHAR) using Long Short-Term Memory (LSTM) networks, advancing beyond basic activity recognition. LSTM was applied to three publicly available datasets—PAMAP2, Complex Human Activities, and WISDM—using accelerometer, gyroscope, and magnetometer sensor data. The research evaluated the effectiveness of both single-sensor (accelerometer) and multi-sensor combinations for recognizing complex activities. The study achieved 94-98% accuracy across datasets, showing that a single accelerometer sensor provides reasonable accuracy, while adding more sensors, like gyroscope and magnetometer, further boosts performance at a resource cost. The LSTM-based approach consistently outperformed traditional methods, including CNNs, in complex activity recognition, demonstrating its robustness in simplifying sensor requirements without compromising accuracy. LSTM networks offer an efficient and accurate solution for complex human activity recognition, balancing performance and resource optimization.

References

E. Kim, S. Helal, D. Cook, Human activity recognition and pattern discovery, IEEE Pervasive Comput. 9 (1) (2009) 48–53.

J.K. Aggarwal, L. Xia, Human activity recognition from 3d data: A review, Pattern Recognit. Lett. 48 (2014) 70–80.

E. Soleimani, E. Nazerfard, Cross-subject transfer learning in human activity recognition systems using generative adversarial networks, Neurocomputing 426 (2021) 26–34.

C. Torres-Huitzil, A. Alvarez-Landero, Accelerometer-based human activity recognition in smartphones for healthcare services, in: Mobile Health, Springer, 2015, pp. 147–169.

A. Zahin, R.Q. Hu, et al., Sensor-based human activity recognition for smart healthcare: A semi-supervised machine learning, in: International Conference on Artificial Intelligence for Communications and Networks, Springer, 2019, pp. 450–472.

J. Manjarres, P. Narvaez, K. Gasser, W. Percybrooks, M. Pardo, Physical workload tracking using human activity recognition with wearable devices, Sensors 20 (1) (2020) 39.

N. Zhu, T. Diethe, M. Camplani, L. Tao, A. Burrows, N. Twomey, D. Kaleshi, M. Mirmehdi, P. Flach, I. Craddock, Bridging e-health and the internet of things: The sphere project, IEEE Intell. Syst. 30 (4) (2015) 39–46.

Y. Du, Y. Lim, Y. Tan, A novel human activity recognition and prediction in smart home based on interaction, Sensors 19 (20) (2019) 4474.

O.D. Lara, M.A. Labrador, A survey on human activity recognition using wearable sensors, IEEE Commun. Surv. Tutor. 15 (3) (2012) 1192–1209.

F. Demrozi, G. Pravadelli, A. Bihorac, P. Rashidi, Human activity recognition using inertial, physiological and environmental sensors: a comprehensive survey, 2020, arXiv preprint arXiv:2004.08821.

Shoaib, M. Bosch, S. Incel, O.D. Scholten, P.J.M. Complex Human Activity Recognition Using Smartphone and Wrist Worn Motion Sensors. Sensors 2016, 16, 426. https://doi.org/10.3390/s16040426

M.M. Hassan, M.Z. Uddin, A. Mohamed, A. Almogren, A robust human activity recognition system using smartphone sensors and deep learning, Future Gener. Comput. Syst. 81 (2018) 307–313.

F. Attal, S. Mohammed, M. Dedabrishvili, F. Chamroukhi, L. Oukhellou, Y. Amirat, Physical human activity recognition using wearable sensors, Sensors 15 (12) (2015) 31314–31338.

C. Xu, D. Chai, J. He, X. Zhang, S. Duan, Innohar: a deep neural network for complex human activity recognition, Ieee Access 7 (2019) 9893–9902.

N. Lane, Y. Xu, H. lu, S. Hu, T. Choudhury, A. Campbell, F. Zhao, Enabling large-scale human activity inference on smartphones using community similarity networks (CSN), in: UbiComp’11 - Proceedings of the 2011 ACM Conference on Ubiquitous Computing, 2011, pp. 355–364.

G. Weiss, J. Lockhart, The impact of personalization on smartphone-based activity recognition, in: AAAI Publications, Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012.

A. Ferrari, D. Micucci, M. Mobilio, P. Napoletano, On the personalization of classification models for human activity recognition, IEEE Access PP (2020)

R. Solis Castilla, A. Akbari, R. Jafari, B.J. Mortazavi, Using intelligent personal annotations to improve human activity recognition for movements in natural environments, IEEE J. Biomed. Health Inf. (2020) 1.

D. Garcia-Gonzalez, D. Rivero, E. Fernandez-Blanco, M.R. Luaces, A public domain dataset for real-life human activity recognition using smartphone sensors, Sensors 20 (8) (2020) 2200.

D. Anguita, A. Ghio, L. Oneto, X. Parra, J.L. Reyes-Ortiz, A public domain dataset for human activity recognition using smartphones, in: Esann, 2013.

J.R. Kwapisz, G.M. Weiss, S.A. Moore, Activity recognition using cell phone accelerometers, ACM SigKDD Explor. Newsl. 12 (2) (2011) 74–82.

A. Ignatov, Real-time human activity recognition from accelerometer data using convolutional neural networks, Appl. Soft Comput. 62 (2018) 915–922.

N. Sikder, M.S. Chowdhury, A.S. Arif, A.-A. Nahid, Human activity recognition using multichannel convolutional neural network, in: 2019 5th Int. Conf. Adv. Electr. Eng, 2019.

S. Seto, W. Zhang, Y. Zhou, Multivariate time series classification using dynamic time warping template selection for human activity recognition, in: 2015 IEEE Symposium Series on Computational Intelligence, IEEE, 2015, pp. 1399–1406.

W. Sousa, E. Souto, J. Rodrigres, P. Sadarc, R. Jalali, K. El-Khatib, A comparative analysis of the impact of features on human activity recognition with smartphone sensors, in: Proceedings of the 23rd Brazillian Symposium on Multimedia and the Web, ACM, 2017, pp. 397–404.

J. Figueiredo, G. Gordalina, P. Correia, G. Pires, L. Oliveira, R. Martinho, R. Rijo, P. Assuncao, A. Seco, R. Fonseca-Pinto, Recognition of human activity based on sparse data collected from smartphone sensors, in: 2019 IEEE 6th Portuguese Meeting on Bioengineering, ENBENG, IEEE, 2019, pp. 1–4.

R.-A. Voicu, C. Dobre, L. Bajenaru, R.-I. Ciobanu, Human physical activity recognition using smartphone sensors, Sensors 19 (3) (2019) 458.

Z. Chen, Q. Zhu, Y.C. Soh, L. Zhang, Robust human activity recognition using smartphone sensors via CT-PCA and online SVM, IEEE Trans. Ind. Inform. 13 (6) (2017) 3070–3080.

C.A. Ronao, S.-B. Cho, Human activity recognition with smartphone sensors using deep learning neural networks, Expert Syst. Appl. 59 (2016) 235–244.

F. Hernández, L.F. Suárez, J. Villamizar, M. Altuve, Human activity recognition on smartphones using a bidirectional LSTM network, in: 2019 XXII Symposium on Image, Signal Processing and Artificial Vision, STSIVA, IEEE, 2019, pp. 1–5.

M. Badshah, Sensor-based human activity recognition using smartphones, 2019.

S. Wan, L. Qi, X. Xu, C. Tong, Z. Gu, Deep learning models for real-time human activity recognition with smartphones, Mob. Netw. Appl. (2019) 1–13.

W. Qi, H. Su, C. Yang, G. Ferrigno, E. De Momi, A. Aliverti, A fast and robust deep convolutional neural networks for complex human activity recognition using smartphone, Sensors 19 (17) (2019) 3731.

Q. Teng, K. Wang, L. Zhang, J. He, the layer-wise training convolutional neural networks using local loss for sensor-based human activity recognition, IEEE Sens. J. 20 (13) (2020) 7265–7274.

Y.E. Ustev, O. Durmaz Incel, C. Ersoy, User, device and orientation independent human activity recognition on mobile phones: Challenges and A proposal, in: Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, ACM, 2013, pp. 1427–1436.

V. Janko, N. Rešçiç, M. Mlakar, V. Drobnič, M. Gams, G. Slapničar, M. Gjoreski, J. Bizjak, M. Marinko, M. Luštrek, A new frontier for activity recognition: The sussex-huawei locomotion challenge, in: Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, 2018, pp. 1511–1520.

S. Rosati, G. Balestra, M. Knaflitz, Comparison of different sets of features for human activity recognition by wearable sensors, Sensors 18 (12) (2018) 4189.

D. Nielsen, Tree boosting with xgboost-why does xgboost win" every" machine learning competition? (Master’s thesis), NTNU, 2016.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., Scikit-learn: Machine learning in python, J. Mach. Learn. Res. 12 (2011) 2825–2830.

T. Chen, C. Guestrin, Xgboost: A scalable tree boosting system, in: Proceedings of the 22nd Acm Sigkdd International Conference on Knowledge Discovery and Data Mining, 2016, pp. 785–794.

C. Cortes, V. Vapnik, Support-vector networks, Mach. Learn. 20 (3) (1995) 273–297.

R. Rifkin, A. Klautau, In defense of one-vs-all classification, J. Mach. Learn. Res. 5 (2004) 101–141.

J.R. Quinlan, Induction of decision trees, Mach. Learn. 1 (1) (1986) 81–106.

J.R. Quinlan, C4. 5: Programs for Machine Learning, Elsevier, 2014.

L. Breiman, J. Friedman, C.J. Stone, R.A. Olshen, Classification and Regression Trees, CRC Press, 1984.

H. Taud, J. Mas, Multilayer perceptron (MLP), in: Geomatic Approaches for Modeling Land Change Scenarios, Springer, 2018, pp. 451–455.

D.P. Kingma, J. Ba, Adam: A method for stochastic optimization, 2014, arXiv preprint arXiv:1412.6980.

I. Rish, et al., An empirical study of the naive Bayes classifier, in: IJCAI 2001 Workshop on Empirical Methods in Artificial Intelligence, Vol. 3, (22) 2001, pp. 41–46.

K.P. Murphy, et al., Naive Bayes Classifiers, 18, (60) University of British Columbia, 2006.

L.E. Peterson, K-nearest neighbor, Scholarpedia 4 (2) (2009) 1883.

P. Cunningham, S.J. Delany, K-nearest neighbour classifiers–, 2020, arXivpreprint arXiv:2004.04523.

L. Breiman, Random forests, Mach. Learn. 45 (1) (2001) 5–32.

S. Athey, J. Tibshirani, S. Wager, et al., Generalized random forests, Ann. Statist. 47 (2) (2019) 1148–1178.

M. Hossin, M. Sulaiman, A review on evaluation metrics for data classification evaluations, Int. J. Data Min. Knowl. Manag. Process 5 (2) (2015) 1.

M. Grandini, E. Bagli, G. Visani, Metrics for multi-class classification: an overview, 2020, arXiv preprint arXiv:2008.05756.

M. Bekkar, H.K. Djemaa, T.A. Alitouche, Evaluation measures for models’ assessment over imbalanced data sets, J. Inf. Eng. Appl. 3 (10) (2013).

R. Kohavi, et al., A study of cross-validation and bootstrap for accuracy estimation and model selection, in: Ijcai, Vol. 14, (2) Montreal, Canada, 1995, pp. 1137–1145.

P. Liashchynskyi, P. Liashchynskyi, Grid search, random search, genetic algorithm: A big comparison for nas, 2019, arXiv preprint arXiv:1912. 06059.

D. Garcia-Gonzalez, D. Rivero, E. Fernandez-Blanco, M. R. Luaces, A public domain dataset for real-life human activity recognition using smartphone sensors, 2020, Mendeley Data, V2, Available online: https://data.mendeley. com/datasets/3xm88g6m6d/2.