Reconstruction of occluded Skeletons using Generative Adversarial Networks for Human Activity Recognition

Hassan Nawaz; Nadeem Anjum

Authors

Hassan Nawaz Capital University of Science and Technology
Nadeem Anjum Capital University of Science and Technology

Keywords:

Human Activity Recognition, Occluded Skeleton Reconstruction, Generative Adversarial Networks, BiLSTM, Transformer

Abstract

Human Activity Recognition (HAR) using 3D skeleton motion data plays a vital role in surveillance, healthcare, and human–computer interaction; however, its performance degrades significantly under real-world occlusion conditions. This study proposes a comparative GAN-based framework using CRNN+BiLSTM and Transformer architectures as generator networks to reconstruct occluded 3D human skeletons, demonstrating superior reconstruction and activity recognition performance across multiple occlusion scenarios. The UTKinect-Action3D dataset was used, which is publicly available, and in this dataset, we have RGB images, grayscale images, and 3D skeleton points text data. Skeleton data were manually occluded to simulate eight occlusion scenarios, including left arm, right arm, left leg, right leg, left arm and leg, right arm and leg, both arms, and both legs occlusions. A Generative Adversarial Network (GAN) is employed, where the generator is implemented using (i) CRNN+ BiLSTM and (ii) Transformer models, while the discriminator is based on LSTM. Reconstructed skeletons are subsequently fed into an LSTM-based classifier for activity recognition. The proposed GAN-based reconstruction models significantly enhance skeleton recovery and human activity recognition under occlusion. Compared to the GAN-CRNN (Min) and GAN-CRNN (Max) benchmarks, the GAN-CRNN+ BiLSTM achieves average improvements of 32.7% ± 2.4 and 18.3% ± 1.9, respectively, while the GAN-Transformer attains higher average gains of 35.1% ± 2.1 and 20.5% ± 1.7, demonstrating statistically significant improvements (p < 0.05), particularly in complex occlusion scenarios. These results demonstrate the effectiveness and robustness of the proposed architectures for handling severe skeletal occlusions. The performance gains are attributed to bidirectional temporal modeling in BiLSTM and global spatio-temporal attention in Transformers. The proposed GAN-based reconstruction framework effectively mitigates occlusion effects and significantly enhances human activity recognition accuracy.

References

Z. Sun, Q. Ke, H. Rahmani, M. Bennamoun, G. Wang, and J. Liu, “Human Action Recognition From Various Data Modalities: A Review,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 3, pp. 3200–3225, Mar. 2023, doi: 10.1109/TPAMI.2022.3183112.

F. Angelini, Z. Fu, Y. Long, L. Shao, and S. M. Naqvi, “2D pose-based real-time human action recognition with occlusion-handling,” IEEE Trans. Multimed., vol. 22, no. 6, pp. 1433–1446, Jun. 2020, doi: 10.1109/TMM.2019.2944745.

Ilias Giannakos, Eirini Mathe, “A study on the Effect of Occlusion in Human Activity Recognition,” ACM Int. Conf. Proceeding Ser., 2021, [Online]. Available: https://dl.acm.org/doi/10.1145/3453892.3461337

Rubén San-Segundo, Fernando Fernández-Martínez, “Skeleton Reconstruction Using Generative Adversarial Networks for Human Activity Recognition Under Occlusion,” Sensors, vol. 25, no. 5, p. 1567, 2025, doi: https://doi.org/10.3390/s25051567.

Yu Kong, Yun Fu, “Human Action Recognition and Prediction: A Survey,” arXiv:1806.11230, 2022, [Online]. Available: https://arxiv.org/abs/1806.11230

Bahareh Nikpour, Narges Armanfard, “Spatio-temporal hard attention learning for skeleton-based activity recognition,” Pattern Recognit., vol. 139, p. 109428, 2023, doi: https://doi.org/10.1016/j.patcog.2023.109428.

M. Rahevar and A. Ganatra, “Spatial–Temporal gated graph attention network for skeleton-based action recognition,” Pattern Anal. Appl. 2023 263, vol. 26, no. 3, pp. 929–939, Jun. 2023, doi: 10.1007/S10044-023-01179-3.

Wentian Xin, Ruyi Liu, “Transformer for Skeleton-based action recognition: A review of recent advances,” Neurocomputing, vol. 537, pp. 164–186, 2023, doi: https://doi.org/10.1016/j.neucom.2023.03.001.

C. Beldek, E. Sariyildiz, S. L. Phung, and G. Alici, “GDA-YOLO11: Amodal Instance Segmentation for Occlusion-Robust Robotic Fruit Harvesting,” IEEE Trans. AgriFood Electron., pp. 1–9, 2026, doi: 10.1109/TAFE.2026.3670352.

Nilaksh Das, Sheng-Yun Peng, Duen Horng Chau, “SkeleVision: Towards Adversarial Resiliency of Person Tracking with Multi-Task Learning,” arXiv:2204.00734, 2022, [Online]. Available: https://arxiv.org/abs/2204.00734

Chao Li, Qiaoyong Zhong, Di Xie, Shiliang Pu, “Skeleton-based Action Recognition with Convolutional Neural Networks,” arXiv:1704.07595, 2017, [Online]. Available: https://arxiv.org/abs/1704.07595

Pichao Wang, Zhaoyang Li, Yonghong Hou, Wanqing Li, “Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networks,” arXiv:1611.02447, 2016, [Online]. Available: https://arxiv.org/abs/1611.02447

Y. Hou, Z. Li, P. Wang, and W. Li, “Skeleton Optical Spectra-Based Action Recognition Using Convolutional Neural Networks,” IEEE Trans. Circuits Syst. Video Technol., vol. 28, no. 3, pp. 807–811, Mar. 2018, doi: 10.1109/TCSVT.2016.2628339.

C. Li, Y. Hou, P. Wang, and W. Li, “Joint Distance Maps Based Action Recognition with Convolutional Neural Networks,” IEEE Signal Process. Lett., vol. 24, no. 5, pp. 624–628, May 2017, doi: 10.1109/LSP.2017.2678539.

Mengyuan Liu, Hong Liu, “Enhanced skeleton visualization for view invariant human action recognition,” Pattern Recognit., vol. 68, pp. 346–362, 2017, doi: https://doi.org/10.1016/j.patcog.2017.02.030.

Q. Ke, S. An, M. Bennamoun, F. Sohel, and F. Boussaid, “SkeletonNet: Mining Deep Part Features for 3-D Action Recognition,” IEEE Signal Process. Lett., vol. 24, no. 6, pp. 731–735, Jun. 2017, doi: 10.1109/LSP.2017.2690339.

Zhixuan Yang, Timing Li, “Semi-supervised Human Activity Recognition with individual difference alignment,” Expert Syst. Appl., vol. 275, p. 126976, 2025, doi: https://doi.org/10.1016/j.eswa.2025.126976.

Xi’ang Li, Jinqi Luo, “ActivityGAN: generative adversarial networks for data augmentation in sensor-based human activity recognition,” UbiComp/ISWC 2020 Adjun. - Proc. 2020 ACM Int. Jt. Conf. Pervasive Ubiquitous Comput. Proc. 2020 ACM Int. Symp. Wearable Comput., 2020, [Online]. Available: https://dl.acm.org/doi/10.1145/3410530.3414367

Alireza Abedin, Hamid Rezatofighi, Damith C. Ranasinghe, “Guided-GAN: Adversarial Representation Learning for Activity Recognition with Wearables,” arXiv:2110.05732, 2021, [Online]. Available: https://arxiv.org/abs/2110.05732

M. M. Rahman, S. Z. Gurbuz, and M. G. Amin, “Physics-Aware Generative Adversarial Networks for Radar-Based Human Activity Recognition,” IEEE Trans. Aerosp. Electron. Syst., vol. 59, no. 3, pp. 2994–3008, Jun. 2023, doi: 10.1109/TAES.2022.3221023.

Elnaz Soleimani, Ehsan Nazerfard, “Cross-subject transfer learning in human activity recognition systems using generative adversarial networks,” Neurocomputing, vol. 426, pp. 26–34, 2021, doi: https://doi.org/10.1016/j.neucom.2020.10.056.

“Fully Connected Generative Adversarial Network for Human Activity Recognition | IEEE Journals & Magazine | IEEE Xplore.” Accessed: Apr. 22, 2026. [Online]. Available: https://ieeexplore.ieee.org/document/9893100

L. Qu, Y. Wang, T. Yang, and Y. Sun, “Human Activity Recognition Based on WRGAN-GP-Synthesized Micro-Doppler Spectrograms,” IEEE Sens. J., vol. 22, no. 9, pp. 8960–8973, May 2022, doi: 10.1109/JSEN.2022.3164152.

J. Wang, Y. Chen, Y. Gu, Y. Xiao, and H. Pan, “SensoryGANs: An Effective Generative Adversarial Framework for Sensor-based Human Activity Recognition,” Proc. Int. Jt. Conf. Neural Networks, vol. 2018-July, Oct. 2018, doi: 10.1109/IJCNN.2018.8489106.

Harshala Gammulle, Simon Denman, Sridha Sridharan, Clinton Fookes, “Multi-Level Sequence GAN for Group Activity Recognition,” arXiv:1812.07124, 2018, [Online]. Available: https://arxiv.org/abs/1812.07124

“Find Open Datasets and Machine Learning Projects | Kaggle.” Accessed: Mar. 31, 2026. [Online]. Available: https://www.kaggle.com/datasets