Analysis of Social Media Imagery for Crisis Management Applications

Gul-e-Rana Iftikhar; Masood Ahmad Arbab; Muhammad Iftikhar Khan; Atif Sardar Khan

Authors

Gul-e-Rana Iftikhar Department of Computer Systems Engineering, University of Engineering Technology, Peshawar, Pakistan
Masood Ahmad Arbab Department of Computer Systems Engineering, University of Engineering Technology, Peshawar, Pakistan
Muhammad Iftikhar Khan Department of Electrical Engineering, University of Engineering Technology, Peshawar, Pakistan
Atif Sardar Khan US–Pakistan Center for Advanced Studies in Energy, University of Engineering & Technology, Peshawar, 25000, Pakistan

Keywords:

CNN, Disasters, Fusion, Social media networks, SVM

Abstract

Social media data holds immense potential for real-time disaster response. This study explores leveraging deep learning to automatically detect disaster-related information across various social media platforms. By analyzing the performance of different models in identifying relevant content, we aim to reduce information gathering delays and support timely rescue efforts. Faster information gathering translates to quick deployment of rescue teams, potentially saving lives and minimizing property damage. We evaluate these models on a benchmark dataset and explore the potential of combining them for even greater accuracy. Among the models, VGG16 achieved an accuracy of 81% in identifying disaster-related content. Additionally, exploring different fusion techniques for combining these models further improved accuracy to 83% with Hybrid Fusion. This research offers valuable insights for future exploration of deep learning techniques in disaster management.

References

Z. Wang and X. Ye, “Social media analytics for natural disaster management,” Int. J. Geogr. Inf. Sci., vol. 32, no. 1, pp. 49–72, Jan. 2018, doi: 10.1080/13658816.2017.1367003.

M. Y. Kabir and S. Madria, “A deep learning approach for tweet classification and rescue scheduling for efective disaster management,” GIS Proc. ACM Int. Symp. Adv. Geogr. Inf. Syst., pp. 269–278, Nov. 2019, doi: 10.1145/3347146.3359097;WGROUP:STRING:ACM.

S. G. Sevil, O. Kucuktunc, P. Duygulu, and F. Can, “Automatic tag expansion using visual similarity for photo sharing websites,” Multimed. Tools Appl., vol. 49, no. 1, pp. 81–99, Aug. 2010, doi: 10.1007/S11042-009-0394-5/METRICS.

M. L. Shyu, S. C. Chen, M. Chen, C. Zhang, and K. Sarinnapakorn, “Capturing high-level image concepts via affinity relationships in image database retrieval,” Multimed. Tools Appl., vol. 32, no. 1, pp. 73–92, Jan. 2007, doi: 10.1007/S11042-006-0059-6/METRICS.

N. N. C. Ahmad, Sheharyar Ahmad, Kashif Ahmad, “Convolutional Neural Networks for Disaster Images Retrieval,” MediaEval, vol. 17, pp. 13–15, 2017, [Online]. Available: https://ceur-ws.org/Vol-1984/Mediaeval_2017_paper_11.pdf

P. H. Kashif Ahmad, Pogorelov Konstantin, Michael Riegler, Nicola Conci, “CNN and GAN Based Satellite and Social Media Data Fusion for Disaster Detection,” MediaEval, vol. 17, pp. 13–15, 2017, [Online]. Available: http://slim-sig.irisa.fr/me17/Mediaeval_2017_paper_15.pdf

R. R. Yager and D. P. Filev, “Induced Ordered Weighted Averaging operators,” IEEE Trans. Syst. Man, Cybern. Part B Cybern., vol. 29, no. 2, pp. 141–150, Apr. 1999, doi: 10.1109/3477.752789.

D.-T. D.-N. Dao, Minh-Son Pham, Quang-Nhat-Minh, “A Domain-based Late-Fusion for Disaster Image Retrieval from Social Media,” MediaEval, vol. 17, pp. 13–15, 2017, [Online]. Available: https://ceur-ws.org/Vol-1984/Mediaeval_2017_paper_24.pdf

M. S. Dao, P. Quang Nhat Minh, A. Kasem, and M. S. Haja Nazmudeen, “A context-aware late-fusion approach for disaster image retrieval from social media,” ICMR 2018 - Proc. 2018 ACM Int. Conf. Multimed. Retr., pp. 266–273, Jun. 2018, doi: 10.1145/3206025.3206047;PAGE:STRING:ARTICLE/CHAPTER.

Y. Wei et al., “Cross-Modal Retrieval With CNN Visual Features: A New Baseline,” IEEE Trans. Cybern., vol. 47, no. 2, pp. 449–460, Feb. 2017, doi: 10.1109/TCYB.2016.2519449.

A. Eutamene, H. Belhadef, and M. K. Kholladi, “New Process Ontology-Based Character Recognition,” Commun. Comput. Inf. Sci., vol. 240 CCIS, pp. 137–144, 2011, doi: 10.1007/978-3-642-24731-6_13.

M. A. I. Osama A. Shawky, Ahmed Hagag, El-Sayed A. El-Dahshan a, “Remote sensing image scene classification using CNN-MLP with data augmentation,” Optik (Stuttg)., vol. 221, p. 165356, 2020, doi: https://doi.org/10.1016/j.ijleo.2020.165356.

B. R. Eike Blomeier, Sebastian Schmidt, “Drowning in the Information Flood: Machine-Learning-Based Relevance Classification of Flood-Related Tweets for Disaster Management,” Information, vol. 15, no. 3, p. 149, 2024, doi: https://doi.org/10.3390/info15030149.

GeeksforGeeks. (n.d.), “CNN in machine learning”, [Online]. Available: https://www.geeksforgeeks.org/cnn-in-machine-learning/

J. Gu et al., “Recent advances in convolutional neural networks,” Pattern Recognit., vol. 77, pp. 354–377, May 2018, doi: 10.1016/J.PATCOG.2017.10.013.

B. Dickson, “What are convolutional neural networks (CNN)?,” Bdtechtalks.Com., 2020, [Online]. Available: https://towardsml.wordpress.com/2018/10/16/deep-learning-series-p2-understanding-convolutional-neural-networks/

P. Mahajan, “Fully Connected vs Convolutional Neural Networks,” Medium, 2020, [Online]. Available: https://medium.com/swlh/fully-connected-vs-convolutional-neural-networks-813ca7bc6ee5

N. B. D. Timea Bezdan, “Convolutional Neural Network Layers and Architectures,” Int. Sci. Conf. Inf. Technol. Data Relat. Res., 2019, doi: https://doi.org/10.15308/Sinteza-2019-445-451.

I. K. Georgios Kontonatsios, Sally Spencer, Peter Matthew, “Using a neural network-based feature extraction method to facilitate citation screening for systematic reviews,” Expert Syst. with Appl. X, vol. 6, p. 6100030, 2020, doi: https://doi.org/10.1016/j.eswax.2020.100030.

J. Cervantes, F. Garcia-Lamont, L. Rodríguez-Mazahua, and A. Lopez, “A comprehensive survey on support vector machine classification: Applications, challenges and trends,” Neurocomputing, vol. 408, pp. 189–215, Sep. 2020, doi: 10.1016/J.NEUCOM.2019.10.118.

H. Ergun, Y. C. Akyuz, M. Sert, and J. Liu, “Early and Late Level Fusion of Deep Convolutional Neural Networks for Visual Concept Recognition,” https://doi.org/10.1142/S1793351X16400158, vol. 10, no. 3, pp. 379–397, Nov. 2016, doi: 10.1142/S1793351X16400158.

H. Gunes and M. Piccardi, “Affect recognition from face and body: Early fusion vs. late fusion,” Conf. Proc. - IEEE Int. Conf. Syst. Man Cybern., vol. 4, pp. 3437–3443, 2005, doi: 10.1109/ICSMC.2005.1571679.

C. G. M. Snoek, M. Worring, and A. W. M. Smeulders, “Early versus late fusion in semantic video analysis,” Proc. 13th ACM Int. Conf. Multimedia, MM 2005, pp. 399–402, 2005, doi: 10.1145/1101149.1101236;TOPIC:TOPIC:CONFERENCE-COLLECTIONS>MM;PAGE:STRING:ARTICLE/CHAPTER.

S. Balaji, “Binary Image classifier CNN using TensorFlow,” Medium, 2020, [Online]. Available: https://medium.com/techiepedia/binary-image-classifier-cnn-using-tensorflow-a3f5d6746697

N. Thakur, E. Bhattacharjee, R. Jain, B. Acharya, and Y. C. Hu, “Deep learning-based parking occupancy detection framework using ResNet and VGG-16,” Multimed. Tools Appl., vol. 83, no. 1, pp. 1941–1964, Jan. 2024, doi: 10.1007/S11042-023-15654-W/METRICS.

M. I. Firoj Alam, Ferda Ofli, “CrisisMMD: Multimodal Twitter Datasets from Natural Disasters,” Proc. Int. AAAI Conf. Web Soc. Media, vol. 12, no. 1, 2018, doi: https://doi.org/10.1609/icwsm.v12i1.14983.