|
|
|
| Relevance between Image and Text of Social Media Posts for Disasters |
| Li Gang, Zhang Ji, Mao Jin, Ma Chao |
| Center for Studies of Information Resources, Wuhan University, Wuhan 430072 |
|
|
|
|
Abstract In this study, we understand the relevance between the images and the texts of Weibo during disasters via images and texts analyses. Considering posts related to typhoon mangosteen as research objects, we developed a text and text relevance classification model based on the image semantic understanding framework. This was done using deep and machine learning methods to build a classification model after extracting the features of the images and texts. The deep learning method is superior to the traditional machine learning method when implementing the text and relevance classification task. The experimental data contain information regarding a single type of disaster; however, the annotation data are not used effectively to ensure the consistency of the data. The classification model helps understand the content of microblog during disasters, and the deep learning model can better classify the relevance between the images and the texts.
|
|
Received: 27 November 2019
|
|
|
|
1 丁兆云, 贾焰, 周斌. 微博数据挖掘研究综述[J]. 计算机研究与发展, 2014, 51(4): 691-706.
2 高隽, 谢昭, 张骏, 等. 图像语义分析与理解综述[J]. 模式识别与人工智能, 2010, 23(2): 191-202.
3 朱雪. 微博在应对自然灾害事件中的作用[J]. 新闻世界, 2013(9): 132-133.
4 Houston J B, Hawthorne J, Perreault M F, et al. Social media and disasters: A functional framework for social media use in disaster planning, response, and research[J]. Disasters, 2015, 39(1): 1-22.
5 Kongthon A, Haruechaiyasak C, Pailai J, et al. The role of social media during a natural disaster: A case study of the 2011 Thai flood[J]. International Journal of Innovation and Technology Management, 2014, 11(3): 1440012.
6 Al-Saggaf Y, Simmons P. Social media in Saudi Arabia: Exploring its use during two natural disasters[J]. Technological Forecasting and Social Change, 2015, 95: 3-15.
7 Yin J, Lampert A, Cameron M, et al. Using social media to enhance emergency situation awareness[J]. IEEE Intelligent Systems, 2012, 27(6): 52-59.
8 Cresci S, Tesconi M, Cimino A, et al. A linguistically-driven approach to cross-event damage assessment of natural disasters from social media messages[C]// Proceedings of the 24th International Conference on World Wide Web. New York: ACM Press, 2015: 1195-1200.
9 Peters R, de Albuquerque J P. Investigating images as indicators for relevant social media messages in disaster management[C]// Proceedings of the 12th International Conference on Information Systems for Crisis Response and Management, 2015.
10 Goh D H L, Ang R P, Chua A Y K, et al. Why we share: A study of motivations for mobile media sharing[C]// Proceedings of the International Conference on Active Media Technology. Heidelberg: Springer, 2009: 195-206.
11 Zhao Z Y, Zhu H Y, Xue Z H, et al. An image-text consistency driven multimodal sentiment analysis approach for social media[J]. Information Processing & Management, 2019, 56(6): 102097.
12 Eakins J P. Automatic image content retrieval - Are we getting anywhere?[C]// Proceedings of Third International Conference on Electronic Library and Visual Information Research, 1996: 123-135.
13 王晓光, 徐雷, 李纲. 敦煌壁画数字图像语义描述方法研究[J]. 中国图书馆学报, 2014, 40(1): 50-59.
14 曹健, 陈红倩, 毛典辉, 等. 基于局部特征的图像目标识别问题综述[J]. 中南大学学报(自然科学版), 2013, 44(S2): 258-262.
15 Jaimes A, Chang S F. Conceptual framework for indexing visual information at multiple levels[C]// Proceedings of SPIE: Internet Imaging, 1999, 3964: 2-15.
16 陈金菊, 欧石燕. 数字图像语义标注模型比较与分析[J]. 图书情报工作, 2018, 62(6): 116-124.
17 徐雷, 王晓光. 叙事型图像语义标注模型研究[J]. 中国图书馆学报, 2017, 43(5): 70-83.
18 Liu J, Wang B, Li M J, et al. Dual cross-media relevance model for image annotation[C]// Proceedings of the 15th International Conference on Multimedia. New York: ACM Press, 2007: 605-614.
19 Yeh R A, Do M N, Schwing A G. Unsupervised textual grounding: Linking words to image concepts[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 6125-6134.
20 Jiang T, Tan A H. Discovering image-text associations for cross-media web information fusion[C]// Proceedings of the European Conference on Principles of Data Mining and Knowledge Discovery. Heidelberg: Springer, 2006: 561-568.
21 Hu Y T, Zheng L, Yang Y, et al. Twitter100k: A real-world dataset for weakly supervised cross-media retrieval[J]. IEEE Transactions on Multimedia, 2018, 20(4): 927-938.
22 胡慧君, 刘亚, 刘茂福, 等. 面向微博图文关系识别的统一特征空间映射方法[J]. 武汉大学学报(理学版), 2017, 63(5): 448-452.
23 袁于飞, 杨舒. 秋台风“山竹”将袭华南或为今年登陆我国最强台风[N]. 光明日报, 2018-9-15(5).
24 Pennington J, Socher R, Manning C. Glove: Global vectors for word representation[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2014: 1532-1543.
25 Tsoumakas G, Katakis I. Multi-label classification: An overview[J]. International Journal of Data Warehousing and Mining, 2007, 3(3): 1-13.
26 Fay M P, Proschan M A. Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules[J]. Statistics Surveys, 2010, 4: 1-39. |
|
|
|
