|
|
|
| Social Media Multimodal Analysis for Emergency Management |
| Xu Yuan1, Mao Jin1,2, Li Gang1,2 |
1.School of Information Management, Wuhan University, Wuhan 430072
2.Center for Studies of Information Resources, Wuhan University, Wuhan 430072 |
|
|
|
|
Abstract Social media has become an important information source for situation awareness during emergencies. Multimodal information such as text, images, and videos viewed through social media can be widely used for emergency information management. Multimodal information analysis during emergencies is a challenging topic and has received significant attention from both academia and industry. We review studies on social media analysis in emergencies. The multi-dimensional characteristics of multimodal information in social media are analyzed using three dimensions: content, spatial-temporal, and information dissemination. The key methods and technologies of multimodal information analysis are summarized from three aspects: information acquisition, information integration, and information mining. Finally, we construct a multimodal information analysis framework for emergency management based on this review and propose future research directions on information acquisition, description, analysis, and visualization. We expect this survey to provide guidance for research on and practice of social media multimodal information analysis and to improve the ability to manage emergencies.
|
|
Received: 05 February 2021
|
|
|
|
1 Reuter C, Ludwig T, Kaufhold M A, et al. Emergency services? attitudes towards social media: a quantitative and qualitative survey across Europe[J]. International Journal of Human-Computer Studies, 2016, 95: 96-111.
2 Zhang C, Fan C, Yao W L, et al. Social media for intelligent public information and warning in disasters: an interdisciplinary review[J]. International Journal of Information Management, 2019, 49: 190-207.
3 Steiger E, de Albuquerque J P, Zipf A. An advanced systematic literature review on spatiotemporal analyses of Twitter data[J]. Transactions in GIS, 2015, 19(6): 809-834.
4 曾大军, 曹志冬. 突发事件态势感知与决策支持的大数据解决方案[J]. 中国应急管理, 2013(11): 15-23.
5 Alam F, Ofli F, Imran M. Descriptive and visual summaries of disaster events using artificial intelligence techniques: case studies of Hurricanes Harvey, Irma, and Maria[J]. Behaviour & Information Technology, 2020, 39(3): 288-318.
6 Cho S E, Jung K, Park H W. Social media use during Japan’s 2011 earthquake: how Twitter transforms the locus of crisis communication[J]. Media International Australia, 2013, 149(1): 28-40.
7 Vieweg S E. Situational awareness in mass emergency: a behavioral and linguistic analysis of Microblogged communications[D]. Ann Arbor: University of Colorado at Boulder, 2012.
8 Rudra K, Ghosh S, Ganguly N, et al. Extracting situational information from Microblogs during disaster events: a classification-summarization approach[C]// Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. New York: ACM Press, 2015: 583-592.
9 Huang Q Y, Xiao Y. Geographic situational awareness: mining tweets for disaster preparedness, emergency response, impact, and recovery[J]. ISPRS International Journal of Geo-Information, 2015, 4(3): 1549-1568.
10 David C C, Ong J C, Legara E F T. Tweeting supertyphoon Haiyan: evolving functions of Twitter during and after a disaster event[J]. PLoS one, 2016, 11(3): e0150190.
11 Kim J, Bae J, Hastak M. Emergency information diffusion on online social media during storm Cindy in US[J]. International Journal of Information Management, 2018, 40: 153-165.
12 Peters R, Albuquerque J P D. Investigating images as indicators for relevant social media messages in disaster management[C]// Proceedings of the ISCRAM 2015 Conference, 2015: 24-27.
13 李纲, 张霁, 毛进, 等. 灾害事件下社交媒体图文相关性研究[J]. 情报学报, 2020, 39(11): 1223-1231.
14 Chae J, Thom D, Jang Y, et al. Public behavior response analysis in disaster events utilizing visual analytics of microblog data[J]. Computers & Graphics, 2014, 38: 51-60.
15 Avvenuti M, Cresci S, La Polla M N, et al. Nowcasting of earthquake consequences using big social data[J]. IEEE Internet Computing, 2017, 21(6): 37-45.
16 Spence P R, Lachlan K A, Lin X L, et al. Variability in Twitter content across the stages of a natural disaster: implications for crisis communication[J]. Communication Quarterly, 2015, 63(2): 171-186.
17 Crooks A, Croitoru A, Stefanidis A, et al. Earthquake: Twitter as a distributed sensor system[J]. Transactions in GIS, 2013, 17(1): 124-147.
18 Kent J D, Capello H T. Spatial patterns and demographic indicators of effective social media content during the Horsethief Canyon fire of 2012[J]. Cartography and Geographic Information Science, 2013, 40(2): 78-89.
19 Takahashi B, Tandoc E C, Carmichael C. Communicating on Twitter during a disaster: an analysis of tweets during Typhoon Haiyan in the Philippines[J]. Computers in Human Behavior, 2015, 50: 392-398.
20 李纲, 陈思菁, 毛进, 等. 自然灾害事件微博热点话题的时空对比分析[J]. 数据分析与知识发现, 2019, 3(11): 1-15.
21 Hagen L, Keller T, Neely S, et al. Crisis communications in the age of social media: a network analysis of Zika-related tweets[J]. Social Science Computer Review, 2018, 36(5): 523-541.
22 Dong R S, Li L B, Zhang Q P, et al. Information diffusion on social media during natural disasters[J]. IEEE Transactions on Computational Social Systems, 2018, 5(1): 265-276.
23 曹学艳, 段飞飞, 方宽, 等. 网络论坛视角下突发事件舆情的关键节点识别及分类研究[J]. 图书情报工作, 2014, 58(4): 65-70.
24 Kim J, Hastak M. Social network analysis: characteristics of online social networks after a disaster[J]. International Journal of Information Management, 2018, 38(1): 86-96.
25 Yoo E, Rand W, Eftekhar M, et al. Evaluating information diffusion speed and its determinants in social media networks during humanitarian crises[J]. Journal of Operations Management, 2016, 45: 123-133.
26 Son J, Lee H K, Jin S, et al. Content features of tweets for effective communication during disasters: a media synchronicity theory perspective[J]. International Journal of Information Management, 2019, 45: 56-68.
27 Sutton J, Spiro E S, Johnson B, et al. Warning tweets: serial transmission of messages during the warning phase of a disaster event[J]. Information, Communication & Society, 2014, 17(6): 765-787.
28 Li L F, Zhang Q P, Wang X, et al. Characterizing the propagation of situational information in social media during COVID-19 epidemic: a case study on Weibo[J]. IEEE Transactions on Computational Social Systems, 2020, 7(2): 556-562.
29 陈思菁, 李纲, 毛进, 等. 突发事件信息传播网络中的关键节点动态识别研究[J]. 情报学报, 2019, 38(2): 178-190.
30 Zheng X, Sun A X. Collecting event-related tweets from twitter stream[J]. Journal of the Association for Information Science and Technology, 2019, 70(2): 176-186.
31 赵洁, 何文娟, 鲁晓阳. 一种基于新闻学的微博事件特征选择方法[J]. 情报学报, 2015, 34(4): 424-431.
32 Pogrebnyakov N, Maldonado E. Didn’t roger that: social media message complexity and situational awareness of emergency responders[J]. International Journal of Information Management, 2018, 40: 166-174.
33 马超, 李纲, 陈思菁, 等. 基于多模态数据语义融合的旅游在线评论有用性识别研究[J]. 情报学报, 2020, 39(2): 199-207.
34 Yang Y M, Ha H Y, Fleites F, et al. Hierarchical disaster image classification for situation report enhancement[C]// Proceedings of the 2011 IEEE International Conference on Information Reuse & Integration. IEEE, 2011: 181-186.
35 Alam F, Ofli F, Imran M. Processing social media images by combining human and machine computing during crises[J]. International Journal of Human-Computer Interaction, 2018, 34(4): 311-327.
36 杨腾飞, 解吉波, 李振宇, 等. 微博中蕴含台风灾害损失信息识别和分类方法[J]. 地球信息科学学报, 2018, 20(7): 906-917.
37 Rexiline Ragini J, Rubesh Anand P M, Bhaskar V. Mining crisis information: a strategic approach for detection of people at risk through social media analysis[J]. International Journal of Disaster Risk Reduction, 2018, 27: 556-566.
38 邱奇志, 周三三, 刘长发, 等. 基于文体和词表的突发事件信息抽取研究[J]. 中文信息学报, 2018, 32(9): 56-65, 74.
39 Cheng Z Y, Caverlee J, Lee K. You are where you tweet: a content-based approach to Geo-locating Twitter users[C]// Proceedings of the 19th ACM International Conference on Information and Knowledge Management. New York: ACM Press, 2010: 759-768.
40 Dalvi N, Kumar R, Pang B. Object matching in tweets with spatial models[C]// Proceedings of the Fifth ACM International Conference on Web Search and Data Mining. New York: ACM Press, 2012: 43.
41 Lee B, Hwang B Y. A study of the correlation between the spatial attributes on Twitter[C]// Proceedings of the 2012 IEEE 28th International Conference on Data Engineering Workshops. IEEE, 2012: 337-340.
42 Doersch C, Singh S, Gupta A, et al. What makes Paris look like Paris?[J]. ACM Transactions on Graphics, 2012, 31(4): 1-9.
43 Vo N, Jacobs N, Hays J. Revisiting IM2GPS in the deep learning era[C]// Proceedings of the 2017 IEEE International Conference on Computer Vision. IEEE, 2017: 2640-2649.
44 Imran M, Ofli F, Caragea D, et al. Using AI and social media multimodal content for disaster response and management: opportunities, challenges, and future directions[J]. Information Processing & Management, 2020, 57(5): 102261.
45 Poblete B, Guzmán J, Maldonado J, et al. Robust detection of extreme events using Twitter: worldwide earthquake monitoring[J]. IEEE Transactions on Multimedia, 2018, 20(10): 2551-2561.
46 Sakaki T, Okazaki M, Matsuo Y. Tweet analysis for real-time event detection and earthquake reporting system development[J]. IEEE Transactions on Knowledge and Data Engineering, 2013, 25(4): 919-931.
47 Zhou X M, Chen L. Event detection over twitter social media streams[J]. The VLDB Journal, 2014, 23(3): 381-400.
48 Y?lmaz Y, Hero A O. Multimodal event detection in Twitter hashtag networks[J]. Journal of Signal Processing Systems, 2018, 90(2): 185-200.
49 Daly S, Thom J A. Mining and classifying image posts on social media to analyse fires[C]// Proceedings of the International Conference on Information Systems, 2016: 1-14.
50 Pohl D, Bouchachia A, Hellwagner H. Online indexing and clustering of social media data for emergency management[J]. Neurocomputing, 2016, 172: 168-179.
51 Wu Q H, Ma S L, Liu Y Z. Sub-event discovery and retrieval during natural hazards on social media data[J]. World Wide Web, 2016, 19(2): 277-297.
52 Meladianos P, Xypolopoulos C, Nikolentzos G, et al. An optimization approach for sub-event detection and summarization in Twitter[C]// Proceedings of the European Conference on Information Retrieval. Cham: Springer, 2018: 481-493.
53 Rudra K, Banerjee S, Ganguly N, et al. Summarizing situational tweets in crisis scenario[C]// Proceedings of the 27th ACM Conference on Hypertext and Social Media. New York: ACM Press, 2016: 137-147.
54 Xu Z, Liu Y H, Yen N Y, et al. Crowdsourcing based description of urban emergency events using social media big data[J]. IEEE Transactions on Cloud Computing, 2020, 8(2): 387-397.
55 Kuang C, Tang J Y, Liu Z Y, et al. ImgWordle: image and text visualization for events in microblogging services[C]// Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces. New York: ACM Press, 2014: 371-372.
56 Xu Z, Liu Y H, Zhang H, et al. Building the multi-modal storytelling of urban emergency events based on crowdsensing of social media analytics[J]. Mobile Networks and Applications, 2017, 22(2): 218-227.
57 李显鑫. 基于图文相关的突发事件图片采集研究[D]. 武汉: 武汉大学, 2020.
58 Bian J W, Yang Y, Zhang H W, et al. Multimedia summarization for social events in Microblog stream[J]. IEEE Transactions on Multimedia, 2015, 17(2): 216-228.
59 Schinas M, Papadopoulos S, Kompatsiaris Y, et al. Visual event summarization on social media using topic modelling and graph-based ranking algorithms[C]// Proceedings of the 5th ACM on International Conference on Multimedia Retrieval. New York: ACM Press, 2015: 203-210.
60 Qian X M, Li M D, Ren Y Y, et al. Social media based event summarization by user-text-image co-clustering[J]. Knowledge-Based Systems, 2019, 164: 107-121.
61 刘雅姝, 张海涛, 徐海玲, 等. 多维特征融合的网络舆情突发事件演化话题图谱研究[J]. 情报学报, 2019, 38(8): 798-806.
62 安璐, 杜廷尧, 余传明, 等. 突发公共卫生事件的微博主题演化模式和时序趋势——以Twitter和Weibo的埃博拉微博为例[J]. 情报资料工作, 2016(5): 44-52.
63 Huang Q Y, Cervone G, Zhang G M. A cloud-enabled automatic disaster analysis system of multi-sourced data streams: an example synthesizing social media, remote sensing and Wikipedia data[J]. Computers, Environment and Urban Systems, 2017, 66: 23-37.
64 赵又霖, 庞烁, 吴宗大. 社会感知数据驱动下突发事件应急管理的时空语义模型构建研究[J]. 情报科学, 2021, 39(2): 44-53.
65 吴林, 安璐, 孙冉. 面向企业舆情监测的事件画像与高危人群预测研究[J]. 信息资源管理学报, 2020, 10(1): 15-28.
66 夏立新, 陈健瑶, 余华娟. 基于事理图谱的多维特征网络舆情事件可视化摘要生成研究[J]. 情报理论与实践, 2020, 43(10): 157-164.
67 张晨昕, 饶元, 樊笑冰, 等. 基于社交媒体的事件脉络挖掘研究进展[J]. 中文信息学报, 2019, 33(11): 15-30.
68 Xu S Z, Kong L, Zhang Y. A picture paints a thousand words: a method of generating image-text timelines[C]// Proceedings of the 21st ACM International Conference on Information and Knowledge Management. New York: ACM Press, 2012: 2511-2514.
69 Utz S, Schultz F, Glocka S. Crisis communication online: how medium, crisis type and emotions affected public reactions in the Fukushima Daiichi nuclear disaster[J]. Public Relations Review, 2013, 39(1): 40-46.
70 Kumar A, Garg G. Sentiment analysis of multimodal twitter data[J]. Multimedia Tools and Applications, 2019, 78(17): 24103-24119.
71 Huang F R, Wei K M, Weng J, et al. Attention-based modality-gated networks for image-text sentiment analysis[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2020, 16(3): 1-19.
72 曹树金, 岳文玉. 突发公共卫生事件微博舆情主题挖掘与演化分析[J]. 信息资源管理学报, 2020, 10(6): 28-37.
73 安璐, 欧孟花. 突发公共卫生事件利益相关者的社会网络情感图谱研究[J]. 图书情报工作, 2017, 61(20): 120-130.
74 安璐, 胡俊阳, 李纲. 基于主题一致性和情感支持的评论意见领袖识别方法研究[J]. 管理科学, 2019, 32(1): 3-13.
75 Ragini J R, Anand P M R, Bhaskar V. Big data analytics for disaster response and recovery through sentiment analysis[J]. International Journal of Information Management, 2018, 42: 13-24.
76 Purohit H, Castillo C, Diaz F, et al. Emergency-relief coordination on social media: automatically matching resource requests and offers[J/OL]. First Monday, 2014, 19(1). (2013-12-28). https://doi.org/10.5210/fm.v19i1.4848.
77 Harb J G D, Ebeling R, Becker K. A framework to analyze the emotional reactions to mass violent events on Twitter and influential factors[J]. Information Processing & Management, 2020, 57(6): 102372.
78 徐永顺, 周宇, 刘渊, 等. 重大突发公共卫生事件中的居民情感需求挖掘[J]. 图书馆论坛, 2021, 41(9): 76-86.
79 Yang T F, Xie J B, Li G Q, et al. Social media big data mining and spatio-temporal analysis on public emotions for disaster mitigation[J]. ISPRS International Journal of Geo-Information, 2019, 8(1): 29.
80 Kryvasheyeu Y, Chen H, Obradovich N, et al. Rapid assessment of disaster damage using social media activity[J]. Science Advances, 2016, 2(3): e1500779.
81 Liang Y, Caverlee J, Mander J. Text vs. images: on the viability of social media to assess earthquake damage[C]// Proceedings of the 22nd International Conference on World Wide Web. New York: ACM Press, 2013: 1003-1006.
82 Kropivnitskaya Y, Tiampo K F, Qin J H, et al. Real-time earthquake intensity estimation using streaming data analysis of social and physical sensors[J]. Pure and Applied Geophysics, 2017, 174(6): 2331-2349.
83 Nguyen D T, Ofli F, Imran M, et al. Damage assessment from social media imagery data during disasters[C]// Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017. New York: ACM Press, 2017: 569-576.
84 Li X K, Caragea D, Zhang H Y, et al. Localizing and quantifying damage in social media images[C]// Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. IEEE, 2018: 194-201.
85 de Albuquerque J P, Herfort B, Brenning A, et al. A geographic approach for combining social media and authoritative data towards identifying useful information for disaster management[J]. International Journal of Geographical Information Science, 2015, 29(4): 667-689.
86 Arthur R, Boulton C A, Shotton H, et al. Social sensing of floods in the UK[J]. PLoS One, 2018, 13(1): e0189327.
87 Brouwer T, Eilander D, van Loenen A, et al. Probabilistic flood extent estimates from social media flood observations[J]. Natural Hazards and Earth System Sciences, 2017, 17(5): 735-747.
88 Li Z L, Wang C Z, Emrich C T, et al. A novel approach to leveraging social media for rapid flood mapping: a case study of the 2015 South Carolina floods[J]. Cartography and Geographic Information Science, 2018, 45(2): 97-110. |
|
|
|
