|
|
|
| Sentiment Analysis of Online Users Based on Multimodal Co-attention |
| Fan Tao1, Wu Peng2, Wang Hao1, Ling Chen2 |
1.School of Information Management, Nanjing University, Nanjing 210023
2.School of Economics and Management, Nanjing University of Science and Technology, Nanjing 210094 |
|
|
|
|
Abstract The sentiment of online users significantly influences the evolution and development of online public opinion events. Accurately recognizing the sentiment of online users has practical implications for managing online public opinion events. Extant studies about the sentiment analysis of online users are mostly based on texts, lacking in research on the combination of texts and images. In multimodal sentiment analysis, existing research typically unifies the comprehensive unimodal features and performs high-dimensional fusion. This can easily cause information redundancy and introduce the noise, ignoring the interaction and correlation between different modalities. Thus, we propose a sentiment analysis model based on multimodal co-attention to analyze the sentiment of online users, unifying word-guided and image-guided attention mechanisms. To this end, we conduct empirical experiments on multimodal datasets, such as “COVID-19.” The results show that the proposed model based on multimodal co-attention is superior to past models and can capture the interaction and relationship between different modalities.
|
|
Received: 06 May 2020
|
|
|
|
1 吴鹏, 强韶华, 高庆宁. 基于SOAR模型的网民群体负面情感建模研究[J]. 中国管理科学, 2018, 26(3): 126-138.
2 Wu P, Li X T, Shen S, et al. Social media opinion summarization using emotion cognition and convolutional neural networks[J]. International Journal of Information Management, 2020, 51: 101978.
3 Zhang W, Wang M, Zhu Y C. Does government information release really matter in regulating contagion-evolution of negative emotion during public emergencies? From the perspective of cognitive big data analytics[J]. International Journal of Information Management, 2020, 50: 498-514.
4 范涛, 吴鹏, 曹琪. 基于深度学习的多模态融合网民情感识别研究[J]. 信息资源管理学报, 2020, 10(1): 39-48.
5 Majumder N, Hazarika D, Gelbukh A, et al. Multimodal sentiment analysis using hierarchical fusion with context modeling[J]. Knowledge-Based Systems, 2018, 161: 124-133.
6 Huang F R, Zhang X M, Zhao Z H, et al. Image-text sentiment analysis via deep multimodal attentive fusion[J]. Knowledge-Based Systems, 2019, 167: 26-37.
7 Zhang Q, Fu J L, Liu X Y, et al. Adaptive co-attention network for named entity recognition in tweets[C]// Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2018: 5674-5681.
8 吴鹏, 应杨, 沈思. 基于双向长短期记忆模型的网民负面情感分类研究[J]. 情报学报, 2018, 37(8): 845-853.
9 辜丽琼, 夏志杰, 宋祖康, 等. 基于在线网民评论情感追踪分析的企业危机舆情应对研究[J]. 情报理论与实践, 2019, 42(12): 67-73.
10 夏一雪, 王娟, 何巍, 等. 基于舆情大数据的突发事件负面情感引导“时度效”研究[J]. 图书与情报, 2019(5): 120-126.
11 朱晓霞, 宋嘉欣, 孟建芳. 基于动态主题——情感演化模型的网络舆情信息分析[J]. 情报科学, 2019, 37(7): 72-78.
12 张鹏, 崔彦琛, 兰月新, 等. 基于扎根理论与词典构建的微博突发事件情感分析与舆情引导策略[J]. 现代情报, 2019, 39(3): 122-131, 143.
13 Balazs J A, Velásquez J D. Opinion mining and information fusion: a survey[J]. Information Fusion, 2016, 27: 95-110.
14 Kamps J, Marx M, Mokken R J, et al. Words with attitude[M]. Institute for Logic, Language and Computation, University of Amsterdam, 2001.
15 黄伟, 范磊. 基于多分类器投票集成的半监督情感分类方法研究[J]. 中文信息学报, 2016, 30(2): 41-49, 106.
16 唐慧丰, 谭松波, 程学旗. 基于监督学习的中文情感分类技术比较研究[J]. 中文信息学报, 2007, 21(6): 88-94, 108.
17 Kim Y. Convolutional neural networks for sentence classification[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2014: 1746-1751.
18 Ma Y K, Peng H Y, Cambria E. Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM[C]// Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2018: 5876-5883.
19 Yanulevskaya V, Uijlings J, Bruni E, et al. In the eye of the beholder: employing statistical analysis and eye tracking for analyzing abstract paintings[C]// Proceedings of the 20th ACM International Conference on Multimedia. New York: ACM Press, 2012: 349-358.
20 Lu X, Suryanarayan P, Adams R B, et al. On shape and the computability of emotions[C]// Proceedings of the 20th ACM International Conference on Multimedia. New York: ACM Press, 2012: 229-238.
21 Machajdik J, Hanbury A. Affective image classification using features inspired by psychology and art theory[C]// Proceedings of the 18th ACM International Conference on Multimedia. New York: ACM Press, 2010: 83-92.
22 Peng X S, Zhang X M, Li Y P, et al. Research on image feature extraction and retrieval algorithms based on convolutional neural network[J]. Journal of Visual Communication and Image Representation, 2020, 69: 102705.
23 Chen T, Borth D, Darrell T, et al. DeepSentiBank: visual sentiment concept classification with deep convolutional neural networks[OL]. [2014-10-30]. https://arxiv.org/pdf/1410.8586v1.pdf.
24 Yang J F, She D Y, Sun M, et al. Visual sentiment prediction based on automatic discovery of affective regions[J]. IEEE Transactions on Multimedia, 2018, 20(9): 2513-2525.
25 Campos V, Jou B, Giró-I-Nieto X. From pixels to sentiment: fine-tuning CNNs for visual sentiment prediction[J]. Image and Vision Computing, 2017, 65: 15-22.
26 Poria S, Cambria E, Bajpai R, et al. A review of affective computing: from unimodal analysis to multimodal fusion[J]. Information Fusion, 2017, 37: 98-125.
27 Poria S, Chaturvedi I, Cambria E, et al. Convolutional MKL based multimodal emotion recognition and sentiment analysis[C]//Proceedings of the 2016 IEEE 16th International Conference on Data Mining. IEEE, 2016: 439-448.
28 Pérez Rosas V, Mihalcea R, Morency L P. Multimodal sentiment analysis of Spanish online videos[J]. IEEE Intelligent Systems, 2013, 28(3): 38-45.
29 Zadeh A, Chen M H, Poria S, et al. Tensor fusion network for multimodal sentiment analysis[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2017: 1103-1114.
30 Williams J, Comanescu R, Radu O, et al. Dnn multimodal fusion techniques for predicting video sentiment[C]// Proceedings of Grand Challenge and Workshop on Human Multimodal Language. Stroudsburg: Association for Computational Linguistics, 2018: 64-72.
31 Song K S, Nho Y H, Seo J H, et al. Decision-level fusion method for emotion recognition using multimodal emotion recognition information[C]// Proceedings of the 2018 15th International Conference on Ubiquitous Robots. IEEE, 2018: 472-476.
32 Deng J, Dong W, Socher R, et al. ImageNet: a large-scale hierarchical image database[C]// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009: 248-255.
33 Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[OL]. [2014-12-19]. https://arxiv.org/pdf/1409.1556v4.pdf.
34 Mikolov T, Sutskever I, Chen K, et al. Distributed representations of words and phrases and their compositionality[C]// Proceedings of the 26th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates, 2013: 3111-3119.
35 Greff K, Srivastava R K, Koutník J, et al. LSTM: a search space odyssey[J]. IEEE Transactions on Neural Networks and Learning Systems, 2017, 28(10): 2222-2232.
36 吴鹏, 刘恒旺, 沈思. 基于深度学习和OCC情感规则的网络舆情情感识别研究[J]. 情报学报, 2017, 36(9): 972-980.
37 金占勇, 田亚鹏, 白莽. 基于长短时记忆网络的突发灾害事件网络舆情情感识别研究[J]. 情报科学, 2019, 37(5): 142-147, 154.
38 Rhanoui M, Mikram M, Yousfi S, et al. A CNN-BiLSTM model for document-level sentiment analysis[J]. Machine Learning and Knowledge Extraction, 2019, 1(3): 832-847. |
|
|
|
