|
|
|
| Automatic Identification Method of Policy Irony Comments Based on Large Language Models |
| Huo Chaoguang1,3, Yin Zhuo1, Yang Yuan1, Yang Wancheng1, Ru Runyu1, Huo Fanfan2 |
1.School of Information Resource Management, Renmin University of China, Beijing 100872
2.Institute of Scientific and Technical Information of China, Beijing 100038
3.Research Center for Digital Humanities, Renmin University of China, Beijing 100872 |
|
|
|
|
Abstract Policy irony comments are extreme and sharp expressions whereby the public voices their opinions on public policies. Automatic and accurate identification is crucial for monitoring policy opinions. Given the scarcity of research on automatic identification methods for policy irony comments and the multiple difficulties involved, this paper proposes a method for automatically identifying policy irony comments based on large language model frameworks. Specifically, using the ChpoBERT, LLaMA-2, GPT-2, and StructBERT frameworks, models for the automatic identification of policy irony comments were constructed and compared. Based on a dataset of 111,628 valid policy comments collected from Sina Weibo, the first dataset of policy irony comments was manually annotated. Additionally, based on the presence or absence of topic labels, the data were further divided into two datasets—one with and one without topic labels—for model training and evaluation. We found that the model built on ChpoBERT achieved the best performance in terms of accuracy, recall, and F1 score, followed by the model built on LLaMA-2. After fine-tuning, the models demonstrated certain performance guarantees. The models constructed in this study establish clear and comparable baseline models for research on the accurate identification of policy irony comments, providing methodological support for policy sentiment monitoring.
|
|
Received: 01 April 2024
|
|
|
|
1 Joshi A, Bhattacharyya P, Carman M J. Automatic sarcasm detection[J]. ACM Computing Surveys, 2017, 50(5): Article No.73.
2 Tay Y, Tuan L A, Hui S C, et al. Reasoning with sarcasm by reading in-between[C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2018: 1010-1020.
3 Huang L, Gino F, Galinsky A D. The highest form of intelligence: sarcasm increases creativity for both expressers and recipients[J]. Organizational Behavior and Human Decision Processes, 2015, 131: 162-177.
4 Bhargava R, Agarwal A, Sharma Y. FAID: feature aftermath for irony discernment[C]// Proceedings of the 9th International Conference on Cloud Computing, Data Science & Engineering. Piscataway: IEEE, 2019: 675-680.
5 Maragoudakis M, Loukis E, Charalabidis Y. A review of opinion mining methods for analyzing citizens’ contributions in public policy debate[C]// Proceedings of the Third International Conference on Electronic Participation. Heidelberg: Springer, 2011: 298-313.
6 Driss O B, Mellouli S, Trabelsi Z. From citizens to government policy-makers: social media data analysis[J]. Government Information Quarterly, 2019, 36(3): 560-570.
7 Hofmann S, Beverungen D, R?ckers M, et al. What makes local governments' online communications successful? Insights from a multi-method analysis of Facebook[J]. Government Information Quarterly, 2013, 30(4): 387-396.
8 Reddick C G, Chatfield A T, Ojo A. A social media text analytics framework for double-loop learning for citizen-centric public services: a case study of a local government Facebook use[J]. Government Information Quarterly, 2017, 34(1): 110-125.
9 Ceron A, Negri F. The “social side” of public policy: monitoring online public opinion and its mobilization during the policy cycle[J]. Policy & Internet, 2016, 8(2): 131-147.
10 Jabalameli S, Xu Y Q, Shetty S. Spatial and sentiment analysis of public opinion toward COVID-19 pandemic using twitter data: at the early stage of vaccination[J]. International Journal of Disaster Risk Reduction, 2022, 80: 103204.
11 Purwitasari D, Putra C B P, Raharjo A B. A stance dataset with aspect-based sentiment information from Indonesian COVID-19 vaccination-related tweets[J]. Data in Brief, 2023, 47: 108951.
12 Catelli R, Pelosi S, Comito C, et al. Lexicon-based sentiment analysis to detect opinions and attitude towards COVID-19 vaccines on Twitter in Italy[J]. Computers in Biology and Medicine, 2023, 158: 106876.
13 Yao Z R, Yang J Y, Liu J L, et al. Comparing tweet sentiments in megacities using machine learning techniques: in the midst of COVID-19[J]. Cities, 2021, 116: 103273.
14 Li Z Y, Dai Y L, Li X L. Construction of sentimental knowledge graph of Chinese government policy comments[J]. Knowledge Management Research & Practice, 2022, 20(1): 73-90.
15 Li L, Li W X, Xu W J. Government affairs microblog investigation and research[C]// Proceedings of the 2023 7th International Seminar on Education, Management and Social Sciences. Paris: Atlantis Press, 2023: 251-259.
16 Sun Y F, Wang Z H, Zhang B, et al. Residents' sentiments towards electricity price policy: evidence from text mining in social media[J]. Resources, Conservation and Recycling, 2020, 160: 104903.
17 Zhao L. Internet public opinion on controversial policy: a case study on Weibo comments on an e-commerce live-streaming policy[C]// Proceedings of the 2nd International Conference on Literature, Art and Human Development. Paris: Atlantis Press, 2020: 559-565.
18 马超, 余辉, 夏文蕾, 等. 政务微博评论中情感极性分析方法研究——以上海公安机构微博为例[J]. 现代情报, 2020, 40(3): 157-168.
19 Li M, Shi Y C. Sentiment analysis and prediction model based on Chinese government affairs microblogs[J]. Heliyon, 2023, 9(8): e19091.
20 Riloff E, Qadir A, Surve P, et al. Sarcasm as contrast between a positive sentiment and negative situation[C]// Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2013: 704-714.
21 Bharti S K, Babu K S, Jena S K. Parsing-based sarcasm sentiment recognition in twitter data[C]// Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. New York: ACM Press, 2015: 1373-1380.
22 Maynard D, Greenwood M A. Who cares about sarcastic tweets? Investigating the impact of sarcasm on sentiment analysis[C]// Proceedings of the 9th International Conference on Language Resources and Evaluation. European Language Resources Association, 2014: 4238-4243.
23 González-Ibá?ez R, Muresan S, Wacholder N. Identifying sarcasm in Twitter: a closer look[C]// Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: Association for Computational Linguistics, 2011: 581-586.
24 Govindan V, Balakrishnan V. A machine learning approach in analysing the effect of hyperboles using negative sentiment tweets for sarcasm detection[J]. Journal of King Saud University-Computer and Information Sciences, 2022, 34(8): 5110-5120.
25 Kumar P, Sarin G. WELMSD-word embedding and language model based sarcasm detection[J]. Online Information Review, 2022, 46(7): 1242-1256.
26 白晓雷, 霍瑞雪. 融合语言特征的反讽文本识别模型研究[J]. 通信技术, 2021, 54(1): 73-76.
27 Son L H, Kumar A, Sangwan S R, et al. Sarcasm detection using soft attention-based bidirectional long short-term memory model with convolution network[J]. IEEE Access, 2019, 7: 23319-23328.
28 Pradhan R, Agarwal G, Singh D. Comparative analysis for sentiment in tweets using LSTM and RNN[C]// Proceedings of the Intelligent Systems and Computing. Singapore: Springer, 2021: 713-725.
29 Mohan A, Nair A M, Jayakumar B, et al. Sarcasm detection using bidirectional encoder representations from transformers and graph convolutional networks[J]. Procedia Computer Science, 2023, 218: 93-102.
30 Sharma D K, Singh B, Agarwal S, et al. Sarcasm detection over social media platforms using hybrid ensemble model with fuzzy logic[J]. Electronics, 2023, 12(4): 937.
31 潘宏鹏, 汪东, 刘忠轶, 等. 考虑反讽语义识别的协同双向编码舆情评论情感分析研究[J]. 情报杂志, 2022, 41(5): 99-105, 111.
32 段恺. 文学批评关键词“反讽”之研究[J]. 北京化工大学学报(社会科学版), 2007(3): 43-47.
33 Warrens M J. Chance-corrected measures for 2×2 tables that coincide with weighted kappa[J]. The British Journal of Mathematical and Statistical Psychology, 2011, 64(2): 355-365.
34 Devlin J, Chang M W, Lee K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: Association for Computational Linguistics, 2019: 4171-4186.
35 霍朝光, 霍帆帆, 王婉如, 等. 基于WordBERT和BiLSTM的政策工具自动分类方法研究[J]. 图书情报知识, 2023, 40(3): 129-138.
36 沈思, 陈猛, 冯暑阳, 等. ChpoBERT: 面向中文政策文本的预训练模型[J]. 情报学报, 2023, 42(12): 1487-1497.
37 Wang W, Bi B, Yan M, et al. StructBERT: incorporating language structures into pre-training for deep language understanding[C/OL]// Proceedings of the 8th International Conference on Learning Representations. (2019-12-20) [2024-04-03]. https://openreview.net/attachment?id=BJgQ4lSFPH&name=original_pdf. 责任编辑 王克平） |
|
|
|
