|
|
|
| A Text Semantic Matching Model Based on Knowledge Enhancement |
| Zhang Zhengang, Yu Chuanming |
| School of Information and Safety Engineering, Zhongnan University of Economics and Law, Wuhan 430073 |
|
|
|
|
Abstract Text semantic matching models have been widely used in information retrieval, text mining, and other fields. Current methods mainly predict the semantic relationship of text pairs from the perspective of the text itself, ignoring external knowledge. We propose a new text semantic matching model based on knowledge enhancement to address this issue. The model utilizes knowledge graph entities as external knowledge, effectively models the text’s external knowledge information, and adaptively filters the noise in the external knowledge. Our model achieves the best results for most indicators in Natural Language Inference and Paraphrase Identification. This research will aid in the application of knowledge graphs in text semantic matching tasks and provide a reference for applying knowledge graphs to the information field.
|
|
Received: 30 March 2023
|
|
|
|
1 Yang R Q, Zhang J H, Gao X, et al. Simple and effective text matching with richer alignment features[C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2019: 4699-4709.
2 Tiwari P, Jaiswal A K, Garg S, et al. SANTM: efficient self-attention-driven network for text matching[J]. ACM Transactions on Internet Technology, 2022, 22(3): Article No.55.
3 冯文政, 唐杰. 融合深度匹配特征的答案选择模型[J]. 中文信息学报, 2019, 33(1): 118-124.
4 梁柱, 沈思, 叶文豪, 等. 基于结构内容特征的裁判文书自动推荐研究[J]. 情报学报, 2022, 41(2): 167-175.
5 施元鹏, 单剑峰. 基于文本相似度的简历匹配推荐算法研究[J]. 计算机仿真, 2022, 39(4): 441-444, 491.
6 李纲, 余辉, 毛进. 基于多层语义相似的技术供需文本匹配模型研究[J]. 数据分析与知识发现, 2021, 5(12): 25-36.
7 Wang X, Yang H M. MGMSN: multi-granularity matching model based on Siamese neural network[J]. Frontiers in Bioengineering and Biotechnology, 2022, 10: 839586.
8 Wang H Z, Wang S, Han Y H. Detecting fake news on Chinese social media based on hybrid feature fusion method[J]. Expert Systems with Applications, 2022, 208: 118111.
9 张文慧, 汪美玲, 侯志荣. 融合语境语义差异特征的短文本匹配模型[J]. 北京大学学报(自然科学版), 2023, 59(1): 30-38.
10 魏明珠, 郑荣, 高志豪, 等. 融合知识图谱和深度神经网络的产业新兴技术预测模型研究[J]. 情报学报, 2022, 41(11): 1134-1148.
11 Liu J T, Zhang Z Q, Guo Z, et al. KEPT: knowledge enhanced prompt tuning for event causality identification[J]. Knowledge-Based Systems, 2023, 259: 110064.
12 林健成, 蔺晓川. 知识对话系统的零资源在线更新[J]. 中文信息学报, 2022, 36(5): 76-84, 93.
13 宋鹏程, 单丽莉, 孙承杰, 等. 基于查询路径排序的知识库问答系统[J]. 中文信息学报, 2021, 35(11): 109-117, 126.
14 丁亮, 何彦青. 融合领域知识与深度学习的机器翻译领域自适应研究[J]. 情报科学, 2017, 35(10): 125-132.
15 Chen T, Xu R F, He Y L, et al. A gloss composition and context clustering based distributed word sense representation model[J]. Entropy, 2015, 17(12): 6007-6024.
16 Lyu B E, Chen L, Zhu S, et al. LET: linguistic knowledge enhanced graph transformer for Chinese short text matching[C]// Proceedings of the 35th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2021: 13498-13506.
17 Zhou G Y, Liu Y, Liu F, et al. Improving question retrieval in community question answering using world knowledge[C]// Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2013: 2239-2245.
18 Chen Q, Zhu X D, Ling Z H, et al. Neural natural language inference models enhanced with external knowledge[C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2018: 2406-2417.
19 Huang P S, He X D, Gao J F, et al. Learning deep structured semantic models for web search using clickthrough data[C]// Proceedings of the 22nd ACM International Conference on Information & Knowledge Management. New York: ACM Press, 2013: 2333-2338.
20 Hu B T, Lu Z D, Li H, et al. Convolutional neural network architectures for matching natural language sentences[C]// Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2014: 2042-2050.
21 Palangi H, Deng L, Shen Y L, et al. Deep sentence embedding using long short-term memory networks: analysis and application to information retrieval[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2016, 24(4): 694-707.
22 Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2017: 5998-6008.
23 Devlin J, Chang M W, Lee K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: Association for Computational Linguistics, 2019: 4171-4186.
24 Liu Y H, Ott M, Goyal N, et al. RoBERTa: a robustly optimized BERT pretraining approach[OL]. (2019-07-26) [2022-06-11]. https://arxiv.org/pdf/1907.11692.pdf.
25 Lan Z Z, Chen M D, Goodman S, et al. ALBERT: a lite BERT for self-supervised learning of language representations[OL]. (2020-02-09) [2022-06-11]. https://arxiv.org/pdf/1909.11942.pdf.
26 Sun Y, Wang S H, Li Y K, et al. ERNIE 2.0: a continual pre-training framework for language understanding[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(5): 8968-8975.
27 Sun Y, Wang S H, Feng S K, et al. ERNIE 3.0: large-scale knowledge enhanced pre-training for language understanding and generation[OL]. (2021-07-05) [2022-06-11]. http://arxiv.org/pdf/2107.02137.pdf.
28 Wan S X, Lan Y Y, Guo J F, et al. A deep architecture for semantic matching with multiple positional sentence representations[C]// Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2016: 2835-2841.
29 Pang L, Lan Y Y, Guo J F, et al. Text matching as image recognition[C]// Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2016: 2793-2799.
30 Gong Y C, Luo H, Zhang J. Natural language inference over interaction space[OL]. (2018-05-26) [2020-04-02]. https://arxiv.org/pdf/1709.04348.pdf.
31 Huang G, Liu Z, van Der Maaten L, et al. Densely connected convolutional networks[C]// Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2261-2269.
32 Wang Z G, Hamza W, Florian R. Bilateral multi-perspective matching for natural language sentences[C]// Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2017: 4144-4150.
33 Chen Q, Zhu X D, Ling Z H, et al. Enhanced LSTM for natural language inference[C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2017: 1657-1668.
34 Zhang K, Lyu G Y, Wu L, et al. LadRa-net: locally aware dynamic reread attention net for sentence semantic matching[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(2): 853-866.
35 Miller G A. WordNet: a lexical database for English[J]. Communications of the ACM, 1995, 38(11): 39-41.
36 Xia T Y, Wang Y, Tian Y, et al. Using prior knowledge to guide BERT’s attention in semantic textual matching tasks[C]// Proceedings of the Web Conference 2021. New York: ACM Press, 2021: 2466-2475.
37 Annervaz K M, Basu Roy Chowdhury S, Dukkipati A. Learning beyond datasets: knowledge graph augmented neural networks for natural language processing[C]// Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: Association for Computational Linguistics, 2018: 313-322.
38 Bordes A, Usunier N, Garcia-Duran A, et al. Translating embeddings for modeling multi-relational data[C]// Proceedings of the 27th Annual Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2013: 2799-2807.
39 Wang X Y, Kapanipathi P, Musa R, et al. Improving natural language inference using external knowledge in the science questions domain[C]// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2019: 7208-7215.
40 Liu W J, Zhou P, Zhao Z, et al. K-BERT: enabling language representation with knowledge graph[C]// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2020: 2901-2908.
41 Zhang Z Y, Han X, Liu Z Y, et al. ERNIE: enhanced language representation with informative entities[C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2019: 1441-1451.
42 Yamada I, Asai A, Shindo H, et al. LUKE: deep contextualized entity representations with entity-aware self-attention[C]// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2020: 6442-6454.
43 Song M N, Zhao W, HaiHong E. KGAnet: a knowledge graph attention network for enhancing natural language inference[J]. Neural Computing and Applications, 2020, 32(18): 14963-14973.
44 Huang W Y, Qu Q, Yang M. Interactive knowledge-enhanced attention network for answer selection[J]. Neural Computing and Applications, 2020, 32(15): 11343-11359.
45 Li H Z, Wang W H, Liu Z R, et al. A novel locality-sensitive hashing relational graph matching network for semantic textual similarity measurement[J]. Expert Systems with Applications, 2022, 207: 117832.
46 Xu C L, Wang H, Wu S L, et al. Tag-enhanced dynamic compositional neural network over arbitrary tree structure for sentence representation[J]. Expert Systems with Applications, 2021, 181: 115182.
47 Xu C L, Wang H, Wu S L, et al. TreeLSTM with tag-aware hypernetwork for sentence representation[J]. Neurocomputing, 2021, 434: 11-20.
48 Tai K S, Socher R, Manning C D. Improved semantic representations from tree-structured long short-term memory networks[C]// Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2015: 1556-1566.
49 Zeb A, Saif S, Chen J D, et al. Complex graph convolutional network for link prediction in knowledge graphs[J]. Expert Systems with Applications, 2022, 200: 116796.
50 Ji S X, Pan S R, Cambria E, et al. A survey on knowledge graphs: representation, acquisition, and applications[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(2): 494-514.
51 Dettmers T, Minervini P, Stenetorp P, et al. Convolutional 2D knowledge graph embeddings[C]// Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2018: 1811-1818.
52 Jiang G Y, Liu S, Zhao Y, et al. Fake news detection via knowledgeable prompt learning[J]. Information Processing & Management, 2022, 59(5): 103029.
53 Wang X Z, Gao T Y, Zhu Z C, et al. KEPLER: a unified model for knowledge embedding and pre-trained language representation[J]. Transactions of the Association for Computational Linguistics, 2021, 9: 176-194.
54 Xu B, Xu Y, Liang J Q, et al. CN-DBpedia: a never-ending Chinese knowledge extraction system[C]// Proceedings of the 30th International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems. Cham: Springer, 2017: 428-438.
55 Song C G, Ning N W, Zhang Y L, et al. Knowledge augmented transformer for adversarial multidomain multiclassification multimodal fake news detection[J]. Neurocomputing, 2021, 462: 88-100.
56 Yu C M, Xue H D, Jiang Y F, et al. A simple and efficient text matching model based on deep interaction[J]. Information Processing & Management, 2021, 58(6): 102738.
57 Wehrmann J, Kolling C, Barros R C. Adaptive cross-modal embeddings for image-text alignment[C]// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2020: 12313-12320.
58 Khot T, Sabharwal A, Clark P. SciTail: a textual entailment dataset from science question answering[C]// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2018: 5189-5197.
59 Marelli M, Menini S, Baroni M, et al. A SICK cure for the evaluation of compositional distributional semantic models[C]// Proceedings of the 9th International Conference on Language Resources and Evaluation. Stroudsburg: Association for Computational Linguistics, 2014: 216-223.
60 Dolan B W, Brockett C. Automatically constructing a corpus of sentential paraphrases[C]// Proceedings of the Third International Workshop on Paraphrasing. Stroudsburg: Association for Computational Linguistics, 2005: 9-16.
61 Liu X, Chen Q C, Deng C, et al. LCQMC: a large-scale Chinese question matching corpus[C]// Proceedings of the 27th International Conference on Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2018: 1952-1962.
62 Chen J, Chen Q C, Liu X, et al. The BQ corpus: a large-scale domain-specific Chinese corpus for sentence semantic equivalence identification[C]// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2018: 4946-4951.
63 Xu L, Hu H, Zhang X W, et al. CLUE: a Chinese language understanding evaluation benchmark[C]// Proceedings of the 28th International Conference on Computational Linguistics. International Committee on Computational Linguistics, 2020: 4762-4772.
64 Hu H, Richardson K, Xu L, et al. OCNLI: original Chinese natural language inference[C]// Findings of the Association for Computational Linguistics: EMNLP 2020. Stroudsburg: Association for Computational Linguistics, 2020: 3512-3526.
65 Pennington J, Socher R, Manning C. GloVe: global vectors for word representation[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2014: 1532-1543.
66 Yang B S, Yih S W T, He X D, et al. Embedding entities and relations for learning and inference in knowledge bases[C]// Proceedings of International Conference on Learning Representations. Washington: ICLR, 2015: 1-12.
67 Chen L, Zhao Y B, Lyu B E, et al. Neural graph matching networks for Chinese short text matching[C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2020: 6152-6158.
68 Cui Y M, Che W X, Liu T, et al. Pre-training with whole word masking for Chinese BERT[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3504-3514. |
|
|
|
