|
|
|
| Journal Evaluation Method Based on the Integration of Peer-Review Consensus and Bibliometric Indicators |
| Zhang Xu1, Lyu Dongqing1, Ruan Xuanmin1, Cheng Ying1, Ju Xiufang2 |
1.School of Information Management, Nanjing University, Nanjing 210023
2.Chinese Social Sciences Research Evaluation Center, Nanjing University, Nanjing 210093 |
|
|
|
|
Abstract Current research on journal evaluation lacks in-depth consideration about integrating qualitative and quantitative methods. This study takes advantage of the consensus of peer review among first-rate journals and proposes a new indicator, Journal Authoritativeness Factor (JAF), by integrating peer-review consensus and bibliometric indicators with the help of TrustRank. We also introduce the Journal Authoritativeness and Impact Index (JAII) by combining JAF and Journal Impact Factor (JIF). To verify the effectiveness of JAF and JAII, this study conducts empirical research by selecting journals recommended by the Federation of Management Societies of China (FMS) as the “gold standard” and citation data from 2000 to 2018 in the Chinese Social Sciences Citation Index (CSSCI) database as evaluation data. Results show that JAF is robust among different citation windows. Compared with the two baseline indicators, the ranking of journals by JAF and by FMS are more consistent, with the correlation between the journal ranking of JAII and FMS being as high as 0.716.
|
|
Received: 13 July 2021
|
|
|
|
1 Serenko A, Bontis N. First in, best dressed: the presence of order-effect bias in journal ranking surveys[J]. Journal of Informetrics, 2013, 7(1): 138-144.
2 Cameron B D. Trends in the usage of ISI bibliometric data: uses, abuses, and implications[J]. portal: Libraries and the Academy, 2005, 5(1): 105-125.
3 Truex D, University G S, Cuellar M, et al. Assessing scholarly influence: using the hirsch indices to reframe the discourse[J]. Journal of the Association for Information Systems, 2009, 10(7): 560-594.
4 苏金燕. 人文社会科学期刊评价中同行评议与影响因子相关性分析[J]. 图书情报知识, 2020(3): 128-137.
5 Serenko A, Bontis N. What’s familiar is excellent: the impact of exposure effect on perceived journal quality[J]. Journal of Informetrics, 2011, 5(1): 219-223.
6 Page L, Brin S, Motwani R, et al. The PageRank citation ranking: bringing order to the Web[R]. Stanford InfoLab, 1998.
7 Maslov S, Redner S. Promise and pitfalls of extending Google’s PageRank algorithm to citation networks[J]. Journal of Neuroscience, 2008, 28(44): 11103-11105.
8 Ma N, Guan J C, Zhao Y. Bringing PageRank to the citation analysis[J]. Information Processing & Management, 2008, 44(2): 800-810.
9 SCImago journal & country rank[EB/OL]. [2020-12-22]. http://www.scimagojr.com/index.php.
10 Carl B. Eigenfactor: measuring the value and prestige of scholarly journals[J]. College & Research Libraries News, 2007, 68(5): 314-316.
11 Allen L, Jones C, Dolby K, et al. Looking for landmarks: the role of expert review and bibliometric analysis in evaluating scientific publication outputs[J]. PLoS One, 2009, 4(6): e5910.
12 Lowry P B, Moody G D, Gaskin J, et al. Evaluating journal quality and the association for information systems senior scholars’ journal basket via bibliometric measures: do expert journal assessments add value?[J]. MIS Quarterly, 2013, 37(4): 993-1012.
13 俞立平, 潘云涛, 武夷山. 学术期刊评价中不同利益主体关系研究[J]. 科学学与科学技术管理, 2009, 30(12): 43-47.
14 Gy?ngyi Z, Garcia-Molina H, Pedersen J. Combating Web spam with TrustRank[C]// Proceedings of the Thirtieth International Conference on Very Large Data Bases. VLDB Endowment, 2004: 576-587.
15 Garfield E. Citation indexes for science: a new dimension in documentation through association of ideas[J]. Science, 1955, 122(3159): 108-111.
16 Sharma O P. Journal impact factor: an essential primary quality indicator[J]. Current Science, 2007, 93(1): 5.
17 Rousseau R. Sir, why am I not cited or why are multi-authored papers more cited than others?[J]. Journal of Documentation, 1992, 48(1): 79-80.
18 Pinski G, Narin F. Citation influence for journal aggregates of scientific publications: theory, with application to the literature of physics[J]. Information Processing & Management, 1976, 12(5): 297-312.
19 Chen P, Xie H, Maslov S, et al. Finding scientific gems with Google’s PageRank algorithm[J]. Journal of Informetrics, 2007, 1(1): 8-15.
20 李江, 孙建军. 链接分析对引文分析的启示: 从PageRank到Paperank[J]. 情报学报, 2009, 28(4): 618-625.
21 Ding Y, Yan E J, Frazho A, et al. PageRank for ranking authors in co-citation networks[J]. Journal of the American Society for Information Science and Technology, 2009, 60(11): 2229-2243.
22 Yan E J, Ding Y. Discovering author impact: a PageRank perspective[J]. Information Processing & Management, 2011, 47(1): 125-134.
23 Bollen J, Rodriquez M A, van de Sompel H. Journal status[J]. Scientometrics, 2006, 69(3): 669-687.
24 苏成, 潘云涛, 袁军鹏, 等. 基于PageRank的期刊评价研究[J]. 中国科技期刊研究, 2009, 20(4): 614-617.
25 Cheang B, Chu S K W, Li C S, et al. A multidimensional approach to evaluating management journals: refining PageRank via the differentiation of citation types and identifying the roles that management journals play[J]. Journal of the Association for Information Science and Technology, 2014, 65(12): 2581-2591.
26 van Raan A F J. Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises[J]. Scientometrics, 1996, 36(3): 397-420.
27 Lewison G, Cottrell R, Dixon D. Bibliometric indicators to assist the peer review process in grant decisions[J]. Research Evaluation, 1999, 8(1): 47-52.
28 Gunashekar S, Wooding S, Guthrie S. How do NIHR peer review panels use bibliometric information to support their decisions?[J]. Scientometrics, 2017, 112(3): 1813-1835.
29 Goldstein H, Maier G. The use and valuation of journals in planning scholarship: peer assessment versus impact factors[J]. Journal of Planning Education and Research, 2010, 30(1): 66-75.
30 Maier G. Impact factors and peer judgment: the case of regional science journals[J]. Scientometrics, 2006, 69(3): 651-667.
31 Tsai C F, Hu Y H, George Ke S W. A Borda count approach to combine subjective and objective based MIS journal rankings[J]. Online Information Review, 2014, 38(4): 469-483.
32 Serenko A, Bontis N. Global ranking of knowledge management and intellectual capital academic journals[J]. Journal of Knowledge Management, 2009, 13(1): 4-15.
33 Serenko A, Bontis N. Global ranking of knowledge management and intellectual capital academic journals: 2013 update[J]. Journal of Knowledge Management, 2013, 17(2): 307-326.
34 Serenko A, Bontis N. Global ranking of knowledge management and intellectual capital academic journals: 2017 update[J]. Journal of Knowledge Management, 2017, 21(3): 675-692.
35 Chen Y L, Chen X H. An evolutionary PageRank approach for journal ranking with expert judgements[J]. Journal of Information Science, 2011, 37(3): 254-272.
36 宋晓晨, 李梦豪, 周良. 一种新型期刊评价方法——基于论文作者简介的分析[J]. 情报学报, 2018, 37(9): 874-881.
37 Opatrny T. Playing the system to give low-impact journal more clout[J]. Nature, 2008, 455(7210): 167.
38 Franck G. Scientific communication—A vanity fair?[J]. Science, 1999, 286(5437): 53-55.
39 van Noorden R. Brazilian citation scheme outed[J]. Nature, 2013, 500(7464): 510-511.
40 Cronin B. The citation process: the role and significance of citation in scientific communication[M]. London: Taylor Graham, 1984.
41 Harter S P. Psychological relevance and information science[J]. Journal of the American Society for Information Science, 1992, 43(9): 602-615.
42 谢娟, 成颖, 孙建军, 等. 基于信息使用环境理论的引用行为研究: 参考文献分析的视角[J]. 中国图书馆学报, 2018, 44(5): 59-75.
43 González-Pereira B, Guerrero-Bote V P, Moya-Anegón F. A new approach to the metric of journals’ scientific prestige: the SJR indicator[J]. Journal of Informetrics, 2010, 4(3): 379-391.
44 Franceschet M. The difference between popularity and prestige in the sciences and in the social sciences: a bibliometric analysis[J]. Journal of Informetrics, 2010, 4(1): 55-63.
45 Gyongyi Z, Garcia-Molina H. Seed selection in TrustRank[R]. Stanford University, 2004.
46 Sombatsompop N, Kositchaiyong A, Markpin T, et al. Scientific evaluations of citation quality of international research articles in the SCI database: Thailand case study[J]. Scientometrics, 2006, 66(3): 521-535.
47 李超. “HIF指数”评价科技期刊学术影响的机理与实践[J]. 情报理论与实践, 2011, 34(7): 44-48.
48 王建军, 杨德礼. 管理学CSSCI检索期刊学术影响力分析[J]. 情报学报, 2010, 29(1): 142-150.
49 王映. 加权TOPSIS与RSR法在学术期刊影响力综合评价中的应用研究[J]. 图书情报工作, 2013, 57(2): 92-96.
50 FMS管理科学高质量期刊(中文期刊)[EB/OL]. [2021-03-15]. http://www.fms-journal.net/journals_cn.
51 南京大学. 文科一流期刊目录(2017年修订)[EB/OL]. [2020-12-10]. https://skch.nju.edu.cn//glgz/cgxg/cgrd/20160719/i43082.html.
52 张琳, SivertsenGunnar. 科学计量与同行评议相结合的科研评价——国际经验与启示[J]. 情报学报, 2020, 39(8): 806-816.
53 Association of Universities in the Netherlands. Standard evaluation protocol 2015-2021[R/OL]. (2014-03-21). https://www.rug.nl/about-ug/organization/quality-assurance/research/sep2015-2021. pdf.
54 Ancaiani A, Anfossi A F, Barbara A, et al. Evaluating scientific research in Italy: the 2004-10 research evaluation exercise[J]. Research Evaluation, 2015, 24(3): 242-255.
55 李江, 姜明利, 李玥婷. 引文曲线的分析框架研究——以诺贝尔奖得主的引文曲线为例[J]. 中国图书馆学报, 2014, 40(2): 41-49.
56 Jacsó P. Five-year impact factor data in the Journal Citation Reports[J]. Online Information Review, 2009, 33(3): 603-614.
57 俞立平, 王作功, 孙建红. 时间窗口对学术期刊评价的影响研究[J]. 情报杂志, 2017, 36(10): 137-140.
58 刘海燕, 李琳, 李雪蓉. 《系统工程理论与实践》1981—2015年文献计量分析[J]. 系统工程理论与实践, 2017, 37(3): 805-816.
59 Ayres I, Vars F E. Determinants of citations to articles in elite law reviews[J]. The Journal of Legal Studies, 2000, 29(S1): 427-450.
60 Stremersch S, Verniers I, Verhoef P C. The quest for citations: drivers of article impact[J]. Journal of Marketing, 2007, 71(3): 171-193.
61 李明, 柯青, 石进, 等. 学术期刊作者机构指数AAI排名实证探索——以中文高等教育学期刊为例[J]. 西南民族大学学报(人文社科版), 2019, 40(9): 236-240. |
|
|
|
