|
|
|
| Dual-Dimension Scientist Evaluation Framework Based on the Disruptive and Consolidating Impact of Their Papers |
| Yang Alex J.1,2, Kong Jia1,2, Zhang Yiwei1,2, Wang Hao1,2, Deng Sanhong1,2 |
1.School of Information Management, Nanjing University, Nanjing 210023
2.Jiangsu Key Laboratory of Data Engineering and Knowledge Service, Nanjing 210023 |
|
|
|
|
Abstract Using the rich mutual citation information in citation networks, the citation links of scientific papers can be categorized into two citation types: disruptive and consolidating. This study adopted a dual perspective to examine a scientist's disruptive and consolidating impact and proposes a measurement framework that divides scientists into two academic characteristics. We can obtain valuable insights by condensing various types of scientists into these two categories. To evaluate the consistency and effectiveness of this framework, we conducted an empirical analysis using the American Physical Society (APS) dataset, which comprises 463,348 papers, 9,370,286 citation links, and 234,086 post-disambiguated scholars. We employed Kendall's tau correlation, the identification proportion, and the average rank as evaluation metrics. The results indicate that disruptive citations and the disruptive h-index indicate a high consistency with traditional indices while surpassing benchmark indices in terms of the convergence efficiency. The disruptive influence of scientists serves as an indicator of their innovation level and potential. The dual measurement framework accurately and effectively captures the influence of scientists, providing a means for the early identification of future innovative researchers, reforming performance and reward systems, reviewing and evaluating funding projects, and formulating incentive policies for scientific research.
|
|
Received: 26 February 2023
|
|
|
|
1 Winnink J J, Tijssen R J W, van Raan A F J. Searching for new breakthroughs in science: how effective are computerised detection algorithms?[J]. Technological Forecasting and Social Change, 2019, 146: 673-686.
2 Fortunato S, Bergstrom C T, B?rner K, et al. Science of science[J]. Science, 2018, 359(6379): eaao0185.
3 Jo W S, Liu L, Wang D S. See further upon the giants: quantifying intellectual lineage in science[J]. Quantitative Science Studies, 2022, 3(2): 319-330.
4 Ghosal T, Tiwary P, Patton R, et al. Towards establishing a research lineage via identification of significant citations[J]. Quantitative Science Studies, 2022, 2(4): 1511-1528.
5 Kuhn T S. Historical structure of scientific discovery[J]. Science, 1962, 136(3518): 760-764.
6 Wu L F, Wang D S, Evans J A. Large teams develop and small teams disrupt science and technology[J]. Nature, 2019, 566(7744): 378-382.
7 Park M, Leahey E, Funk R J. Papers and patents are becoming less disruptive over time[J]. Nature, 2023, 613(7942): 138-144.
8 Uzzi B, Mukherjee S, Stringer M, et al. Atypical combinations and scientific impact[J]. Science, 2013, 342(6157): 468-472.
9 苏成, 赵志耘, 赵筱媛, 等. 颠覆性技术新阐释: 概念、内涵及特征[J]. 情报学报, 2021, 40(12): 1253-1262.
10 崔怡雯, 赵筱媛, 苏成, 等. 面向颠覆性创新的技术监测分类体系研究[J]. 情报学报, 2021, 40(12): 1288-1293.
11 Wang D S, Barabási A L. The science of science[M]. Cambridge: Cambridge University Press, 2021.
12 Funk R J, Owen-Smith J. A dynamic network measure of technological change[J]. Management Science, 2017, 63(3): 791-817.
13 Bornmann L, Devarakonda S, Tekles A, et al. Disruptive papers published in Scientometrics: meaningful results by using an improved variant of the disruption index originally proposed by Wu, Wang, and Evans (2019)[J]. Scientometrics, 2020, 123(2): 1149-1155.
14 Wang S Y, Ma Y X, Mao J, et al. Quantifying scientific breakthroughs by a novel disruption indicator based on knowledge entities[J]. Journal of the Association for Information Science and Technology, 2023, 74(2): 150-167.
15 Wei C L, Li J, Shi D B. Quantifying revolutionary discoveries: evidence from Nobel prize-winning papers[J]. Information Processing & Management, 2023, 60(3): 103252.
16 Wang R J, Zhou Y H, Zeng A. Evaluating scientists by citation and disruption of their representative works[J]. Scientometrics, 2023, 128(3): 1689-1710.
17 Yin Y A, Wang Y, Evans J A, et al. Quantifying the dynamics of failure across science, startups and security[J]. Nature, 2019, 575(7781): 190-194.
18 Foster J G, Rzhetsky A, Evans J A. Tradition and innovation in scientists’ research strategies[J]. American Sociological Review, 2015, 80(5): 875-908.
19 Bu Y, Waltman L, Huang Y. A multidimensional framework for characterizing the citation impact of scientific publications[J]. Quantitative Science Studies, 2021, 2(1): 155-183.
20 Bower J L, Christensen C M. Disruptive technologies: catching the wave[M]. Boston: Harvard Business Review, 1995.
21 Zeng A, Shen Z S, Zhou J L, et al. The science of science: from the perspective of complex systems[J]. Physics Reports, 2017, 714/715: 1-73.
22 Fontana M, Iori M, Montobbio F, et al. New and atypical combinations: an assessment of novelty and interdisciplinarity[J]. Research Policy, 2020, 49(7): 104063.
23 Wang J, Veugelers R, Stephan P. Bias against novelty in science: a cautionary tale for users of bibliometric indicators[J]. Research Policy, 2017, 46(8): 1416-1436.
24 Shibayama S, Wang J. Measuring originality in science[J]. Scientometrics, 2020, 122(1): 409-427.
25 Mokyr J. Punctuated equilibria and technological progress[J]. The American Economic Review, 1990, 80(2): 350-354.
26 Arthur W B. The nature of technology: what it is and how it evolves[M]. New York: Free Press, 2009.
27 Arthur W B. The structure of invention[J]. Research Policy, 2007, 36(2): 274-287.
28 Einstein A. The foundation of the general theory of relativity[J]. Annalen Der Physik, 1916, 354(7): 769-822.
29 Watson J D, Crick F H C. Molecular structure of nucleic acids: a structure for deoxyribose nucleic acid[J]. Nature, 1953, 171(4356): 737-738.
30 Bak P, Tang C, Wiesenfeld K. Self-organized criticality: an explanation of the 1/f noise[J]. Physical Review Letters, 1987, 59(4): 381-384.
31 Lin Y L, Frey C B, Wu L F. Remote collaboration fuses fewer breakthrough ideas[J]. Nature, 2023, 623(7989): 987-991.
32 Lin Y L, Evans J A, Wu L F. New directions in science emerge from disconnection and discord[J]. Journal of Informetrics, 2022, 16(1): 101234.
33 Yang A J, Deng S H, Wang H, et al. Disruptive coefficient and 2-step disruptive coefficient: novel measures for identifying vital nodes in complex networks[J]. Journal of Informetrics, 2023, 17: 101411.
34 Yang A J, Hu H T, Zhao Y H, et al. From consolidation to disruption: a novel way to measure the impact of scientists and identify laureates[J]. Information Processing & Management, 2023, 60(5): 103420.
35 Azoulay P. Small research teams ‘disrupt’ science more radically than large ones[J]. Nature, 2019, 566(7744): 330-332.
36 Hirsch J E. An index to quantify an individual's scientific research output[J]. Proceedings of the National Academy of Sciences of the United States of America, 2005, 102(46): 16569-16572.
37 Roldan-Valadez E, Salazar-Ruiz S Y, Ibarra-Contreras R, et al. Current concepts on bibliometrics: a brief review about impact factor, Eigenfactor score, CiteScore, SCImago Journal Rank, Source-Normalised Impact per Paper, H-index, and alternative metrics[J]. Irish Journal of Medical Science, 2019, 188(3): 939-951.
38 Waltman L, van Eck N J. The inconsistency of the h-index[J]. Journal of the American Society for Information Science and Technology, 2012, 63(2): 406-415.
39 Bornmann L, Tekles A. Convergent validity of several indicators measuring disruptiveness with milestone assignments to physics papers by experts[J]. Journal of Informetrics, 2021, 15(3): 101159.
40 Herrmannova D, Patton R M, Knoth P, et al. Do citations and readership identify seminal publications?[J]. Scientometrics, 2018, 115(1): 239-262.
41 Cagan R. The San Francisco Declaration on research assessment[J]. Disease Models & Mechanisms, 2013, 6(4): 869-870.
42 Hicks D, Wouters P, Waltman L, et al. Bibliometrics: the Leiden Manifesto for research metrics[J]. Nature, 2015, 520(7548): 429-431.
43 Rosenthal R. The file drawer problem and tolerance for null results[J]. Psychological Bulletin, 1979, 86(3): 638-641.
44 Jia T, Wang D S, Szymanski B K. Quantifying patterns of research-interest evolution[J]. Nature Human Behaviour, 2017, 1(4): Article No.0078.
45 Wang D S, Uzzi B. Weak ties, failed tries, and success[J]. Science, 2022, 377(6612): 1256-1258.
46 Rzhetsky A, Foster J G, Foster I T, et al. Choosing experiments to accelerate collective discovery[J]. Proceedings of the National Academy of Sciences of the United States of America, 2015, 112(47): 14569-14574.
47 俞立平, 张矿伟, 蒋长兵. 推进代表作评价存在的问题与对策研究[J]. 情报学报, 2021, 40(4): 345-353.
48 Li D, Agha L. Big names or big ideas: do peer-review panels select the best science proposals?[J]. Science, 2015, 348(6233): 434-438.
49 Ruan X M, Lyu D Q, Gong K L, et al. Rethinking the disruption index as a measure of scientific and technological advances[J]. Technological Forecasting and Social Change, 2021, 172: 121071.
50 Bornmann L, Devarakonda S, Tekles A, et al. Are disruption index indicators convergently valid? The comparison of several indicator variants with assessments by peers[J]. Quantitative Science Studies, 2020, 1(3): 1242-1259.
51 Leydesdorff L, Tekles A, Bornmann L. A proposal to revise the disruption index[J]. Profesional de la Información, 2021, 20(1): e300121.
52 Lyu D Q, Gong K L, Ruan X M, et al. Does research collaboration influence the “disruption” of articles? Evidence from neurosciences[J]. Scientometrics, 2021, 126(1): 287-303.
53 Liang G Q, Lou Y, Hou H Y. Revisiting the disruptive index: evidence from the Nobel Prize-winning articles[J]. Scientometrics, 2022, 127(10): 5721-5730.
54 Leydesdorff L, Bornmann L. Disruption indices and their calculation using web-of-science data: indicators of historical developments or evolutionary dynamics?[J]. Journal of Informetrics, 2021, 15(4): 101219.
55 Chen J Y, Shao D A, Fan S K. Destabilization and consolidation: conceptualizing, measuring, and validating the dual characteristics of technology[J]. Research Policy, 2021, 50(1): 104115.
56 Popper K. The logic of scientific discovery[M]. New York: Harper and Row, 1934.
57 Liu L, Wang Y, Sinatra R, et al. Hot streaks in artistic, cultural, and scientific careers[J]. Nature, 2018, 559(7714): 396-399.
58 Sinatra R, Wang D S, Deville P, et al. Quantifying the evolution of individual scientific impact[J]. Science, 2016, 354(6312): aaf5239.
59 Wang J. Citation time window choice for research impact evaluation[J]. Scientometrics, 2013, 94(3): 851-872.
60 Wang D S, Song C M, Barabási A L. Quantifying long-term scientific impact[J]. Science, 2013, 342(6154): 127-132.
61 Nacher J C, Hayashida M, Akutsu T. Emergence of scale-free distribution in protein-protein interaction networks based on random selection of interacting domain pairs[J]. Biosystems, 2009, 95(2): 155-159.
62 Wolcott H N, Fouch M J, Hsu E R, et al. Modeling time-dependent and-independent indicators to facilitate identification of breakthrough research papers[J]. Scientometrics, 2016, 107(2): 807-817.
63 Lü L Y, Zhou T, Zhang Q M, et al. The h-index of a network node and its relation to degree and coreness[J]. Nature Communications, 2016, 7: Article No.10168.
64 Petersen A M, Fortunato S, Pan R K, et al. Reputation and impact in academic careers[J]. Proceedings of the National Academy of Sciences of the United States of America, 2014, 111(43): 15316-15321.
65 Xu S Q, Mariani M S, Lü L Y, et al. Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data[J]. Journal of Informetrics, 2020, 14(1): 101005.
66 Mariani M S, Medo M, Zhang Y C. Identification of milestone papers through time-balanced network centrality[J]. Journal of Informetrics, 2016, 10(4): 1207-1223.
67 Wang J J, Xu S Q, Mariani M S, et al. The local structure of citation networks uncovers expert-selected milestone papers[J]. Journal of Informetrics, 2021, 15(4): 101220.
68 Way S F, Morgan A C, Larremore D B, et al. Productivity, prominence, and the effects of academic environment[J]. Proceedings of the National Academy of Sciences of the United States of America, 2019, 116(22): 10729-10733. 责任编辑 魏瑞斌） |
|
|
|
