Scientometric indices of Russian centers active in oncology/hematology research
Maksim B. Khrustalev, Artem V. Tishkov, Natalia Yu. Turbina, Anna A. Maksimova
Accepted 06 December 2019
The aim of this survey was to perform scientometric evaluation of the major Russian institutions conducting research in the field of oncology and hematology, relying primarily on the common quality indicators and impact of the research results, to аssess their position in the scientific topic clusters (TC) according to international citation databases.
Materials and methods
A comparison was made between five organizations conducting research in the field of oncology and hematology, i.e., four National Medical Research Centers (NMRCs), and Raisa Gorbacheva Memorial Research Institute of Children Oncology, Hematology and Transplantation (R. Gorbacheva Institute) which is a part of I. Pavlov St. Petersburg State Medical University, using the following scientometric indices: citation, collective Hirsch index, as well as the productivity of research determined as relative share of publications in the scientific TC, as well as Field-weighted Citation Impact, using the SciVal platform. The list of publications was limited to keywords defining the field of research of these organizations.
Comparative evaluation of research publication activity in oncohematology has shown the leading position of R. Gorbacheva Institute, as a part of I. Pavlov St. Petersburg State Medical University, which was not included into the NMRCs network. Its rating was only slightly lower than the indices of N. Petrov National Medical Research Center of Oncology and N. Blokhin National Medical Research Center of Oncology. The overall indices of the citation impact based on SciVal analytic platform assessed for R. Gorbacheva Institute are at a level compared to the figures for world publications in the selected topic clusters.
Efficiency of clinical research at an educational institution, evaluated as the number of highly cited publications proved to be not lower, but sometimes even higher than in specialized research institutions working in the field. Appropriate publishing activity evaluated by the SciVal system showed that the funding authorities providing research financiation should recognize R. Gorbacheva Memorial Research Institute of Pediatric Oncology, Hematology and Transplantation at the Pavlov University as a perspective university-based platform for research in relevant areas, along with existing specialized scientific institutions.
Hematology, oncology, national medical research centers, medical university, scientometric indexes, bibliometry, citation analysis.
Over recent years, scientometric criteria have been increasingly used, in order to evaluate the effectiveness of scientific institutions, enabling us to present the fact-based characteristics of the institutions, and their impact on development of world science, if used in correct manner. To some degree, such an evaluation is important for monitoring the quality of both individual and team work, as well as efficient use of resources allocated for research.
Similarly, scientometric methods are increasingly used to answer the questions of preferential financial support for certain areas of research, and discern the priority products and technologies, as well as to resolve problems with research staff recruitment .
Current science management is largely focused on accountability. Therefore, quantitative evaluation of research activities sufficiently adds to qualitative assessments of research (i.e., peer reviewing of projects and articles). Hence, bibliometric evaluation with its measurements of research output and citation impact seems to be a quite important quantitative approach .
The Strategy for the Development of Medical Science is one of the key documents that determines the development trends of medical science in Russia. According to it, the main goal is to create high-tech innovative products ensuring the improvement of public health based on the transfer of innovative technologies to practical healthcare. To achieve this goal, it was necessary to arrange a network of leading centers for the prioritized areas .
To arrange such a system and provide methodological support of research projects for the leading institutions, a Decree №125 was issued by the Ministry of Healthcare of Russian Federation (March 21, 2017) entitled: "On arranging measures for establishment of network of National Research and Practical Medical Centers". Its implementation started by creating a network of National Medical Research Centers (NMRCs) on the basis of specialized research institutions subordinated to the Ministry of Healthcare of the Russian Federation that are leaders in the top areas of medical science. At the same time, the educational priorities in medical research were assigned to specialized research centers, but not to appropriate educational institutions (e.g., medical universities), being in controbersy with global trends in the basic science development. Thus, a network of 22 NMRCs was arranged [4, 5], involving the four research centers that conduct research in the field of oncology and hematology.
Hence, along with the National Research Center for Hematology, this study also included other major Russian centers active in clinical and experimental hematology, i.e., N. N. Blokhin National Medical Research Center of Oncology, Dmitry Rogachev National Medical Research Center of Pediatric Hematology, Oncology and Immunology, N. N. Petrov National Medical Research Center of Oncology, and Raisa Gorbacheva Memorial Research Institute of Pediatric Oncology, Hematology and Transplantation which is a part the St. Petersburg I. Pavlov Medical University.
Among the scientific indices of the institutions that are widely used in the reports, there is a clear bias towards quantitative characteristics, such as total number of articles, the number of articles indexed in the Web of Science (WoS) and Scopus international databases, total impact factor of the journals in which the articles were published, etc. At the same time, the indices characterizing quality of published works (citation scores) and, indirectly, the reputation of the institution (Hirsch index of the organization) are practically not used. Nevertheless, the Hirsch index is most difficult to manipulate by researchers and management staff at the institutions, and, therefore, it can be more objective when comparing scientific reputation characteristics .
The aim of our work was to perform a scientometric evaluation of these organizations, relying primarily on their quality indicators and impact of the research results. Moreover, our task was to determine the position of R. Gorbacheva Memorial Research Institute of Pediatric Oncology, Hematology and Transplantation (R. Gorbacheva Institute), a large university research center which is not an autonomous legal entity, and which was not put on the list of NMRCs with special support for research activities.
Materials and methods
Our study included five major institutions active in Hematology/Oncology over last 3 years (2016-2018), i.e., N. N. Petrov National Medical Research Center of Oncology (St. Petersburg), National Medical Research Center of Hematology (Moscow), N. Blokhin National Medical Research Center of Oncology (Moscow), Dmitry Rogachev National Medical Research Center of Pediatric Hematology, Oncology and Immunology (Moscow), and Raisa Gorbacheva Memorial Research Institute of Pediatric Oncology, Hematology and Transplantation at the I. Pavlov First St. Petersburg State Medical Unibersity.
For the convenience of this study, we concentrated precisely on the qualitative and reputation indices of the work of organizations for the period of 2016 to 2018. WoS and Scopus databases were used as a data source, which allowed filtering publications in journals that did not receive worldwide recognition. Publications marked "articles" were taken into account. This is especially important, since a Department of Hematology at the Pavlov University was also included into the study, and so without the filtering, publications by students and postgraduates occupied at the University, in general, could give a distorted characteristic of publications from the local researchers . The list of publications was limited to keywords defining the field of research (Table 1), thus allowing to filter out most of the publications, first of all, by other branches of the I. Pavlov St. Petersburg Medical University that were not related to the specific area of Hematology/Oncology/Transplantation. Hence, the analysis was limited by the scientific publications from R. Gorbacheva Memorial Research Institute of Pediatric Oncology, Hematology and Transplantation, and the I. Pavlov University Department of Hematology, Transfusiology and Transplantation.
To evaluate the research activities of major haematological centers in Russia, we have calculated the widely used scientometric indicators, e.g., the citation index, which can be calculated for publications and also for organizations and individual researchers or research teams.
However, these commonly used indices have a significant drawback, i.e., they vary greatly for different areas, which can lead to distortions and inaccurate data. There are various approaches to the allocation of scientific fields for calculating the weighted average indicators in each of them.
Hence, to normalize indicators in the WoS system, classification is used based on the topics of the journals included in indexing. But this approach has a number of significant drawbacks, the main one of which is the inadequate processing of publications in interdisciplinary journals and other journals with a wide coverage. Therefore, more accurate, it seems the selection of areas based on individual publications, and not on the level of entire journals .
To perform the statistical evaluation, we used MS Excel software for plotting the diagrams. The ScVal database concerning the specialized St. Petersburg and Moscow centers were kindly provided by Dr. Mark A. Akoev (Laboratory of Scientometry, B. Yeltsin Ural Federal University, Yekaterinburg, Russia).
Comparison of the characteristics of organizations was done by the number of articles and citations for the period 2016-2018, and by the Hirsch index, calculated on the basis of their publications. The results are presented in Table 2 (according to WoS data) and in Table 3 (according to Scopus data).
Table 2. Common scientometric indices of the four major NMRCs active in Hematology/Oncology, and Raisa Gorbacheva Memorial Research Institute, according to the Web of Science data
N. N. Blokhin National Medical Research Center of Oncology is a leader by the number of WoS-indexed publications (2798 papers). However, the average number of citations per publication proved to be the highest for N. N. Petrov NMRC of Oncology (7.8). For N.N. Blokhin NMRC of Oncology, this figure is almost the same as for R. Gorbacheva Memorial Research Institute, i.e. (6.98 and 6.5, respectively). At the same time, National Research Center for Hematology, ranking second in the number of publications, is significantly behind other surveyed institutions in terms of citation, with average citation number only 1.5 per an article. The same conclusions can be drawn by evaluating Hirsch index of these institutions (Table 3, Fig. 1). The N. Blokhin National Medical Research Center of Oncology (22), N. Petrov National Medical Research Center of Oncology (17) and R. Gorbacheva Memorial Research Institute (10) are in the top three for this index, and the National Research Center for Hematology has a lower Hirsch index level (5). Thus, on the basis of WoS-derived data, the N. N. Blokhin National Medical Research Center of Oncology is the leader in publishing activity in Oncology/Hematology among the studied organizations, and R. Gorbacheva Memorial Research Institute holds a middle position among the research centers working in this field.
Similar results are obtained when evaluating the publications of these institutions according to Scopus data. Here, the same three leading organizations are determined by the Hirsch index, and R. Gorbacheva Memorial Research Institute proved to be a leader in the number of publications referred in Scopus (890 publications over three years).
Table 3. Common scientometric indices of four NMRCs and R. Gorbacheva Memorial Research Institute of Pediatric Oncology, Hematology and Transplantation,according to the Scopus data
Figure 1. Hirsch Index of the four Russian NMRCs (see Table 3), and the R. Gorbacheva Institute (I. Pavlov University), as based on Scopus data
A more profound study on the contribution of these organizations was done using the Elsevier SciVal online platform, which is based on the Scopus database, thus allowing monitoring and analysis of international scientific research using visualization tools and modern metrics for citations, economic and social efficiency. Each scientific topic in the Scopus database is a collection of documents united by a common research interest, grouped together in a SciVal cluster, as based on analysis of direct citations in the lists of document links (a document can have only one topic). While indexing, the newly published documents are added to the relevant topics, as based on the lists of appropriate links. Thus, the clusters reflect a narrow scientific direction, characterized by a set of keywords (e.g., in oncology/hematology), as well as the relationship between authors and cross-citations. As for 2017, 91726 clusters were allocated in SciVal . The SciVal platform provides access to research results from more than 14.000 research institutions from 230 countries. Such wide service coverage allows us to evaluate and compare the research results of similar scientific organizations around the world.
The assessment was done according to the following indicators:
1. Scientific Products (Scholarly Output) – the total number of published research results. SO is an indicator that determines the productivity of scientific work. The following publications are included: journal article; chapter or article in the book; books (monographs, textbooks and reference books); software; report. Excludes: patents, dissertations.
2. The share of publications in the cluster (Publication share, %). A comparison of research results was carried out according to the metrics of the TC.307 cluster, which can be described by the following keywords: Hematopoietic Stem Cell Transplantation; Graft vs Host Disease; Transplants.
3. Field-Weighted Citation Impact is a science-normalized citation rate . Calculated by SciVal, this indicator is equal to the ratio of the number of links received by researchers' publications to the average number of links received by all other similar publications indexed in the Scopus database.
The Field-Weighted Citation Impact of 1.00 indicates that publications cited on average for similar publications issued worldwide.
Field-Weighted Citation Impact higher than 1.00 indicates that publications were cited more often than might be expected based on global average for similar publications. For example, a score of 1.44 means that the results were cited 44% more often than expected.
Field-Weighted Citation Impact less than 1.00 indicates that publications were cited less than would be expected based on the world average for similar publications. For example, a rating of 0.85 means 15% less cited than the world average.
Similar publications are publications in the Scopus database that have the same year of publication, type of publication, and discipline.
Field-Weighted Citation Impact refers to citations received in the year of publication plus the next 3 years.
This indicator is useful for evaluating publications regardless of their differences in size, disciplinary profile, age and type of publication, as well as for assessing the level of citation of a researcher.
Figure. 2. Distribution of publications of R. Gorbacheva Memorial
Research Institute of Pediatric Oncology, Hematology and
Transplantation (Pavlov University) by the SciVal topic clusters (TCs)
Abscissa, number of publications; Ordinate shows the publication share which is the 5-year publication output of the given institution divided by the publication output from the institution ranked #1 worldwide within a particular competency. Publication clusters mean the groups of highly cited publications and the current publications that cite them. Area of the circles reflects the field-weighted citation impact.
The analysis of scientific publications has shown that the performance of R. Gorbacheva Memorial Research Institute in different topic clusters (TC), according to SciVal data, does not correlate with the number of the articles published in the specified TCs. The highest share among high-ranked world publications for R. Gorbacheva Memorial Research Institute corresponds to the TC. 307 (HSCT and adjacent areas), followed by TC.1326 (multilayer films, microparticles), whereas the total number of publications is highest in TC.307 followed by TC.134 (leukemia research), as seen from Table 4 and Fig. 2. On the other hand, we see a correlation between the share among world publications and field-weighted citation impact, thus suggesting that the papers from R. Gorbacheva Memorial Research Institute in TC.307 and TC.1326 are highly shared among the world publications, and also have high field-weighted citation impact. In general, the indices of scientific works from R. Gorbacheva Memorial Research Institute based on SciVal data are at comparable level with the bulk of world publications in the selected topic clusters (TCs).
Table 4. Number of publications by the R. Gorbacheva Memorial Research Institute of Pediatric Oncology, Hematology and Transplantation (Pavlov University) in different topic clusters (TC) according to SciVal data
At the present time, there is no unified solution for a comprehensive model of science, or a list of scientific topics (and their relative value). Therefore, in order to assess the scientific significance of an institution, it is necessary to rely both on metrics obtained from various sources and on the results of expert evaluation.
The comparative assessment method we used is based on the clustering of the SCOPUS citation network. Firstly, this allows a more accurate assessment of the development within narrow range of studies for the organizations performing a wide-range research. Secondly, modern scientific trends consider interdisciplinary basis for many new achievements in research. Taking into account the fact that all clusters have a dynamic nature, it does not allow a general description of the work of a distinct organization, even in a specified field. Comparing efficiency of institutions by the metrics of only one cluster of items does not characterize the work of the organization as a whole, but only allows us to describe the degree of development of research in a single narrow area. Therefore, these results do not allow conclusions about the work of specific organizations, but the results will be useful to consider when choosing institutions developing similar scientific projects.
1. Over the past three years, according to Scopus and WoS, there has been an increase in the number of publications in the field of hematology in the major specialized Russian scientific institutions coordinated into a network of National Medical Research Centers (NMRCs), thus indicating advancements and growing relevance of this clinical topic.
2. The use of widespread scientometric indicators, such as numbers of publications, citation, and the Hirsch index, does not allow an unambiguous conclusion about the leadership of scientific organizations, even those conducting research in one narrow area.
3. The efficiency of research in Hematology and related fields at the Raisa Gorbacheva Memorial Research Institute of Pediatric Oncology, Hematology and Transplantation, a part of St. Petersburg State Medical University, may not be lower, and, sometimes, even higher than in specialized research centers, as evidenced by the number of cited works in international databases.
4. A study of publication activity using the SciVal system showed that the research-funding bodies may consider R. Gorbacheva Memorial Research Institute of Pediatric Oncology, Hematology and Transplantation (at the Pavlov University) a perspective university platform for research in relevant areas of Hematology/Oncology, along with existing specialized scientific institutions.
The authors would like to thank Mark A. Akoev (Scientific Laboratory "Laboratory of Scientometry", The B. Yeltsin Ural Federal University, Yekaterinburg, Russia) for his help in selecting materials.
No conflicts of interest reported.
- Ruiz-Castillo J, Waltman L. Field-normalized citation impact indicators using algorithmically constructed classification systems of science. Journal of Informetrics. 2015;9(1):102-117.
- Bornmann L, Wohlrabe K. Normalisation of citation impact in economics. Scientometrics. 2019;120(2):841-884.
- Demina MA. Legal regulation of scientific and innovative activities of medical organizations. Actual Problems of Russian Law. 2018; 11 (96):116-123 (In Russian). DOI: 10.17803/1994-1471.2018.96.11.116-123.
- Decree of the Ministry of Health of Russia (March 21, 2017, N 125) "On the organization of work on the formation of a network of national scientific and practical medical centers." URL: https://rulaws.ru/acts/Prikaz-Minzdrava-Rossii-ot-11.09.2017-N-622/(In Russian).
- Decree of the Ministry of Health of Russia (September 11, 2017, N 622) "On the network of national medical research centers. URL: https://rulaws.ru/acts/Prikaz-Minzdrava-Rossii-ot-11.09.2017-N-622/ (Accessed: 19.10.2019) (In Russian).
- Pavlou C, Elkind E. Manipulating citation indices in a social context. Proc Intern Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS). 2016: 32-40.
- Khrustalev MB, Tishkov AV, Maksimova AA, Turbina NY. Comparing research performance of national medical research centers and medical universities in Russia according to scientific indicators. University Management: Practice and Analysis. 2019; 23(3):108-118 (In Russian).
- Klavans R, Boyack KW. Research portfolio analysis and topic prominence. Journal of Informetrics. 2017;11(4):1158-1174.
- Akoev MA, Markusova VA, Moskaleva OV, Pislyakov VV. Guidelines for scientometry: indicators of the development of science and technology. Publishing House of the Ural University: Yekaterinburg, 2014 (In Russian).