Evaluating the Impact of Academic Primary Care Research Institutions: A Computational Scientometric Methodology

Context: Academic research programs are increasingly making decisions (hiring, promotion, annual reviews/evaluation, grant funding, etc.) on the basis of quantitative key performance indicators (e.g. scientometrics).

Objective: To illustrate how open-source scientific software (e.g. R/Python) can be used to efficiently and cost-effectively mine these scientometric indicators.

Study Design and Analysis: Retrospective cohort. We identify individual researchers on the basis of persistent identification variables, and extract bibliometric information regarding the cohort. We integrate the mined bibliometric database with other data sources (e.g. Altmetric, journal impact, etc.). We curate a comprehensive scientometric database, and illustrate how descriptive statistics can be used to characterize aspects of research output and impact.

Data Sources: Elsevier Scopus database (bibliometrics); SCImago Journal Rank database; and the Altmetric database (social attention).

Population Studied: Primary care researchers from the University of Toronto Department of Family and Community Medicine (DFCM); as well as researchers from other leading academic primary care research institutions.

Outcomes Measured: Publications, citations, social attention and altmetrics, metrics of research collaboration, and thematic domains of research activity.

Results: The University of Toronto DFCM ranked highly with respect to scientometric measures studied. Future work is needed to better understand organization structure at comparative institutions, and frame the responsible use of scientometric indicators in primary care research institutions.

Conclusions: The proposed scientometric methodology offers a passive, open/transparent, scalable, and cost-effective mechanism for generating performance indicators related to research outputs (at the level of individual publications, researchers, or institutions).