Reviewers are Blinkered by Bibliometrics
"Science panels still rely on poor proxies to judge quality and impact.
That results in risk-averse research, say Paula Stephan, Reinhilde
Veugelers and Jian Wang.
There is a disconnect between the research that reviewers purport to
admire and the research that they actually support. As participants on
multiple review panels and scientific councils, we have heard many
lament researchers' reluctance to take risks. Yet we've seen the same
panels eschew risk and rely on bibliometric indicators for assessments,
despite widespread agreement that they are imperfect measures.
Although journal impact factors (JIFs) were developed to assess journals
and say little about any individual paper, reviewers routinely justify
their evaluations on the basis of where candidates have published. Panel
members judge applicants by Google Scholar results and use citation
counts to score proposals for new research. This practice prevails even
at agencies such as the European Research Council (ERC), which instructs
reviewers not to look up bibliometric measures."