Bibliometric analysis and peer review in research performance assessment : a comparative study of New Zealand academic psychology research

Abstract

In 2003 a Performance-Based Research Fund (PBRF) was introduced in New Zealand, followed by the first quality evaluation of tertiary institutions. The PBRF has been the subject of criticism for being unfair, too costly to implement, disadvantaging emerging researchers, and discouraging scholarly collaboration. This study aims to investigate the correlation between various bibliometric indicators and the 2003 Quality Evaluation ratings in order to understand how bibliometric analysis can be used to complement and counterbalance inherent weaknesses in peer review-based PBRF Quality Evaluation. The overall goal is to establish whether bibliometric analysis can be a useful complementary tool within the current PBRF framework improving its validity and accuracy while decreasing its cost and eliminating its pitfalls. Advanced bibliometric methods developed by the Dutch Centre for Science and Technology Studies (CWTS) and the Australian Research Evaluation and Policy Project (REPP) were used in the research. The results of this bibliometric study of New Zealand academic psychology research indicate that bibliometric indicators can be used to facilitate and cross-validate the PBRF Quality Evaluation as well as to complement it by measuring those aspects of the research performance that are not reflected in the Quality Evaluation (e.g. the level of scholarly collaboration). Bibliometric analysis can also provide essential tools for measuring the impact of the PBRF Quality Evaluation on New Zealand psychology research. The information presented in this research is intended to open discussion on how best to evaluate research performance in the field of psychology. It is not intended to endorse any measure as superior to others, though it is argued that peer review would be greatly enhanced by the availability of bibliometrical data. The broad range of bibliometric indicators that allows measurement of different aspects of university performance is recommended.