Document Detail

Surveys of current status in biomedical science grant review: funding organisations' and grant reviewers' perspectives.
Jump to Full Text
MedLine Citation:
PMID:  20961441     Owner:  NLM     Status:  MEDLINE    
Abstract/OtherAbstract:
BACKGROUND: The objectives of this research were (a) to describe the current status of grant review for biomedical projects and programmes from the perspectives of international funding organisations and grant reviewers, and (b) to explore funders' interest in developing uniform requirements for grant review aimed at making the processes and practices of grant review more consistent, transparent, and user friendly.
METHODS: A survey to a convenience sample of 57 international public and private organisations that give grants for biomedical research was conducted. Nine participating organisations then emailed a random sample of their external reviewers an invitation to participate in a second electronic survey.
RESULTS: A total of 28 of 57 (49%) organisations in 19 countries responded. Organisations reported these problems as frequent or very frequent: declined review requests (16), late reports (10), administrative burden (7), difficulty finding new reviewers (4), and reviewers not following guidelines (4). The administrative burden of the process was reported to have increased over the past 5 years. In all, 17 organisations supported the idea of uniform requirements for conducting grant review and for formatting grant proposals. A total of 258/418 (62%) reviewers responded from 22 countries. Of those, 48% (123/258) said their institutions encouraged grant review, yet only 7% (17/258) were given protected time and 74% (192/258) received no academic recognition for this. Reviewers rated these factors as extremely or very important in deciding to review proposals: 51% (131/258) desire to support external fairness, 47% (120/258) professional duty, 46% (118/258) relevance of the proposal's topic, 43% (110/258) wanting to keep up to date, 40% (104/258) desire to avoid suppression of innovation. Only 16% (42/258) reported that guidance from funders was very clear. In all, 85% (220/258) had not been trained in grant review and 64% (166/258) wanted this.
CONCLUSIONS: Funders reported a growing workload of biomedical proposals that is getting harder to peer review. Just under half of grant reviewers take part for the good of science and professional development, but many report lack of academic and practical support and clear guidance. Around two-thirds of funders supported the development of uniform requirements for the format and peer review of proposals to help ease the current situation.
Authors:
Sara Schroter; Trish Groves; Liselotte Højgaard
Publication Detail:
Type:  Journal Article     Date:  2010-10-20
Journal Detail:
Title:  BMC medicine     Volume:  8     ISSN:  1741-7015     ISO Abbreviation:  BMC Med     Publication Date:  2010  
Date Detail:
Created Date:  2010-11-08     Completed Date:  2011-02-16     Revised Date:  2013-07-03    
Medline Journal Info:
Nlm Unique ID:  101190723     Medline TA:  BMC Med     Country:  England    
Other Details:
Languages:  eng     Pagination:  62     Citation Subset:  IM    
Affiliation:
BMJ Editorial, BMA House, London, UK. sschroter@bmj.com
Export Citation:
APA/MLA Format     Download EndNote     Download BibTex
MeSH Terms
Descriptor/Qualifier:
Biomedical Research / economics*
Efficiency, Organizational
Financing, Organized
International Agencies / economics,  ethics
Peer Review, Research / ethics*,  methods*
Research Support as Topic
Workload
Comments/Corrections

From MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine

Full Text
Journal Information
Journal ID (nlm-ta): BMC Med
ISSN: 1741-7015
Publisher: BioMed Central
Article Information
Copyright ©2010 Schroter et al; licensee BioMed Central Ltd.
open-access:
Received Day: 2 Month: 7 Year: 2010
Accepted Day: 20 Month: 10 Year: 2010
collection publication date: Year: 2010
Electronic publication date: Day: 20 Month: 10 Year: 2010
Volume: 8First Page: 62 Last Page: 62
Publisher Id: 1741-7015-8-62
PubMed Id: 20961441
DOI: 10.1186/1741-7015-8-62

Surveys of current status in biomedical science grant review: funding organisations' and grant reviewers' perspectives
Sara Schroter1 Email: sschroter@bmj.com
Trish Groves1 Email: tgroves@bmj.com
Liselotte Højgaard2 Email: lottepet@rh.dk
1BMJ Editorial, BMA House, London, UK
2Clinical Physiology, Nuclear Medicine & PET, Rigshospitalet, University of Copenhagen, Denmark and ESF, Strasbourg, France

Background

Peer review for submissions to scientific journals has developed over more than 300 years, and there is now a considerable body of evidence on its methods, outcomes, effectiveness, best practice, problems, and ethics [1]. Peer review of applications for research grants has a sparser evidence base and fewer examples of good practice for funders to draw on when developing their systems. This lack of evidence is worrying, because several major funders of biomedical research have reported recently that they are becoming overburdened by workloads and by the complexity and slowness of grant review processes [2,3].

In contrast, editorial peer review at journals has a much wider evidence base and is subject to continuous review and debate. Strengths, weaknesses, ways to reduce bias, and different models in journal peer review have been studied extensively, yet many questions remain unanswered. Although there is evidence on how to conduct peer review fairly and efficiently, there is only limited evidence that journal peer review improves the quality of published biomedical science according to a Cochrane review that systematically reviewed 28 of 61 retrieved studies [4]. International congresses have been held regularly since 1989 to present and share the latest evidence and to develop further research questions, so that biomedical journal editors can try to improve their practices [5]. Additionally, several international organisations gather, further develop, and promote evidence to guide editors; particularly the International Committee of Medical Journal Editors (ICMJE) whose Uniform Requirements for Manuscripts Submitted to Biomedical Journals guidelines ensure the consistency and transparency of manuscripts and guide practice, processes and policies at more than 800 journals [6]. There is, however, no equivalent resource for preparing grants and conducting grant review.

There has been one Cochrane systematic review about grant review practices, but it found evidence only on biases and other process weaknesses and included only 10 studies [7]. The authors concluded 'We were unable to find comparative studies assessing the actual effect of peer review procedures on the quality of the funded research. There is little empirical evidence on the effects of grant giving peer review. Experimental studies assessing the effects of grant giving peer review on importance, relevance, usefulness, soundness of methods, soundness of ethics, completeness and accuracy of funded research are urgently needed. Practices aimed to control and evaluate the potentially negative effects of peer review should be implemented meanwhile' [7]. Until the necessary studies of grant review are done, might funders develop and use a set of uniform requirements, at least partly informed by the evidence on journal peer review?

Ernest Starling was an eminent physiologist who when asked by the British Medical Research Council in the 1920s about how to best distribute funding answered 'get the best of men, give them the equipment you can afford, and leave them alone'. Peer review is the main 'equipment' used by research councils and other funders. To explore the current status of grant review, the eponymous 'Starling Group' of funders, policy makers, researchers, and editors, who first met in Frankfurt to discuss the European Medical Research Councils' strategy for medical research in Europe [3], initiated two surveys.

The first Starling Group study is an international survey to describe the current status of peer review among biomedical funding organisations and the problems they face when evaluating proposals for biomedical project and programme grants. The second is a survey to determine the workload of external grant reviewers, the level of institutional support for this activity, reviewers' motivations and perceived barriers to taking on grant review, and their views on possible solutions. We report on both surveys here.


Methods
Survey of biomedical funding organisations
Sample

We took a convenience sample of biomedical research funding organisations across Europe and also approached key national funders from North America, Australia, and New Zealand to broaden the survey's reach and relevance; members of the Starling Group suggested international public and private grant giving organisations they thought should be included. We also sought the advice of the European Foundation Centre for suggestions for inclusions of private foundations in Europe. The final list of 57 funding organisations included both small and large international funding organisations. The purpose was to include a range of different funders from different countries to illustrate some of the current problems they face rather than to create a representative sample and draw inferences beyond the sample.

Procedures

A draft questionnaire was developed based on discussions with several funding organisations about current practice and common problems with peer review. The questionnaire was then refined by members of the Starling Group and revised before field testing. The questionnaire asked participants to respond to all questions in relation to grants that provide support for biomedical studies addressing a single research question or research theme (project and programme grants) not infrastructure grants or fellowships. We sent an invitation and link to the electronic survey on SurveyMonkey (online survey software; http://www.SurveyMonkey.com) to a named contact person at the funding organisation explaining the purpose of the research and requesting their help. Non-responders were sent an email reminder 2 weeks and 6 weeks after the original mailing.

Survey of external grant reviewers
Sample

Funding organisations that took part in our first survey were invited to survey a random sample of approximately 50 of their reviewers. Nine organisations agreed to take part and emailed invitations to our survey to a small sample of their external reviewers.

Procedures

A draft questionnaire was developed based on known problems with grant peer review and previous research with journal peer reviewers [3]. The draft questionnaire was then finalised by members of the Starling Group. Reviewers were sent an email from their funder informing them of the survey with a link to the questionnaire on SurveyMonkey. Reviewers were informed that the survey was being conducted for research purposes by a team interested in improving grant review processes and that their responses would be confidential. All reviewers were emailed a reminder by their funding organisation to complete the survey approximately 2 weeks after the initial mailing.


Results
Survey of biomedical funding organisations
Sample characteristics

We received a response from 29 (52%) of the 56 funders. Table 1 is a list of participating organisations. Organisations were based in 19 countries, were funded by various sources, and varied in size and the amount of grant money administered. Six large organisations took part. These receive over 1000 proposals a year, and were: the US National Institutes of Health, the Canadian Institutes of Health Research, The Wellcome Trust, the National Health And Medical Research Council (of Australia), the Deutsche Forschungsgemeinschaft (DFG), and the Medical Research Council, UK. Of the 29 responding organisations, 2 do not use external reviewers for project applications but use committees instead (The Swedish Research Council and The Lundbeck Foundation) and for this reason, they were excluded from the analysis of specific questions regarding external reviewers.

Peer review process

A total of 15/29 of the organisations reject 10% or less of research project grant applications based on internal review only. However, 13/29 of the organisations accept 30% or less of project grant applications following external review. A total of 17/27 use an electronic tracking system to contact external reviewers and manage their reviews but few (6/29) use an electronic system for grant applicants to track their proposals during each stage of the decision-making process.

Selection of external reviewers

Only 13/27 organisations limit the number of annual requests to external reviewers in order to reduce individual burden and the potential for bias in the system. However, 18/27 organisations invite more than 3 external reviewers on average to review a single application. In all, 16/27 use programme managers, 15/27 boards/panels/committees, 6/27 board chairpersons, and 10/27 grant applicants to suggest external reviewers.

External reviewer guidelines and forms

The majority (22/27) provide guidelines for external reviewers on what applications should be judged on, 22/27 provide review forms or templates for external reviewers to submit their reviews, and 21/27 a scoring/ranking system to rate specific aspects of proposals.

Transparency of review process

Only 2/27 organisations hide the grant applicants' names from the external reviewers whereas 21/27 hide external reviewers' names from applicants. Only 4/27 hide the names of all reviewers on the funding board/panel from applicants. However, 5/27 organisations commented that they make a list of reviewers' names available (for example, on their website) at the end of the funding round.

During the review process, 7/27 organisations allow grant applicants to see the full external reviewers' reports, 3/27 the funding board/panels' comments/reports, and 5/27 the scores assigned to their application. However, after the decision has been made on their application, these figures rose to 16/27, 18/27, and 13/27, respectively. A total of 22/27 organisations routinely ask their reviewers to declare their conflicts of interest for each proposal reviewed.

Frequency of problems with specific aspects of peer review

Table 2 shows the frequency organisations reported they experience problems with specific of aspects of peer review. Problems reported as very frequent or frequent by at least a quarter of the organisations included reviewers declining to review (16/29) and receiving late reports (10/29). None of the organisations reported very frequent or frequent problems with having an inadequate number of reviewers' reports available at time of assessment, reviewers not declaring their conflicts of interest, or reviewers breaking confidentiality.

Perceived change in specific aspects of peer review over the last 5 years

Over half the sample perceived no change over the last 5 years in the following problems: having an inadequate number of reviewers' reports available at time of assessment (15/29), receiving poor quality reviews (19/29), receiving late reviews (19/29), reviewers not following guidelines appropriately (20/29), reviewers not declaring their conflicts of interest (16/29), reviewers breaking confidentiality (16/29), applicants questioning the conflicts of interest of reviewers (18/29), and applicants questioning the funder's choice of reviewers (20/29) (Table 2).

Many organisations reported the situation was worse now than 5 years ago for reviewers declining to review (13/29), too many applications in the system (12/29), and the administrative burden of process (11/29). Funders reported a better situation now than 5 years ago for applicants recommending inappropriate reviewers (18/29) reviewers declaring their conflicts of interest.

Most important challenges facing funding organisations

The self-reported most important challenges faced by organisations included finding available and suitable reviewers, problems with review quality and time taken to complete reviews, and problems with administration and transparency.

Reviewer incentives: feedback, acknowledgement, and rewards

In all, 14/27 organisations routinely give external peer reviewers the funding board/panel's decision on the proposal(s) they reviewed, but only 3/27 the details of their discussions and decisions. Only about a quarter (7/27) give feedback to reviewers on the usefulness of their reviews.

Some organisations reward or acknowledge their reviewers by naming them on their website (7/27), telling their institutions that they are reviewers for their organisation (4/27), giving feedback on the quality of their reviews (5/27), and by paying them (10/27). Other ways of thanking them included: naming them in the annual report, annual letters of thanks, informing those that give the most useful reports that they are the best, honorariums, giving formal confirmation if requested, invitations to the grant delivery ceremony or inauguration dinners, emails of gratitude and feedback on reviews if requested, allowing reviewers to submit grant applications outside of call deadlines, reviewers' award programs, and dinner invitations.

Development of standards on grant review

We proposed that a set of standards, such as the ICMJE guidelines [4] for the preparation and formatting of manuscripts submitted for publication by biomedical journals, drawn up for grant review might help enable researchers, funders, and peer reviewers to practice grant review more efficiently and effectively. We found considerable support for the idea of the development of a set of standards for the peer review process at grant giving organisations: 17/29 would support this idea, none would not, and 11/29 were unsure. There was also good support for the development of a set of standards for the grant application process: 17/29 would support it, 2/29 would not, and 9/29 were unsure.

Survey of external grant reviewers
Sample characteristics

In all, 9 funding organisations took part and we received an overall response of 258/418 (62%) (Table 3). Two-thirds of respondents were male, and the majority (62%) was aged 41 to 60 years. In all, 77% described themselves as researchers, 31% practicing clinicians, and 17% laboratory scientists. Respondents were working in 22 countries and ranged in their experience of reviewing: 24% had been a grant reviewer for biomedical science proposals for ≤3 years and 47% for at least 10 years. Many respondents were reviewers for several funding organisation; 44% (114/258) had reviewed for 3 or more organisations in the past 12 months.

Time to review

Reviewers were asked to indicate approximately how long it takes them to review a single biomedical science grant proposal (including time reading proposal, making notes, writing comments, completing assessment sheets, attending committees and panels). Only 15% (39/258) said that the whole process took them less than or equal to 3 hours; 30% (77/258) spend on average between 4 and 6 hours, 9% (22/258) spend 7 to 9 hours, 17% (44/258) spend 10 to 24 hours and 10% (27/258) spend more than 24 hour.

Institutional support

A total of 48% (123/258) said their institution or managers encouraged them to take part in biomedical science grant review, yet only 14% (37/258) said their institution or managers knew how much time they spent reviewing and 31% (79/258) knew which funding organisations they reviewed for. A total of 32% (82/258) were expected to review grants in their own time (that is, out of office hours) and only 7% (17/258) were given protected time to conduct grant review. In all, 28% (73/258) said they always conduct biomedical science grant review in their own time, 44% (113/258) often do, 19% (49/258) occasionally and only 1% (2/258) never do.

Academic recognition

A total of 74% (192/258) do not receive any academic recognition for conducting grant review. Comments from the 43 who said they did included the fact that it contributes to promotional review, is recognised within the 'indicators of esteem' element of the UK's Research Assessment Exercise, is one of the metrics for assessing research portfolios in clinical departmental review, is a recognised research service when applying for grants and fellowships, and prestige.

Importance of specific factors in decision to do grant review

Reviewers rated the following as extremely or very important in their decision to review: 51% (131/258) to help external fairness in decision taking by review committees, 47% (120/258) sense of professional duty, 46% (118/258) relevance of the topic, 43% (110/258) wanting to keep up to date on research advances, 40% (104/258) to help ensure innovation is not suppressed (Table 4). In all, 30% (77/258) said the most important motivating factor was a sense of professional duty, 14% (35/258) the opportunity to learn something new, 14% (35/258) to keep up to date on research advances, and 12% (31/258) wanting to help external fairness in decision taking.

Training and guidance in grant review

Only 9% (22/258) had received some formal training in how to conduct biomedical science grant review and 64% (166/258) said they would be interested in receiving training if funding organisations provided it (free of charge). Whilst 63% (162/258) reported that instructions and guidance for external reviewers provided by biomedical science funding organisations are quite clear and that they usually know what they are expect to do as a reviewer, only 16% (42/258) said that these were very clear and they always know what they are expected to do. Only 9% (23/258) reported that the clarity of instructions and guidance varied by organisation.

Barriers to reviewing

At least 25% of the reviewers reported the following factors often or always acted as barriers to undertaking grant review: conflicts with other workload (47%, 121/258), having to review too many journal articles (36%, 93/258), reviewing taking too much time (33%, 85/258), insufficient knowledge on the focus of the application (31%, 80/258), tight deadlines for completing the review (28%, 71/258), and having to review too many grant proposals for funding organisations (25%, 64/258) (Table 5).

Suggestions on how funding organisations might better identify and retain grant reviewers

We solicited suggestions on how the problem of some funding organisations finding it difficult to both identify new and retain current grant reviewers might be improved. We received numerous constructive suggestions and grouped these by broad themes. Recurring themes included the need for: public acknowledgement, some kind of reward or reimbursement, institutions to recognise the importance of reviewing as an activity, tips for making reviewing easier, more internal reviewing/screening of applications, improved guidance and training, improved feedback and communication, improved administration, greater transparency, funding of more applications, an expanded reviewer pool, tips to actively solicit reviewers, and some suggestions for alternative review systems.


Discussion

Our survey of funding organisations showed that funders are challenged by an increasing number of applications and frequent difficulty in finding willing and available reviewers. Some organisations emphasised the need for streamlining and more efficient administrative systems and we found support for the idea of uniform requirements for proposals. The survey of external reviewers showed that reviewers are motivated by a sense of professional duty and fairness despite not receiving academic recognition and frequently having to undertake this time-consuming work in their own time. Reviewers also indicated that they are often sent requests for reviews outside of their expertise suggesting that funding organisations are having difficulty targeting the appropriate reviewers. We also identified a need for improved guidance and training on how to review. These findings are similar to the problems journals experience with peer review [1].

This international survey included a diverse range of funding organisations and identified common problems. We are unaware of any other studies attempting to collect data from funding organisations with such breadth and geographical reach, although individual organisations, particularly the US National Institutes of Health (NIH) [8], have conducted their own internal reviews. We acknowledge that 29 organisations is still a small number of grant giving organisations, that we used convenience sampling which could have introduced bias, and that the response rate to this survey was low (52%). Similarly, our reviewers' survey is the first large survey of external grant reviewers from a range of funding organisations. We asked reviewers about their reviewing practices in general rather than isolated experiences with specific funders, to make the results more generalisable to the wider population of reviewers and funders. We achieved a good response rate (62%) from the reviewers similar to the response rate we have achieved with surveys of journal peer reviewers [9].

The ICMJE set an editorial precedent for the development of uniform requirements. A similar set of guidelines on grant review might enable researchers, funders, and peer reviewers to practise grant review more consistently and, we hope, more efficiently and effectively. Most of the funding organisations in the survey were receptive to the idea of such standards for grant review and reviewers indicated problems with heterogeneous requirements from funders. Further exploration of the feasibility and acceptability of uniform requirements for grant review is required.

With a bursting system reliant on good will, is it time for all funding organisations to more formally recognise contributions from reviewers through, for example, public acknowledgement, certificates, or rewards? Funding organisations should help reviewers to do their job effectively by providing clear guidance and training as well as improved feedback and communication. We are not the first to advocate the need for nurturing reviewers [10], but most have focused on journal peer review and grant review seems relatively neglected.


Conclusions

The workload of biomedical research funders is growing. Our surveys suggest that few funders have used their experiences of deteriorating efficiency or the (albeit limited) evidence base on editorial peer review to assess and improve their processes. We suggest that funders provide clearer guidance to reviewers, draw on evidence from both editorial and grant review to maximise the efficiency and fairness of their work, and come together to consider the development of a set of uniform requirements for submitting and peer reviewing biomedical grant proposals.


Competing interests

SS has no competing interests. TG is a member of Council of the Committee on Publication Ethics, which produces guidance on ethics aspects of peer review and directly advises editors on handling difficult submissions. LH is Chair of EMRC, the European Medical Research Councils, ESF, Strasbourg, EMRC is the membership organisation of the public funders in medical research in Europe. LH is a frequent peer reviewer of grant proposals.


Authors' contributions

SS helped develop the content of the questionnaires, conducted the survey, analysed the results, and wrote the first draft of the manuscript. TG and LH initiated the research, helped develop the content of the questionnaires, and helped write the manuscript. All authors read and approved the final manuscript.


Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1741-7015/8/62/prepub


Acknowledgements

We thank other members of The Starling Group, Dr Anne-Marie Engel (Director of Research, Lundbeck Foundation, Copenhagen), Dr Pamela Reid (Grants Operations Manager, Wellcome Trust, London), Dr Frank Wissing (Programme Director, Deutsche Forschungsgemeinschaft), Professor Håkan Billig (Secretary General of Medicine, Swedish Research Council), and Dr Norka Ruiz Bravo (NIH Deputy Director Extramural Research, National Institutes of Health), for their help with designing the questionnaires. We also thank Mathieu Cantegreil at the European Foundation Centre for encouraging participation from a number of private foundations in Europe, Juliet Walker at the BMJ for administrative help, and of course all the participants for their help with the surveys. This project received no additional external funding.


References
Jefferson T,Godlee F,(Eds)Peer Review in Health SciencesYear: 20032London: BMJ Books
Nationa Institutes for HealthEnhancing peer review at NIHhttp://enhancing-peer-review.nih.gov/index.html
European Medical Research CouncilsPresent Status and Future Strategy for Medical Research in EuropeYear: 2007Strasbourg, France: European Science Foundation
Jefferson T,Rudin M,Brodney Folse S,Davidoff F,Editorial peer review for improving the quality of reports of biomedical studiesCochrane Database Syst RevYear: 200718MR000016
AnonymousGuarding the guardians: research on editorial peer review: selected proceedings from the first international congress on peer review in biomedical publicationJAMAYear: 19902631317144110.1001/jama.263.10.13172304208
International Committee of Medical Journal EditorsUniform Requirements for Manuscripts Submitted to Biomedical Journals: Writing and Editing for Biomedical Publicationhttp://www.ICMJE.org
Demicheli V,Di Pietrantonj C,Peer review for improving the quality of grant applicationsCochrane Database Syst RevYear: 200718MR000003
National Institutes of Health2007-2008 peer review self-study final drafthttp://enhancing-peer-review.nih.gov/meetings/nihpeerreviewreportfinaldraft.pdf
Tite L,Schroter S,Why do reviewers decline to review for journals? A surveyJ Epidemiol Comm HealthYear: 20076191210.1136/jech.2006.049817
Godlee F,Making reviewers visible: openness, accountability, and creditJAMAYear: 20022872762276510.1001/jama.287.21.276212038905

Tables
[TableWrap ID: T1] Table 1 

List of participating funding organisations


Organisation Country Funding basis
National Institutes of Health* (NIH) USA Government

Canadian Institutes of Health Research* Canada Government and small amount of support from donations; additional funds through partnerships with private and public sector agencies

Vetenskapsradet-Medicine (Swedish Research Council) Sweden Government

Fonds voor Wetenschappelijk Onderzoek (FWO) Belgium Private foundation and research council

The Health Foundation UK Independent foundation

Health Research Board of Ireland Ireland Government

The Netherlands Organisation of Health Research and Development (ZonMw), also on behalf of the Netherlands Organisation of Scientific Research (NWO) The Netherlands Government

Swiss National Science Foundation Switzerland Private foundation

NETSCC, part of National Institute for Health Research (NIHR), in England UK Government

Estonian Science Foundation Estonia Government

The Research Council of Norway Norway Government (all the ministries)

Robert Bosch Stiftung Germany Private foundation

Foundation for Polish Science Poland Private foundation

Fondation Fournier Majoie pour l'Innovation Belgium Private foundation

National Institute for Health Research Central Commissioning Facility, which runs four grant funding schemes on behalf of NIHR: Research for Patient Benefit, Invention for Innovation and Research for Innovation, Speculation and Creativity (all project grants) and Programme Grants for Applied Research UK Government

The Wellcome Trust* UK Charity

National Health And Medical Research Council (of Australia)* Australia Government

Fondazione Cariplo Milano Italy Private foundation

Telethon Foundation Italy Italy Charity

Juvenile Diabetes Research Foundation USA Charity

Health Research Council of New Zealand New Zealand Government

Medical Research Council, UK* UK Government

Deutsche Forschungsgemeinschaft* (DFG) Germany Government and private foundation

Hungarian Scientific Research Fund (OTKA) Hungary Government

Academy of Sciences of the Czech Republic. Grant Agency of the Academy of Sciences of the Czech Republic Czech Republic Government

Lundbeck Foundation, Copenhagen Denmark Charity, private foundation, commercial foundation

Fondation Mérieux France Private foundation

Fonds National de la Recherche Scientifique Belgium Private foundation

The Danish Medical Research Council Denmark Government

*Indicates large organisations (receiving over 1000 proposals a year).


[TableWrap ID: T2] Table 2 

Frequency organisations experience problems with specific aspects of peer review and perceived change over time


Frequency of problem, n (%) Perceived change, n (%)

Never Occasionally Frequently Very frequently Not applicable Better situation now than 5 years ago No change Worse situation now than 5 years ago Not applicable
Reviewers declining to review 0 (0) 9 (31) 14 (48) 2 (7) 1 (3) 2 (7) 9 (31) 13 (45) 3 (10)
Difficulty finding new reviewers for your database system 3 (10) 15 (52) 4 (14) 0 (0) 4 (14) 4 (14) 10 (35) 5 (17) 8 (28)
Difficulty retaining good reviewers 4 (14) 17 (59) 1 (3) 0 (0) 3 (10) 3 (10) 14 (48) 4 (14) 5 (17)
Having an inadequate number of reviewers' reports available at time of assessment 2 (7) 23 (79) 0 (0) 0 (0) 1 (3) 4 (14) 15 (52) 3 (10) 5 (17)
Receiving poor quality reviews 1 (3) 22 (76) 3 (10) 0 (0) 0 (0) 5 (17) 19 (66) 0 (0) 3 (10)
Receiving late reviewers' reports 0 (0) 16 (55) 10 (35) 0 (0) 0 (0) 4 (14) 19 (66) 2 (7) 3 (10)
Reviewers not following guidelines appropriately 2 (7) 20 (69) 4 (14) 0 (0) 0 (0) 5 (17) 20 (69) 0 (0) 3 (10)
Reviewers not declaring their conflicts of interest 9 (31) 16 (55) 0 (0) 0 (0) 1 (3) 7 (24) 16 (55) 0 (0) 3 (10)
Reviewers breaking confidentiality 9 (31) 12 (41) 0 (0) 0 (0) 5 (17) 1 (3) 16 (55) 0 (0) 8 (28)
Applicants questioning the conflicts of interest of reviewers 11 (38) 13 (45) 1 (3) 0 (0) 1 (3) 2 (7) 18 (62) 1 (3) 5 (17)
Applicants questioning the funder's choice of reviewers 4 (14) 18 (62) 1 (3) 1 (3) 2 (7) 2 (7) 20 (69) 0 (0) 5 (17)
Applicants recommending inappropriate reviewers 3 (10) 11 (38) 4 (14) 0 (0) 7 (24) 18 (62) 0 (0) 0 (0) 9 (31)
Difficulty in recruiting younger reviewers 9 (31) 8 (28) 2 (7) 1 (3) 6 (21) 6 (21) 11 (38) 2 (7) 6 (21)
Difficulty in recruiting female reviewers 7 (24) 9 (31) 2 (7) 1 (3) 7 (24) 5 (17) 12 (41) 1 (3) 6 (21)
Too many grant applications in the system 6 (21) 12 (41) 4 (14) 1 (3) 3 (10) 2 (7) 9 (31) 12 (41) 5 (17)
Administrative burden of dealing with peer review process 2 (7) 15 (52) 5 (17) 2 (7) 2 (7) 1 (3) 13 (45) 11 (38) 3 (10)

[TableWrap ID: T3] Table 3 

Participating funding organisations and the response rates


Organisation Country Number of surveys sent out* Number of surveys completed Response rate (%)
Fonds voor Wetenschappelijk Onderzoek (FWO) Belgium 41 19 46
Deutsche Forschungsgemeinschaft (DFG) Germany 49 32 65
National Health And Medical Research Council (of Australia) Australia 50 42 84
Hungarian Scientific Research Fund (OTKA) Hungary 50 36 72
NIHR Coordinating Centre for Health Technology Assessment (NCCHTA) UK 50 22 44
Estonian Science Foundation Estonia 50 38 76
Telethon Italy Italy 49 24 49
The Research Council of Norway Norway 31 19 61
National Institute for Health Research Central Commissioning Facility UK 48 26 54
Total 418 258 62

*After correcting for email delivery failures.


[TableWrap ID: T4] Table 4 

Importance of specific factors in reviewers' decisions to conduct review


Not at all important, n (%) Slightly important, n (%) Important, n (%) Very important, n (%) Extremely important, n (%) Most important factor, n (%)
Opportunity to learn something new 5 (2) 46 (18) 98 (38) 70 (27) 24 (9) 35 (14)
Wanting to keep up to date on research advances in specific areas 7 (3) 39 (15) 88 (34) 85 (33) 25 (10) 35 (14)
Relevance of the topic to your own work or interests 5 (2) 35 (14) 85 (33) 91 (35) 27 (11) 25 (10)
Wanting to enhance your CV and career prospects 85 (33) 98 (38) 37 (14) 20 (8) 3 (1) 6 (2)
Reputation of the funding organisation 25 (10) 65 (25) 104 (43) 41 (16) 7 (3) 2 (1)
Wanting to get known as a reviewer 94 (36) 65 (25) 60 (23) 16 (6) 4 (2) 4 (2)
A sense of professional duty 9 (4) 19 (7) 93 (36) 82 (32) 38 (15) 77 (30)
Wanting to help pay back the efforts of others 18 (7) 49 (19) 92 (36) 67 (26) 16 (6) 10 (4)
Wanting to help external fairness in decision taking by grant review committees 8 (3) 22 (9) 82 (32) 88 (34) 43 (17) 31 (12)
Wanting to help ensure innovation is not suppressed 9 (4) 40 (16) 87 (34) 70 (27) 34 (13) 7 (3)
Wanting to keep your reviewing skills up to date 49 (19) 62 (24) 77 (30) 41 (16) 11 (4) 1 (1)

Percentages do not sum to 100% due to rounding and/or missing data.


[TableWrap ID: T5] Table 5 

Frequency of specific factors cited by reviewers as barriers to undertaking grant review


Never, n (%) Sometimes, n (%) Often, n (%) Always, n (%) Not applicable, n (%)
Insufficient interest in the focus of the application 32 (12) 152 (59) 39 (15) 8 (3) 7 (3)
Insufficient knowledge on the focus of the application 14 (5) 136 (53) 50 (19) 30 (12) 9 (4)
Having to review too many grants for funding organisations 46 (18) 114 (44) 54 (21) 10 (4) 12 (5)
Having to review too many journal articles 38 (15) 99 (38) 74 (29) 19 (7) 8 (3)
Long grant applications 67 (26) 104 (40) 47 (18) 10 (4) 9 (4)
Poor quality of the grant applications 67 (26) 107 (42) 38 (15) 12 (5) 11 (4)
Reviewing taking too much time 35 (14) 111 (43) 68 (26) 17 (7) 5 (2)
Inadequate guidance on the requirements for review 116 (45) 98 (38) 15 (6) 2 (1) 7 (3)
Believing that there is a more appropriate reviewer for the application 32 (12) 162 (63) 33 (13) 6 (2) 5 (2)
Tight deadlines for completing the review 36 (14) 125 (48) 59 (23) 12 (5) 4 (2)
Conflicts with other workload 25 (10) 85 (33) 81 (31) 40 (16) 6 (2)
Lack of formal recognition of reviewer contributions 134 (52) 56 (22) 25 (10) 12 (5) 6 (2)
Not being paid for reviewing 141 (55) 52 (20) 19 (7) 13 (5) 9 (4)
Not being paid enough for reviewing 160 (62) 28 (11) 16 (6) 7 (3) 24 (9)
Having conflicting interests with the applicants 68 (26) 126 (49) 17 (7) 20 (8) 7 (3)
Your own success rate with the funder 163 (63) 50 (19) 7 (3) 1 (1) 14 (5)
Lack of fluency in the language in which the proposal is written 172 (67) 35 (14) 3 (1) 4 (2) 23 (9)

Percentages do not sum to 100% due to rounding and/or missing data.



Article Categories:
  • Research Article


Previous Document:  Life-threatening hypersplenism due to idiopathic portal hypertension in early childhood: case report...
Next Document:  Normalisation process theory: a framework for developing, evaluating and implementing complex interv...