2. Methods of Proposal Review

The involvement of knowledgeable peers from outside the Foundation in the review of proposals is the keystone of NSF's proposal review system. Their judgements of the extent to which proposals address established criteria are vital for informing NSF staff and influencing funding recommendations. For this reason, NSF's system of proposal review can accurately be characterized as "merit review with peer evaluation."

Review Processes Used at NSF

NSF programs obtain external peer review by three principal methods: (1) "mail-only," (2) "panel-only," and (3) "mail-plus-panel" review. In addition, site visits by NSF staff and external peers are often used to review proposals for large facilities, centers, and systemic reform initiatives. NSF program officers are given discretion in the specific use of review methods, subject to supervisory approval.

In "mail-only" reviews, peers are sent proposals and asked to submit written comments to NSF through FastLane, NSF's Web-based system for electronic proposal submission and review. These mail reviews are then used by the NSF program officer to support a recommendation for award or decline.

"Panel-only" review refers to the process of soliciting reviews only from those peers who meet in a panel review setting to discuss their reviews and provide advice directly to the program officer. Most programs that use this process provide proposals to panelists and receive their reviews prior to the panel meeting.

Many proposals submitted to NSF are reviewed using some combination of these two processes ("mail-plus-panel" review). Those programs that employ the mail-plus-panel review process have developed several different configurations, such as:

• A peer is asked to submit a written mail review and also serve as a panelist, in effect contributing two reviews for each proposal; and

• A peer is asked to participate only as a panelist, with responsibility only for reviewing and discussing mail reviews written by others and providing verbal and/or written advice to the program officer.

The use of various review methods has changed markedly over time, as shown in Text Figure 4, and the corresponding data in Appendix Table 4. Since 1991 the percentage of NSF proposals reviewed by panel-only has increased from 40 to 46 percent of all proposals. During the same period, there has been a steady decline in the use of mail-only review from 30 to 17 percent. The use of mail-plus-panel review increased from 31 to 38 percent during the past ten years.

Directorate-level data on the use of different review processes during FY 2000 are presented in Appendix Table 5. For both historical and currently practical reasons, NSD Directorates vary in their use of proposal review methods. Mail-plus-panel review was the predominant review process used in the BIO, GEO, and SBE Directorates. Mail-only review was the most common mode of review in MPS. Panel-only review was the predominant method in CISE, ENG and EHR. These trends have major implications for the way that NSF conducts its business. For example, as indicated in Text Figure 5, there is a strong relationship between time to decision and type of review. In FY 2000, 58% of all proposals reviewed by panel-only were processed within six months, compared to 52% for mail-plus-panel and 47% for mail-only.

The increasing trend toward a review process that includes some form of panel review likely results from such factors as: the increasing numbers of proposals that require multidisciplinary reviews (which panels seem better able to provide); the need to reduce proposal processing time (time-to-decision) and the increasing difficulty of obtaining ad-hoc mail reviews.

Text Figure 5
Time to Decision by Type
FY 1999 - 2000

FY	Review Method	0-6 Months	>6-9 Months	>9-12 Months	>12 Months
2000	Mail-plus-Panel	52%	37%	8%	3%
	Mail Only	47%	38%	10%	4%
	Panel Only	58%	32%	8%	2%
1999	Mail-plus-Panel	54%	36%	7%	3%
	Mail Only	50%	36%	10%	4%
	Panel Only	65%	27%	6%	1%

Reviews and Reviewers

NSF policy states that each recommendation for final action on a proposal must be accompanied by at least three external reviews, unless the requirement has been waived under special circumstances. The total numbers of reviews and the average numbers of reviews per proposal obtained by the three different review methods are presented in Text Figure 6. There is considerable variation among the review methods. Some of this difference may be attributed to the way reviewers are counted in various types of panels. For example, a panel might have 10-12 panelists of whom 4 lead reviewers are assigned to write an individual review. But, if all 10-12 panelists weigh in on the panel consensus, all may be counted as reviewers.

Directorate-level data for FY 2000 are presented in Appendix Table 6. The wide variation among directorates in the number of reviews per proposal may possible reflect: (1) their preferences for the different review methods, and (2) differences in the way directorates count reviewers on the panel review process.

Text Figure 6
Reviews per Proposal, FY 2000

	All Methods	Mail-plus-Panel	Mail-Only	Panel-Only
# of Reviews	241,879	143,366	22,085	76,428
# of Proposals	28,559	10,786	4,758	13,015
Reviews per Proposal	8.5	13.3	4.6	5.9

Diversity of the reviewer pool is an important feature of the merit review system. Reviewers from diverse backgrounds help ensure that a wide range of perspectives is taken into consideration in the review process. NSF emphasizes reviewer diversity through a variety of processes, including use of a large and expanding Foundation-wide reviewer database, explicit policy guidance, mandatory training for all program officers, and directorate-level initiatives.

NSF maintains a central electronic database of about 250,000 reviewers. For proposal decisions in FY 2000, 46,897 of these reviewers were sent one or more proposals for mail review and 8,708 reviewers served as panelists. In all, 52,318 individuals served on a panel, were sent a proposal for mail review, or served in both functions.

Potential reviewers are identified from a variety of sources including applicant suggestions, references attached to proposals and published papers, and input from mail reviewers, panelists, and visiting scientists. During FY 2000, approximately 7,000 of the 259,623 records now in the reviewer database were either added or updated. Participation in the peer review process is voluntary. Panelists are reimbursed for expenses; mail reviewers receive no financial compensation. In FY 2000, 60 percent of requests for mail reviews produced responses, which represents an increase from the 59 percent response rate in FY 1999, although it is still not equal to the 64 percent response rate that prevailed in the 1990s.

In FY 2001, NSF will develop systems and policies to enable it to request voluntary demographic data electronically from all reviewers to determine participation levels of members of underrepresented groups in the NSF reviewer pool. A baseline for FY 2002 will be derived from this data.

Reviewer Proposal Ratings

The distribution of average summary ratings⁵ of reviews for awarded and declined proposals is provided in Text Figure 7.

These data indicate considerable overlap among the average reviewer ratings of successful and unsuccessful proposals, most notably in the range of "very good" average ratings. The judgment and discretion of the NSF professional staff is essential to making this difficult separation between awards and declines. Appendix Table 7 indicates that this overlap among the average reviewer ratings is present and similar in degree for each of the three proposal review methods used by NSF (panel-only, mail-only, and mail plus panel).

These data also indicates that a large number of potentially fundable proposals are declined each year. Text Figure 8 indicates that almost $2 billion of declined proposals are rated very good or better and that almost $1 billion are rated as high as the average NSF award. These declined proposals represent a rich portfolio of unfunded opportunities - fertile ground for learning and discovery that lies fallow.

NSF Program Officers

The narrative comments and summary ratings provided by external reviewers are essential inputs that inform the judgment of the program officers who formulate award and decline recommendations to NSF's senior management. In making these recommendations, highly qualified program officers produce and manage a portfolio of awards, addressing such factors as:

• Contributions to human resource and institutional infrastructure development,

• Support for "risky" proposals with potential for significant advances in a field,

• Encouragement of interdisciplinary activities, and

• Achievement of program-level objectives and initiatives.

The number of program officers employed by NSF has remained stable at around 400 for the past five years, despite increases in proposal complexity and general workload.
The distribution of these program officers by characteristics is presented in Text Figure 9.

Text Figure 9
Distribution of NSF Program Officers by Characteristics
As of October 1, 2000

	Assistant Program Directors	Associate Program Directors	Program Directors	Total
Total	11	25	360	396
Male	3 27%	17 68%	229 64%	249 63%
Female	8 73%	8 32%	131 36%	147 37%
Minority	2 18%	4 16%	69 19%	75 19%
White, Non-Hispanic	9 82%	21 84%	291 81%	321 81%
Permanent	10 91%	22 88%	190 53%	222 56%
VSEE	0 0.0%	1 4%	37 10%	38 10%
Temporary	1 9%	1 4%	40 11%	42 10%
IPA	0 0%	1 4%	93 26%	94 24%

Source: NSF Division of Human Resource Management
Notes: VSEE: Individual employed as a Visiting Scientist, Engineer, or Educator (formerly termed "Rotator"). IPA: Individual employed under the Intergovernmental Personnel Act.

Depending on their professional experience, program officers are classified as assistant program director, associate program director, or program director. They can be permanent NSF employees or temporary employees. Some temporary program officers are "on loan" as visiting scientists, engineers, and educators (VSEEs) for up to three years from their host institutions. Others are employed through grants to the home institutions under the terms of the Intergovernmental Personnel Act (IPA).

The number of minority program directors has increased from 73 to 75, while number of female program directors has decreased from 150 to 147. However, the percentages in text Figure 9 are not significantly different than those reported in the FY 1999 Report on the Merit Review System. Of the 222 program directors that have permanent positions at NSF, 23 have reported having a major disability.

Assuring Objectivity in the Merit Review Process

NSF program officers carefully check all proposals for potential conflict-of-interest and select expert outside reviewers with no apparent potential conflicts. All reviewers are instructed to declare potential conflicts. All program officers receive conflicts-of-interest training annually.

Every proposer receives from the NSF program officer a description of the context in which the proposal was reviewed, along with an anonymous verbatim copy of each review that was considered in the review process. A declined PI may ask the cognizant program officer for additional clarification of the decision. If after considering this additional information a PI is not satisfied that the proposal was fairly handled and reasonably reviewed, he or she may request formal reconsideration from the cognizant Assistant Director (AD). This request can be based on the PI's perception of procedural errors or on disagreements over the substantive issues dealt with by reviewers. If the AD upholds the original action, the applicant's institution may request a second reconsideration from the Foundation's Deputy Director (O/DD).
On average, NSF annually declines over 20,000 proposals but receives only 40-50 requests for formal reconsideration. Most program-level decisions are upheld in the reconsideration process. Out of the 247 requests for formal reconsideration of declined proposals during the past six years, 10 decisions have been reversed. The number of requests for formal reconsideration and resulting decisions at both the AD and O/DD levels from FY 1995 through FY 2000 are displayed in Appendix Table 8.

Each program officer's recommendation to award or decline a proposal is subject to a programmatic review by a higher level reviewing official (usually the division director), and an administrative review by a grants officer in the Office of Budget, Finance, and Award Management (BFA). The Director's Review Board (DRB) must review all award recommendations in excess of $1.5 million in any one project year or $6 million over five years.⁶

Changes to the NSB threshold were effective at the start of FY 2000. The new threshold requires NSB approval of awards where the average annual award amount is 1% or more of the awarding directorate's prior year current plan.⁷ In FY 2000, the NSB reviewed 16 actions, including six recommended proposals, two RFPs, and eight new programs and major construction projects.

__________

5 The NSF merit review system emphasizes reviewer narratives over summary ratings. Summary ratings are but one indicator of reviewer judgment of the proposal quality. The written narratives provided by reviewers, the deliberations by panel members, and the expert opinions provided by program officers are all important components of the merit review system. No one component is allowed to dominate over the others.

6 Effective, FY 2001, items subject to DRB review include awards with an average annual award amount of 2.5% or more of a Division's prior year current plan.

7 Other items requiring NSB prior approval are new programs and major construction projects that meet certain specifications.