Documents

HIGHLIGHTS

CONTENTS:

A. Proposals and Awards
     Overview
     Characteristics of Principal Investigators
     Award Amounts

B. Methods of Proposal Review
     Peer Review and Merit Review
     Review Processes used at NSF
     Reviewer
     Reviewer Proposals Ratings
     NSF Program Officers
     Review of Program Officer Award Recommendations

C. Requests for Reconsideration of Declined Proposals

D. Small Grants for Exploratory Research (SGER)

E. Committee of Visitors (COV)

F. Exemptions to the Merit Review Process
     Competitively Reviewed Proposals, Awards and Funding Rates

APPENDIX:

Table 1:  Competitively Reviewed Proposals, Awards and Funding Rates By Directorate, FY 1994-1998
Table 2:  Competitively Reviewed Proposals, Awards and Funding Rates By PI Characteristics, FY 1991-1998
Table 3:  Median and Average Award Amounts by Directorate, FY 1994-1998
Table 4:  Methods of NSF Proposal Review By Directorate, FY 1998
Table 5:  Methods of NSF Proposal Review FY 1988-1998
Table 6:  Average Number of Reviews per Proposal By Method and Directorate, FY 1998
Table 7:  Requests for Formal Reconsideration of Declined Proposals By Directorate, FY 1994-1998
Table 8:  Small Grants for Exploratory Research (SGER) Proposals, Awards and Funding Rates By Directorate, FY 1991-1998
Table 9:  Small Grants for Exploratory Research (SGER) Funding Trends by Directorate, FY 1996-1998
Table 10:  Committee of Visitors Meetings By Directorate

HIGHTLIGHTS

1. This report responds to a long-standing NSB policy requesting that the NSF Director submit an annual report on the NSF merit review system.
2. During FY 1998 NSF received 28,321 proposals. The number of proposals received annually was stable at about 30,000 from 1993 to 1997.
3. A total of 9,280 competitive proposals were funded in FY 1998, a decrease of 5.9% from the previous year. The number of awards made each year has fluctuated between approximately 9,000 and 10,000 over the past five years. The number of awards made in FY 1998 was 7 percent lower than the number made in FY 1994.
4. The NSF-wide funding rate was 33 percent in FY 1998, representing no change from the previous year and remaining level with the five-year maximum reached in FY1994. Directorate funding rates in FY 1998 ranged from 25 percent to 41 percent.
5. Proposals from minority Principal Investigators (PIs) in FY 1998 were funded below the NSF average (31 and 33 percent respectively). Proposals from female PIs in FY 1998 were funded above the NSF average (34 and 33 percent respectively).
6. Proposals from PIs who had received an NSF award in a previous fiscal year (prior PIs) were funded at a higher rate than proposals from new PIs (39 and 25 percent, respectively in FY 1998). This gap has remained stable for several years.
7. Award size increased from the previous fiscal year. The median NSF award amount in FY 1998 was $61,666; the average was $105,382.
8. The most frequent method of proposal review was a combination of mail and panel methods. Sixty-three percent of proposals were reviewed in this manner in FY 1998, an increase of 21 percent in five years.
9. NSF received a total of more than 244,000 reviews in FY 1998, for an average of 8.6 reviews per proposal. The number of reviews received per proposal varied according to review method: 10.6 reviews per proposal for mail-plus-panel review, 5.9 reviews per proposal for panel-only review, and 4.6 reviews per proposal for mail-only review. The response rate to mail review requests remained level at 62 percent.
10. Both proposal pressure and average award size for Small Grants for Exploratory Research (SGER) continued to increase in FY 1998.
11. There were 53 requests for formal reconsideration of declinations submitted to Assistant Directors during FY 1998; 3 of these declinations were reversed at the directorate level. 3 requests for reconsideration were submitted to the Deputy Director; all 3 of these declinations were upheld.
12. There were no lawsuits involving the NSF merit review system in FY 1998.
13. In March 1997, the NSB approved changes to the merit review criteria to reflect the Foundation's strategic plan more effectively. These changes became operational at the start of FY 1998.

The FY 1998 Report on the NSF Merit Review System responds to a National Science Board (NSB)
policy endorsed in 1977 and amended in 1984, requesting that the Director of the National Science Foundation
(NSF) submit an annual report on the NSF proposal review system. This report provides summary information
about levels of proposal and award activity and the process by which proposals are reviewed and awarded.

A.   Proposals and Awards

Overview
During FY 1998, NSF took action on 28,321 competitive, merit reviewed research and education proposals. The number of proposals reviewed annually by NSF had been reasonably stable at about 30,000 proposals since 1994. This drop to the 28,000 level has not been analyzed yet.

NSF funding was awarded to 9,280 of the proposals, resulting in an overall-funding rate of 33 percent. As shown in Text Figure 1, the number of awards made each year has varied between approximately 9,000 and 10,000. In FY 1998 the number of awards decreased by 5.9 percent relative to FY 1997 funding rate. Funding rates vary among directorates, ranging from 25 percent to 41 percent as shown in Appendix Table 1.

Characteristics of Principal Investigators
The number of proposals received from female Principal Investigators (PIs) has increased in most years since 1991. The same has been true for PIs from minority groups, though the increase has been slower than for female PIs (10 percent increase from FY 1991 to 1998 for minority PIs; 19 percent for female PIs.) During FY 1998, about 20 percent of competitively reviewed proposals were from female PIs (a one- percent increase from FY 1997, with a corresponding decrease in proposals from male PIs), and four percent were from minority PIs (level with FY 1997.)

Funding rates of proposals from female PIs have been higher than proposals from male PIs for the past five years. The funding rates for proposals from minority PIs have been above the overall NSF funding rate for only one of the past eight years.

Forty-three percent of the competitively reviewed proposals in FY 1998 were from PIs who had not received an NSF award in a previous fiscal year ('new PIs'). This percentage represents a decrease from 44 percent in FY 1997 and 51 percent in FY 1991. The funding rate for proposals from PIs who had received an NSF award (prior PIs) was higher than proposals from new PIs (39 percent and 25 percent, respectively in FY 1998). The difference in funding rates between proposals from new and prior PIs has remained approximately level since 1993.

Trends in funding rate for all PIs, female and minority PIs, and prior and new PIs are graphed in Text Figure 2. Proposals, awards, funding rates and trends by PI characteristics are presented in Appendix Table 2.

Award Amounts
The median annual award amount (adjusted for multiple year projects) among competitive awards made during FY 1998 was $61,666, a 5.5 percent increase from FY 1997. The average award amount in FY 1998 was $105,382, an increase of 9.6 percent from FY 1997. The difference between the median and average award amounts reflects the effect of numerous small awards on the median, and a few large awards for centers, facilities, and large systemic initiatives on the average award amount. Award amounts have been consistent over the past decade, when adjusted to constant dollars as measured by the Consumer Price Index. There are considerable variations among directorates, as shown in Text Figure 3. Data on median and average award amounts from FY 1994-1998 are presented by directorate in Appendix Table 3.

B.  Methods of Proposal Review

Peer Evaluation and Merit Review
The involvement of knowledgeable peers from outside the Foundation in the review of proposals is the keystone of NSF's proposal review system. Their judgements of the extent to which proposals address established criteria are vital for informing NSF staff and influencing funding recommendations. For this reason, NSF's system of proposal review can accurately be characterized as "merit review with peer evaluation."

Review Processes used at NSF
NSF programs obtain external peer review by two principal methods, mail and panel. In addition to mail and panel reviews, site visits by NSF staff and external peers are often used to review proposals for large facilities, centers, and systemic reform initiatives. NSF program officers are given discretion in the specific use of review methods, subject to supervisory approval. For example, some programs try to reduce proposal pressure by requiring submission of preliminary proposals. Review of preliminary proposals varies widely, ranging from non-binding advice form program officers to proposers, to binding decisions from external reviewers. Systematic data on the use of preliminary proposals are not available.

In "mail-only" reviews, peers are sent proposals and asked to submit written comments to NSF by postal mail, facsimile, electronic mail, or through FastLane, NSF's Web-based system for electronic proposal submission and review. These mail reviews are then used by the NSF program officer directly to support a recommendation for award or decline.

"Panel-only" review refers to the process of soliciting reviews only from those peers who meet in a panel review setting to discuss their reviews and provide advice directly to the program officer. Most programs that use this process mail proposals out to panelists and receive their reviews prior to the panel meeting. Other programs provide panelists with access to the proposals at the beginning of the panel meeting, allowing them a period of time during which they prepare their reviews at the meeting.

Most proposals submitted to NSF are reviewed using some combination of these two processes ("mail-plus-panel" review). Those programs that employ the mail-plus-panel review process have developed several different configurations, such as:

• A peer is asked to submit a written mail review and also serve as a panelist, in effect contributing two reviews for each proposal; and
  
  
• A peer is asked to participate only as a panelist, with responsibility only for reviewing and discussing mail reviews written by others and providing verbal and/or written advice to the program officer.

The mail-plus-panel method was used for 63 percent of proposals reviewed during FY 1998; 21 percent of proposals were reviewed by mail-only, and 17 percent by panel-only. Directorates vary in their use of proposal review methods. Mail-plus-panel review was the most common review process used in the BIO, CSE, ENG, GEO, and SBE Directorates. Mail-only review was the predominant mode of review in MPS. Panel-only review was the most commonly used method in EHR. Directorate-level data on the use of different review processes during FY 1998 are presented in Appendix Table 4.

The use of various review processes has varied over time. The percentage of NSF proposals reviewed by mail-plus-panel has increased from 42 to 63 percent of all proposals since FY 1989.

There has been a steady decline in the use of mail-only review from 39 to 21 percent during the past decade. The use of panel-only review has varied less than other methods, increasing slowly for several years and then declining from 24 to 17 percent during the past four years. These trends are shown in Text Figure 4, and the corresponding data are presented in Appendix Table 5.

NSF policy states that each recommendation for final action on a proposal must be accompanied by at least three external reviews, unless the requirement has been waived (see section F, below). The total numbers of reviews and the average numbers of reviews peer proposal obtained by these different methods are presented in Text Figure 5. Directorate-level data for FY 1998 are presented in Appendix Table 6.

Diversity of the reviewer pool is an important feature of the NSF merit review system. Reviewers from diverse backgrounds help ensure that a wide range of perspectives are taken into consideration in the review process. NSF emphasizes reviewer diversity through a variety of processes, including use of a large and expanding Foundation-wide reviewer database, explicit policy guidance, mandatory training for all program officers, and directorate-level initiatives. NSF maintains a central electronic database of 241,000 reviewers. For proposal decisions in FY 1998, 47,000 of these reviewers were sent one or more proposals for mail review, 31,700 reviewed at least one proposal by mail, and 7,800 reviewers served as panelists. In all, 51,000 individuals either served on a panel, were sent a proposal for mail review, or served in both functions.

Potential reviewers are identified from a variety of sources including applicant suggestions, references attached to proposals and published papers, and input from mail reviewers, panelists, and visiting scientists. During FY 1998, approximately 28,500 of the 241,000 records now in the reviewer database were either added or updated.

Participation in the peer review process is voluntary. Panelists are reimbursed for expenses; mail reviewers receive no financial compensation. In FY 1998, 62 percent of requests for mail reviews produced responses, which represents a slight decrease from the 64 percent response rate that has been stable since 1991.

Reviewer Proposal Ratings

The NSF merit review system emphasizes reviewer narratives over summary ratings. Summary ratings are but one indicator of reviewer judgment of the proposal quality. The written narratives provided by reviewers, the deliberations by panel members, and the expert opinions provided by program officers are all important components of the merit review system. No one component is allowed to dominate over the others.

The distribution of average summary ratings of reviews for awarded and declined proposals is provided in Text Figure 6. Only those ratings provided for mail-only and mail-plus-panel reviewers have been included. Panel-only reviewers often submit comments without a summary rating. These data indicate considerable overlap among the average reviewer ratings of successful and unsuccessful proposals, most notably in the range of "very good" average ratings. The judgment of NFS staff is essential to making this difficult separation between awards and declines.

NSF Program Officers
The narrative comments and summary ratings provided by external reviewers are essential inputs in NSF's system of merit review. Once received, these inputs inform the judgment of the program officers who formulate award and decline recommendations to NSF's senior management. These program officers are scientists, engineers, and educators to whom NSF looks for expert judgment and program management. In making recommendations to award or decline proposals, these highly qualified individuals produce and manage a portfolio of awards addressing NSF's strategic goals and related factors such as;

• Contributions to human resource and institutional infrastructure development,
  
  
• Support for "risky" proposals with potential for significant advances in a field,
  
  
• Encouragement of interdisciplinary activities, and
  
  
• Achievement of program-level objectives and initiatives.

The number of program officers employed by NSF has remained stable at slightly over 400 for the past five years, despite increases in proposal pressure and general workload. Depending on their professional experience, program officers are classified as assistant program director, associate program director, or program director. They can be permanent NSF employees or temporary employees. Some temporary program officers are "on loan" as visiting scientists, engineers, and educators (VSEEs) for up to three years from their host institutions. Others are employed through grants to the home institutions under the terms of the Intergovernmental Personnel Act. The distribution of these program officers among these and other categories is presented in Text Figure 7.

Review of Program Officer Award Recommendations
Each program officer's recommendation to award or decline a proposal is subject to a programmatic review by a higher level reviewing official (usually the division director), and an administrative review by a grants officer in the Office of Budget, Finance, and Award Management (BFA). All award recommendations in excess of $1.5 million in any one project year or $6 million over five years must be reviewed by the Director's Review Board (DRB). Awards in excess of a $3 million commitment during a project year, or $15 million over five years, require approval by the National Science Board.

Notes: Percentages of all Program Officers are presented and may not add to 100% due to rounding.
VSEE:  Individual employed as a Visiting Scientist, Engineer, or Educator (formerly termed "Rotator")
IPA:  Individual employed under the Intergovernmental Personnel Act

C.   Requests for Reconsideration of Declined Proposals

NSF policy includes several mechanisms that provide proposers with information on how the review process led to a decline recommendation, and on procedures for obtaining additional explanations for declinations. These policies help to ensure that NSF's review has been fair and reasonable, and that NSF maintains the openness, quality, and integrity of the merit review process.

Every proposer receives from the NSF program officer a description of the context in which the proposal was reviewed, along with a verbatim copy of each review that was considered in the review process. A declined PI may ask the cognizant program officer for additional clarification of the decision. If after considering this additional information a PI is not satisfied that the proposal was fairly handled and reasonably reviewed, he or she may request formal reconsideration from the cognizant assistant director (AD). This request can be based on the PI's perception of procedural errors or on disagreements over the substantive issues dealt with by reviewers. If the AD upholds the original action, the applicant's institution may request a second reconsideration from the Foundation's Deputy Director (O/DD).

On average, NSF annually declines over 20,000 proposals but receives, on average, only 50 requests for formal reconsideration. Most program-level decisions are upheld in the reconsideration process. The number of requests for formal reconsideration and resulting decisions at both the AD and O/DD levels from FY 1994 through FY 1998 are displayed in Appendix Table 7.

D.  Small Grants for Exploratory Research (SGER)

Since the beginning of FY 1990, the Small Grants for Exploratory Research (SGER) option has permitted program officers throughout the Foundation to make short-term (one to two years), small-scale (less than $50 K) grants without formal external review. Characteristics of activities which can be supported by an SGER award include:

• Preliminary work on untested and novel ideas;
• Application of new approaches to "old" topics;
• Ventures into emerging research areas; and
• Narrow windows of opportunity for data collection, such as natural disasters and infrequent phenomena.

The funding rate for SGERs in FY 1998 was 83 percent. The SGER funding rate is much higher than for regular, competitively reviewed proposals in large part because potential SGER applicants are encouraged to contact an NSF program officer before submitting an SGER proposal to determine its appropriateness for the SGER funding option. As potential SGER applicants have become familiar with this practice, the SGER funding rate has increased from 55 percent in its first year (FY 1990) to 83 percent in FY 1998. Additional details are shown in Text Figure 8.

Coincident with these policy changes, the downward trends in SGER proposal pressure and number of awards were reversed in FY 1997. NSF received 242 SGER proposals in FY 1997 and made 193 awards. NSF received 299 SGER proposals in FY 1998 and made 247 awards. Directorates vary in the degree to which their program officers discourage potential proposers from submitting inappropriate ideas as formal SGER proposals. As a result, FY 1998 funding rates for SGER proposals varied among directorates from 56 to 96 percent. Directorate-level data on SGER proposal pressure and funding rates are presented in Appendix Table 8.

The total amount awarded to SGERs also increased in FY 1998 to $12,320,200. This represents an all-time high for SGER support and a 46 percent increase from FY 1997. The average SGER award amount in FY 1998 was $49,879, a 14 percent increase relative to the FY 1997 average award amount of $43,591. Despite these increases, the total NSF investment in SGERs remains less than one percent of the operating budget for research and education, far below the five percent that program officers may commit to SGER awards. The history of SGER awards by directorate from FY 1996 to FY 1998 is presented in Appendix Table 9.

E.  Committees of Visitors (COV)

Committees of Visitors (COV) are panels of external experts convened to review the technical and managerial stewardship of NSF programs or clusters of programs. Each program that awards grants or cooperative agreements normally is reviewed on a three-year cycle. There are currently 179 such programs at NSF; 54 programs were reviewed during FY 1998. A list of all programs subject to review by a Committee of Visitors and the fiscal year of the most recent review is provided in Appendix Table 10. Due to reorganization of several directorates, COV review of a number of programs has been delayed. Where this has occurred, the directorates have been notified and have agreed to correct this situation by scheduling COVs during FY 1999 and FY 2000.

Each COV must operate in accordance with the Federal Advisory Committee Act (FACA) of 1972. In compliance with FACA regulations, virtually all COVs are established as subcommittees of an existing chartered directorate advisory committee, and the COV report is reviewed and approved by the parent advisory committee. The cognizant assistant director (AD) provides the parent advisory committee with a written response to each COV report. The COV's report and the AD's response are public documents; some have been publicized in the professional literature.

F.  Exemptions to the Merit Review Process

Authorized exemptions to the peer review process are listed in NSF Manual 10, Section 122 (Attachment I) and include routine award actions such as continuing grant increments and no-cost extensions. In special circumstances, the Director or designee may waive peer review requirements. Such waivers of peer review were granted 7 times during FY 1998; 5 for OIA, 1 for SBE and 1 for CISE.

Source:  NSF Enterprise Information System, as of February, 1999

Source:  NSF Enterprise Information System, as of February, 1999

Source:  NSF Enterprise Information System, as of February, 1999.

Source:  NSF Enterprise Information System, as of February 1999.

Note:
 Number of proposals differs slightly from previous table due to coding variations.

Source:  NSF Enterprise Information System, as of February 1999.

Notes:
Peers participating as both a mail and panel reviewer for the same proposal are counted as one review in this table.
"Other" includes the Office of Polar Programs and the Office of Integrative Activities.

Source:  NSF Enterprise Information System, as of February 1999.

If you have any questions or comments concerning this report,
please send your them to Dr. Albert Bridgewater at abridgew@nsf.gov

Back to NSB Publications