3. Government Performance and Results Act (GPRA)

Introduction

Because of its importance to the success of NSF's mission, "operating a credible, efficient merit review system" is cited as one of the four critical factors for success in NSF's FY 2001-2006 GPRA Strategic Plan. The NSF Inspector General has also identified merit review in her annual list of management and performance challenges:⁸

" ..... NSF should enhance its efforts to expand the peer review community to ensure diversity with respect to race, gender, geography and type of school, providing the chance to participate to all who are qualified."

Not surprisingly then, several of the investment process goals in the FY 2000 GPRA Performance Plan are focused on various aspects of the award selection process, such as the use of the merit review criteria, the need to keep the awards system open to new people and new ideas, and the time it takes to process a proposal. NSF's progress in meeting these goals is described below.

GPRA Performance Goals and Results

Investment Process Goal 1: At least 90% of NSF funds will be allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process.

Background: The Administration and NSF believe that award selections based on a competitive merit review process with peer evaluation ensure those ideas from the strongest researchers and educators will be identified.

Results: This goal was achieved in FY 2000. As in FY 1999, NSF allocated 95% of its funds to merit reviewed projects. It will be revised for FY 2001, based on OMB revised definitions.⁹

Investment Process Goal 2: NSF performance in implementation of the new merit review criteria is successful when:

• reviewers address the elements of both generic review criteria appropriate to the proposal at hand, and

• when program officers take the information provided into account in their decisions on awards, as judged by external independent experts.

Background: New criteria were used for the first time during FY 1999. The use of both criteria (quality and impact) by both expert reviewers and program staff is an important step in the investment process and for the implementation of NSF's broader goals. To evaluate NSF's progress in meeting this goal, external committees are asked to assess the use of the two merit review criteria by reviewers and program officers.

Results: This goal was not achieved. For FY 2000, a total of 58 out of 64 reports rated programs on their use of both merit review criteria. NSF was judged successful in achieving this goal in only 20 of the 58 reports. These results provide a clearer indication that NSF has a long way to go to fully achieve this goal, but that progress has been made.

Full implementation of this goal is a priority for NSF in FY 2001 and beyond. To do so requires information to be included in proposals, addressed by reviewers, and taken into account by program staff. NSF has taken steps to ensure that incoming proposals contain adequate information for reviewers to evaluate.

For FY 2001, different on-screen pages have been provided in FastLane - NSF's electronic data system - so reviewers can address each merit-review criterion separately. The performance data will be collected from the FastLane database. Program officer reviews of projects will be inspected for use of both criteria.

Related NAPA Study: In response to a directive by the Senate Appropriations Committee that NSF review the procedure and criteria for merit review once the new criteria have been in place for a year, in FY 2000, NSF issued a contract to NAPA to conduct a study of the impact of the new merit review criteria on the nature of the projects NSF supports. In conducting the study, NAPA interviewed key personnel and stakeholders from the S&E community and analyzed a sample of COV reports and proposal documents. The key finding was that it is too soon to make valid judgements about the impact and effectiveness of the new criteria. The NAPA report also highlighted the need to (1) improve the conceptual clarity of the criteria, (2) better communicate with proposers, reviewers and NSF staff about how the criteria are to be used, and (3) improve quantitative measures and performance indicators to track the objectives and implementation of the new criteria. NSF is implementing these suggestions beginning in FY 2001.

Investment Process Goal 3: Identify possible reasons for customer dissatisfaction with NSF's merit review system and with NSF's complaint system. (New goal in FY 2000)

Background: For the past two years, NSF has participated in a national assessment of customer satisfaction along with about 30 other federal agencies. The mechanism used to assess customer satisfaction is the American Customer Satisfaction Index (ACSI), a cross-industry index of customer satisfaction. The Foundation's ACSI results (Baseline: 57 on a scale of 0-100 in FY 1999) for that survey indicated that NSF grant applicants generally hold NSF in high regard and give it high marks for accessibility and usefulness of information. However, NSF received only mid-level scores for its merit review process and for its handling of customer complaints.

Results: This goal was achieved through meetings with principal investigators and research administrators, and additional surveys including the ACSI* survey of awardees only and regional grants seminar surveys. The awardee survey results indicate that NSF customers' primary concern regarding the timeliness and efficiency of the proposal process is the time it takes NSF to reach a funding decision. NSF is striving to improve the time to decision (see Investment Process goal 7). Applicants who stated that they had a specific problem or concern with the quality or fairness of merit review identified two primary concerns: reviews were inappropriate (i.e., reviews did not seem to adequately address the proposed project, in the opinion of the applicant) and reviews were uneven (i.e., the range of review scores included both high and low scores).

Investment Process Goal 4: Identify best practices and training necessary for NSF staff to (1) conduct merit review and answer questions about the review criteria and process, and (2) answer questions from the community and to deal with complaints in a forthright manner. (New goal in FY 2000)

Results: This goal was not achieved. During FY 2000 NSF conducted customer service surveys and solicited other forms of feedback in an effort to pinpoint specific customer issues and to identify effective practices for handling customer complaints within NSF. Further, other Federal agencies were examined to locate a model with similar customer interactions, but no appropriate model was identified. However, models of best practices and NSF staff training are still being developed in FY 2001.

NSF continues to place great importance on these issues and will complete this effort in FY 2001. In addition, NSF will pilot the best of the models in NSF divisions and provide specific customer service training to NSF staff.

Investment Process Goal 5: Improve NSF's overall ACSI index compared to the FY 1999 index of 57 (on a scale of 0-100).

Background: (See Goal 3 background.)

Results: This goal was achieved. NSF achieved an ACSI index of 58 in FY 2000. This feedback is helping NSF to focus its efforts to improve customer service The 2000 ACSI survey indicated that NSF improved slightly in two key areas: 1) timeliness and efficiency of the proposal process and 2) quality and fairness of merit review. These were two of the areas of greatest concern from the FY 1999 survey.

Investment Process Goal 6: Time to Prepare Proposals: 95% of program announcements and solicitations will be available at least three months prior to proposal deadlines or target dates.

Background: NSF staff work toward this goal by limiting the number of special competitions requiring individual program announcements and solicitations, planning for such competitions as far in advance as possible, and initiating clearance processes at least six months prior to the anticipated proposal deadlines.

Results: This goal was not achieved. In FY 2000, 89% of program announcements and solicitations were made available at least three months prior to their deadline/target date. Approximately 8 percent of program announcements and solicitations missed the 90-day time limit by fewer than 5 days. This is a significant improvement over FY 1999, when 75% of announcements met the 3-month standard. The Foundation intends to review and revise the timing of clearance procedures, in order to ensure that web posting of announcements will occur in a timely manner.

Investment Process Goal 7: Maintain the FY 1999 goal to process 70% of proposals within six months of receipt, improving upon the FY 1998 baseline of 58%.

Background: NSF's long term goal continues to be processing 95% of proposals within six months of receipt. In other words, NSF should be able to tell applicants whether their proposals have been declined or recommended for funding within six months of receiving them.

Results: This goal was not achieved. In FY 2000, 54% of proposals were processed within six months of receipt, while an additional 35% of proposals were processed between six and nine months of receipt. In FY 2001, NSF staff will work towards shortening the award processing time by making more effective use of electronic mechanisms in conducting the review, working cooperatively to reduce overloads and bottlenecks, and by carefully tracking the stage of processing and age of all proposals.

In addition, some internal organizations are reconsidering the practice of holding over proposals for potential funding until the next fiscal year, while some have added "performance on prompt handling of proposals" to the performance evaluation criteria of their staff. Moreover, NSF is committed to increasing staffing in FY 2001, to accommodate the anticipated increase in proposals associated with the budget increase and the major initiatives.

Investment Process Goal 8: The percentage of competitive research grants going to new investigators will be at least 30%, 3% over the FY 1998 baseline of 27%.

Background: NSF believes that it is important that the proposal and award process be open to new people and new ideas, to help ensure that NSF is supporting research at the frontier of science and engineering. NSF is committed to maintaining openness in the system and will strive to increase the percentage of awards to new investigators. This goal will be maintained in FY 2001.

Results: This goal was not achieved. The percentage of competitive research grants to new investigators was 28% in FY 2000, one percent higher than in FY 1999. In the early 1990's, NSF had percentages approximating 30 percent of all competitive research grants going to new investigators. The percentage dropped over the mid-1990's, and is now rising slightly.

This is a challenging goal for NSF. NSF will continue to seek creative and innovative proposals from new investigators. Program staff will attend scientific meetings, conferences, and conventions and will conduct site visits to promote awareness of the research opportunities at NSF and to encourage new investigators to submit proposals. NSF will examine trends, such as whether the pool of new investigators is smaller than in previous years or whether they are submitting fewer proposals, and if needed, use this information to modify targets in the future.

Investment Process Goal 11: NSF will identify mechanisms to increase the number of women and underrepresented minorities in the proposal applicant pool, and will identify mechanisms to retain that pool.

Background: Recognizing that progress toward all outcome goals for research and education requires maximal diversity of intellectual thought, NSF is emphasizing attention to enhancing the participation of groups currently underrepresented in science and engineering, including women, underrepresented minorities, and persons with disabilities, in all its programs. The long-term objective is to have a science and engineering workforce that mirrors the U.S. population.

Results: This goal was achieved. NSF continues to work toward increasing diversity in its proposal applicant pool through the following means:

• To place the issue on equal footing with the quality of research being supported, NSF issued Important Notice No. 125 to presidents of universities and colleges encouraging PIs to address the merit review criterion - what are the broader impacts of the proposed activity- which embraces integrating diversity into all NSF supported activities;

• Developing and increasing funding for specialized programs designed to promote diversity;

• Recruiting members of underrepresented groups for merit review panels, COVs, and NSF workshops and conferences; and

• Strongly encouraging women, minorities, and persons with disabilities to compete fully in NSF programs.

NSF is revising this goal to extend its efforts as it continues to pursue diversity in the applicant pool. A new goal in FY 2001 will build on the results of this goal by targeting the reviewer pool. NSF will begin to request voluntary demographic data electronically from all reviewers to determine participation levels of members of underrepresented groups in the NSF reviewer pool. A baseline for FY 2002 will be derived from this data.

__________

8 Inspector General letter to the NSF Director, "IG's Statement Concerning NSF's Most Serious Management and Performance Challenges," January 4, 2001.

9 During FY 2000, the Office of Management and Budget revised the Federal goal, stating that 70-90% of research and development funds should be awarded to merit reviewed projects. Under the new definition, federally-funded research and development centers (FFRDCs) and merit-reviewed scientific research with competitive selection and internal (program) evaluation will not be considered merit reviewed. Taking into account the new definition, NSF has revised its goal for FY 2001 to 85%.