Task Force on Merit Review (MR)
Charge
Background
All National Science Foundation (NSF) proposals, as part of the Merit Review process, are evaluated with respect to two equally important Merit Review Criteria—Intellectual Merit and Broader Impacts. The two-criteria system was instituted in 1997, replacing a four-criteria system in place since 1981, in which reviewers had evaluated researcher performance competence, intrinsic merit of the research, utility or relevance of the research, and effect on the infrastructure of science and engineering. The new system was implemented after careful study by the National Science Board /NSF Staff Task Force on Merit Review whose report, National Science Board and National Science Foundation Staff Task Force on Merit Review: Discussion Report, was released for community comment in November 1996, after which it was revised and approved by the Board in March 1997 At that time, a set of contextual elements was established for each of the two criteria and defined by questions to assist the reviewer in understanding their intent. These elements were seen as not necessarily relevant or complete for the evaluation of all proposals; other considerations may be important for the evaluation of some proposals. Additionally, reviewers were requested to address only those elements that they consider relevant to the proposal at hand and for which they feel qualified to pass judgment. The new system was communicated to the research community via Important Notice 121, New Criteria for NSF Proposals, (http://www.nsf.gov/pubs/1997/iin121/) on July 10, 1997 and implemented October 1, 1997.
Several years later, the NSB was requested by Congress to conduct a review of the NSF merit review process. The Board conducted the review and issued its report in September 2005, concluding that the NSF merit review process is fair and effective, and "remains an international 'gold standard' for review of science and engineering research proposals." In the report, the Board provided several recommendations for NSF to improve the transparency and effectiveness of the NSF merit review process, while preserving the ability of the program officers to identify the most innovative proposals and effectively diversify and balance NSF's research and education portfolio. (FY 2005 Report on the NSF Merit Review System: http://www.nsf.gov/nsb/documents/2006/0306/merit_review.pdf). In response to the Board's recommendations, NSF implemented an agency-wide effort to address quality of reviews, transparency of the award/decline decision, and support of transformative research. (Report to the National Science Board on the National Science Foundations' Merit Review Process FY 2005: http://www.nsf.gov/nsb/documents/2006/0306/merit_review.pdf)
Charge to the NSB Task Force on Merit Review
Five years have passed since the last review of the Merit Review process and a new National Science Foundation Strategic Plan will be issued shortly. Moreover, the current review criteria have now been in effect for over a decade, and in light of reports of some confusion in the field and inconsistency of their application and impact, it is timely for the National Science Board both to evaluate the current criteria with respect to their definitions and the way they are applied to the NSF portfolio of increasingly complex and interdisciplinary projects, and to ask whether the Merit Review process could be enhanced or modified, by clarifying or amending the statements of the Merit Review Criteria.
The NSB Task Force on Merit Review is hereby reconstituted at the February 3-4, 2010 National Science Board meeting. The Task Force is charged with examining the two Merit Review Criteria and their effectiveness in achieving the goals for NSF support for science and engineering research and education. This may include revising the merit review methodology, revising one or both of the merit review criteria and the way they are interpreted and applied, or the task force may find that the methodology and criteria are clear and function as intended with no further changes or action required.
The first steps are for the Board and NSF senior staff to identify relevant data sources, to define the issues, and outline possible options to make the use of review criteria more effective in meeting NSF’s mission. A work plan should be submitted to the Board at its May 2010 meeting with the goal of a report with policy recommendations at the May 2011 meeting. During that one-year period, the Task Force should solicit input widely from the research and stakeholder communities and may solicit special studies as appropriate.
Membership on the NSB Task Force on Merit Review are NSB members: Dr. Alan Leshner, Chairman, Dr. Ray Bowen, Dr. John Bruer, Dr. Esin Gulari, Dr. Lou Lanzerotti, Dr. Douglas Randall, Dr. Diane Souvaine, Dr. Thomas Taylor, and NSF Liaison members on the Task Force, Dr. Lance Haworth, Dr. Tim Killeen, and Mr. Jeff Nesbit.
NSB Task Force on Merit Review Work Plan
Process and Strategies
This work plan describes the process and strategies for gaining input from stakeholders regarding their understanding of the NSF merit review criteria as applied to the proposal and award process, analyzing and discussing the findings, and working with NSF leadership.
The stakeholder groups are both internal and external to NSF and mainly include research communities and their institutions (external) and NSF program officers (internal). The input gained from this study will inform the task force on how best to proceed with follow-up action, which includes detailing the findings, deliberating recommendations, discussing recommendations with NSF leadership, and working together to find the best solutions. This may include revising the merit review methodology, revising one or both of the merit review criteria and the way they are interpreted and applied, or the task force may find that the methodology and criteria are clear and function as intended with no further changes or action required.
The first steps are for the task force to identify relevant data sources, to define the issues, and outline possible options to make the use of the merit review criteria more effective in meeting NSF's mission.
The steps in the process are as follows:
- Determine the way the current criteria, and their instructions, are interpreted and utilized by both proposers and NSF program
staff.
- Survey program officers and NSF leadership - The goal of this internally-focused survey is to gain broad insight as to how program officers apply the criteria when 1) advising Principal Investigators on how they address the criteria within their proposals and 2) directing their reviewers in determining to what extent the criteria has been met within each proposal.
- Solicit and collect feedback from principal investigators and institutional research officials
- Integrate survey input, summarize findings and deliberate recommendations.
- Create draft recommendations and vet with NSF internal and external stakeholder groups that may include NSF staff,
university administration, policymakers, OSTP, and Congress.
- Produce and publish a Task Force Report disclosing the process, strategies, findings and recommendations for the NSF Merit Review process going forward. (Report due by May 2011.)
NSB Merit Review Task Force Timeline
| Date | Task | March 2010 | Task Force teleconference to discuss work plan and finalize draft Task Force charge |
|---|---|
 | |
| March – May 2010 | Task Force members develop a plan of action—determine the questions they want answered; the information necessary to attain the answers; and the means by which to gather the information |
| |
| May 4-5, 2010 | Task Force meeting at Board meeting to discuss next steps in proceeding with the internal and external survey (or study) |
| |
| May – August 2010 | Design and implementation of internal survey to program officers |
| |
| May – Dec. 2010 | Design, clearance, and implementation of external survey to Principal Investigators (Note: Approximately 6-9 months is needed for survey development and the required OMB clearance process for external surveys) |
| |
| August 25-26, 2010 | Task Force meeting at Board meeting to discuss progress on the surveys and results of the internal survey to NSF program officers |
| |
| August – Sept. 2010 | Review and compile findings from internal survey |
| |
| September 2010 | Offsite Board meeting/Informal discussion of progress |
| |
| Sept. – Dec. 2010 | Complete analysis of internal survey findings and begin to formulate recommendations |
| |
| Dec. 1-2, 2010 | Task Force meeting at Board meeting to review and discuss results of external survey |
| |
| Dec. – Feb. 2011 | Complete analysis for external survey, form draft recommendations and vet recommendations with NSF internal and external stakeholders |
| |
| Feb. – May 2011 | Draft final report with findings and recommendations for merit review criteria |
REVIEW-IN-BRIEF OF THE CURRENT CRITERIA
Intellectual Merit
A critical criterion for NSF’s funding of research has been the proposed project's intellectual merit, both in overall quality and in significance to the broader field. A concern has arisen over the past few years, however, that the current system is missing the importance of some more transformative (often also called high-risk, high-payoff) research and that the system has become a bit more conservative as funds have become more constrained, despite efforts by NSF to emphasize transformative research. (See Important Notice No. 130: Transformative Research, Sept. 2007: http://www.nsf.gov/pubs/2007/in130/in130.jsp). In light of the report of the NSB's Task Force on Transformative Research and the recommendation of the Companion Piece to Science and Engineering Indicators 2010, Globalization of Science and Engineering Research (NSB 10-3), namely "The National Science Foundation should assess its two criteria for funding of S&E research to ensure that the criteria encourage the proposing and support of truly transformative research, and should modify the criteria and/or merit review process if the assessment finds modifications necessary to accomplish this goal," special consideration should be given to whether the current criteria and their implementation are accomplishing the goal of supporting the best of all kinds of research.
Broader Impacts
The Broader Impacts criterion identifies the important outcomes and consequences of NSF-supported research. Anecdotal evidence suggests that this requirement can be very confusing to the research community, which continues to express frustration in interpreting and thus responding effectively to the Broader Impacts criterion when creating a proposal. In July 2007, Merit Review Broader Impacts Criterion: Representative Activities (http://www.nsf.gov/pubs/gpg/broaderimpacts.pdf) was made available to PIs in the Grant Proposal Guide, which includes examples of ways that broader impacts could be incorporated into research projects. These examples are quite diverse but that diversity can also make them confusing to proposers and to NSF program staff attempting to address the Broader Impacts criterion in the review and decision process. There also is concern that these examples can appear to be directive yet are not fully inclusive. For example, they do not fully reflect the importance of impacts on such issues as innovation, national security and economic growth. Finally, there appears to be substantial confusion about how best to meet the requirements of this criterion, whether on an individual project level or at the proposing institution level.
Task Force Members
| Co-Chairmen: | Dr. Alan I. Leshner |
| Members: |
Dr. Douglas D. Randall Dr. Diane L. Souvaine |
| Consultant: | Dr. Louis J. Lanzerotti |
| NSF Members: | Dr. Timothy L. Killeen |
| Executive Secretary: | Dr. Joanne S. Tornow |
NSBO Liaison: |
Ms. Kim Silverman |