National Science Foundation |
|
| DATE: | March 6, 1997 |
| TO: | Paul Herer, Executive Secretary, NSB/Staff Task Force on Merit Review |
| FROM: | Susan Cozzens, Director, Office of Policy Support |
| RE: | ANALYSIS OF RESPONSES TO THE NSB/NSF REPORT ON MERIT REVIEW CRITERIA |
This memo provides a brief summary of comments received in writing on the proposed new generic review criteria for NSF awards. NSF solicited the comments through press coverage and through direct contacts among staff, universities, and professional associations. The proposed criteria were posted on the World Wide Web, with a response form to facilitate suggestions and reactions. An email address was also available. A number of comments also arrived in writing. The comments give the Task Force a large number of readings of the new criteria, but they by no means represent a random or representative sample of the research and education communities NSF serves. Though we kept careful track of them, and did some rough counts, which we present below, the numbers represent the result of a very uneven "poll" of the science and engineering community. A few groups were very active in getting members to respond, and some responses were in the name of several people, or an entire association. In addition, many of the responses omitted information about the employment or experience with NSF review of the respondent. Thus the numbers, per se, do not contain any precise information. More interesting were the various issues brought up by respondents. The analysis consists of two parts. To begin, we present some characteristics of the individuals who provided us with their views about the changes in the merit review criteria described in the report. Then, we discuss the views of those individuals about the changes in the criteria, and summarize some issues which were echoed in the comments of a number of the responses. (We considered 20 or so responses to be a significant number.) Characteristics of Individuals Who Responded There have been over 300 responses commenting on the report. About 95% of the responses were sent by e-mail. Most of the respondents were tenured faculty who had experience with the NSF merit review process. Over 90% of the respondents were individuals external to the NSF. About 80% of all respondents provided information about their employment. Of these, over 80% indicated they were employed by universities or colleges. The remaining were individuals scattered among other government, industry, non-profit institutions, other research institutions, etc. Of the respondents clearly from academe, about three quarters were tenured faculty. The others were split between non-tenured faculty and administrative/management personnel. Almost 60% of the respondents indicated some sort of experience with NSF review. (Many of the others no doubt also had such experience, but did not address that question in their responses.) Of these respondents, about 80% had submitted one or more research proposals, about 80% had served as mail reviewers, about a fifth had submitted an education proposal, and almost half had been a member of a review panel. About 15% were or had been members of advisory panels. Respondents' Views About Changes in the Criteria Over 60% of the respondents stated reasonably clearly whether they agreed with or disagreed with the proposed changes. Most respondents who gave an overall positive or negative opinion favored the change. Of the third who presented neither a negative nor positive view about the changes, about a tenth stated explicitly that the change would make no difference. There were groups of individuals of significant size (20 or more) who expressed views about specific changes in the merit review criteria. Individuals who agreed about a particular change often differed in their opinions about the overall merits of the changes. One underlying concern was that the new criteria would dilute NSF's emphasis on excellent science: |
|
|
|
| In criterion 2, a substantial fraction of the respondents indicated that the subelement, "How well does the activity broaden the diversity of participants?" was ambiguous to them. Some respondents also did not understand the meaning of the subelement, "Does the activity enhance scientific and technological literacy?"
A number of comments focused on how the criteria would be used. Many noted that the current wording of the subelements tends to encourage "yes-no" responses instead of more substantive opinions from the reviewers. A number of respondents wanted it made clearer to reviewers that proposals do not have to score well on every single sub-element of the criteria. Many commented that the "one size fits all" approach could be inappropriate, and that NSF may not be able to use one review form for every program. Another presentation issue frequently raised was the problem of getting reviewers to pay attention to the new criteria, or any criteria, since they are printed on the back of the form. The Web-based response form specifically asked whether the new criteria were likely to foster linkages or interdisciplinary research; the response was generally that they would not. A number of respondents were also concerned that the tradeoffs among costs, risks, and benefits were not adequately captured, and that the criteria were not explicit enough about encouraging innovation and creativity. |
|