Contract No.: DOB-01-33044
Report on Efficiency of
Grant Size and Duration
Principal Investigator
FY 2001 Grant Award Survey
and Institutional Survey
July 2002
Janice
Ballou
Matthew
Mishkind
Geraldine
Mooney
Welmoet
van Kammen
Submitted to: National Science Foundation Chief Budget and Systems Operations and Budget Branch 4201 Wilson Boulevard, Room 407N Arlington, VA 22230 Project Officers: Norman Bradburn Bernard McDonald |
Submitted by: Mathematica Policy Research, Inc. P.O. Box 2393 Princeton, NJ 08543-2393 (609) 799-3535 Project Director: |
Mathematica Policy Research, Inc. (MPR) would like to thank the staff at the National Science Foundation (NSF) who provided guidance and resources for this survey. Specifically, Robert Abel (Contracting Officer’s Technical Representative [COTR]), Norman Bradburn, Bernard McDonald, and Vernon Ross gave their assistance during this project. At NSF, Kimberly Deutsch and Elizabeth Velo also contributed to the project. MPR would also like to thank all the Principal Investigator 2001 Grant Award Survey and the Institutional Survey participants. Without their cooperation and thoughtful responses this report would not be possible.
At MPR Frank Potter was the sampling statistician, and John Hall provided statistical assistance. Melissa Wood, David Frank, Irene Crawely, and Susan Golden assisted with the data collection and quality assurance. Programming for the Web survey and data analysis was supervised by Anne Hower; Ron Palanca, Eric French, Scott Reid, Shayne Walcott, Neil deLeon, and Linda Gentzik assisted with the programming and data analysis. Roy Grisham edited the report and Gloria Gustus and Rosita Turkel prepared the final report.
CONTENTS
I. INTRODUCTION AND BACKGROUND
D. ORGANIZATION OF THIS REPORT
E. OTHER CONTEXTUAL CONSIDERATIONS
II. PROFILE OF 2001 NSF FUNDED GRANT PROPOSALS
A. PRINCIPAL INVESTIGATOR CHARACTERISTICS
C. AMOUNT AND DURATION OF FUNDING
F. PROFILE OF INSTITUTION GRANT MANAGEMENT
A. LEVEL OF PRINCIPAL INVESTIGATOR PROPOSAL EFFORT
C. LEVEL OF INSTITUTIONAL PROPOSAL EFFORT
IV. MEASURES OF AWARD EFFICIENCY AND EFFECTIVENESS
A. DIFFERENT MEASURES OF AWARD EFFICIENCY
AND
1. Award Efficiency and Effectiveness: Deviations from Requested
2. Award Efficiency and Effectiveness: Percent of Research Being
3. Award
Efficiency and Effectiveness: Additional Requirements
4. Award Efficiency and Effectiveness: NFS’s Contribution
B. SUMMARY OF DIFFERENT METHODS TO ESTIMATE AWARD
C.  PRIORITY ACTIONS FOR AWARD EFFICIENCY AND
  EFFECTIVENESS: PI PERSPECTIVE
D. PRIORITY ACTIONS FOR AWARD EFFICIENCY AND
EFFECTIVENESS: FIELD OF RESEARCH
V. POTENTIAL IMPACT OF
INCREASED FUNDING OR DURATION
A. THE CONSEQUENCES OF REDUCTIONS IN FUNDING OR
B. THE POTENTIAL BENEFITS OF ADDITIONAL NSF FUNDING OR
DURATION ON THE PI’S CURRENTLY ONGOING BODY OF
C. THE POTENTIAL BENEFITS OF ADDITIONAL NSF FUNDING OR
DURATION ON THE RESEARCH IN GENERAL
D. THE POTENTIAL IMPACT OF ADDITIONAL NSF FUNDING OR
DURATION ON EDUCATIONAL ACTIVITIES
E. IMPACT OF CHANGES ON INSTITUTIONS
F. ADDITIONAL COMMENTS ABOUT THE NSF GRANT PROCESS
Appendix to the NSF Report on
Efficiency of Grant Size and
Duration
APPENDIX A: RESEARCH METHODOLOGY
APPENDIX B: ANNOTATED QUESTIONNAIRES
1. Principal Investigators
2. Institutions
APPENDIX C: NON-SAMPLE INSTITUTION SURVEY INFORMATION
APPENDIX D: VERBATIM RESPONSE CODING FRAME
1. Principal Investigators
2. Institutions
APPENDIX E: PRINCIPAL INVESTIGATOR SURVEY CONTACT
INFORMATION
1. Principal Investigators
2. Institutions
APPENDIX F: INSTITUTIONAL SURVEY CONTACT INFORMATION
APPENDIX G: STATISTICAL TABULATIONS
TABLES
II.1 PROFILE OF FY 2001 NSF GRANT PRINCIPAL INVESTIGATORS
II.2. PROFILE OF FY 2001 NSF GRANT AND OTHER FUNDING
II.3 TYPE OF FY 2001 NSF GRANT BY DIRECTORATE
II.4 PROFILE OF INSTITUTION LEVEL OF GRANT EFFORT
III.1 PI LEVEL OF PROPOSAL EFFORT
III.3 INSTITUTIONAL LEVEL OF EFFORT FOR TYPICAL FY 2002 NSF
IV.1 PROFILE OF AWARD EFFICIENCY AND EFFECTIVENESS
IV.2 AWARD EFFICIENCY AND EFFECTIVENESS FUNDING AMOUNT
TO ACCOMPLISH A 100 PERCENT IN FIVE YEARS
IV.3 MOST IMPORTANT NSF AWARD REFORM FOR PI AREA OF
V.1 NEGATIVE IMPACT OF FUNDING AND/OR DURATION AWARD
V.2 IMPACT OF CHANGE IN AWARD FUNDING AND/OR DURATION
V.3 POSITIVE IMPACT OF ADDITIONAL AWARD FUNDING AND/OR
V.4 IMPACT OF ADDITIONAL FUNDING AND/OR DURATION
V.5 ADDITIONAL NSF AWARD FUNDING AND/OR DURATION:
POTENTIAL INCREASES IN RESEARCH AND EDUCATIONAL
V.6 ADDITIONAL NSF AWARD FUNDING AND/OR DURATION:
INCREASED ABILITY FOR RESEARCH AND EDUCATIONAL
V.7 ADDITIONAL NSF AWARD FUNDING AND/OR DURATION:
POTENTIAL FOR CHANGES IN FIELD OF RESEARCH
V.8 INSTITUTION CHANGES IF NSF INCREASED THE AVERAGE
V.9 INSTITUTION CHANGES IF NSF INCREASED THE AVERAGE
V.10 SUGGESTIONS FOR NSF CHANGES TO REDUCE INSTITUTION
V.11 PRINCIPAL INVESTIGATORS: OTHER COMMENTS ON THE
V.12 INSTITUTIONS: OTHER COMMENTS ON THE NSF GRANT
FIGURES
II.1 CURRENT STUDENT SUPERVISION AVERAGE
NUMBER OF STUDENTS AND POSTDOCTORALS
IV.1. AWARD FUNDING OPTIONS ANNUAL PROJECTED
IV.2 AWARD FUNDING OPTIONS ANNUAL PROJECTED
IV.3 AWARD FUNDING OPTIONS ADDED TO FY 2001
EXAMPLES OF TOTAL PER YEAR FUNDING
IV.4 CURRENT OTHER NSF FUNDING
IV.5 CURRENT NON-NSF FUNDING (Means)
This Executive Summary provides an overview
of the key results of two surveys.
Readers are encouraged to read
the full report for more comprehensive information and additional details on
the survey methodology.
Overview of Reasons for Conducting the
Survey
The National Science Foundation (NSF) is responsible for advancing science and engineering in the United States. About 50 percent of the federal nonmedical fundamental research at U.S. colleges and universities is from NSF’s support of nearly 200,000 scientists, engineers, teachers, and students each year. Thus, the research productivity and research training of the U.S. scientific and engineering community depend heavily on NSF funding. However, a key question that needs to be answered is: What constitutes the right size NSF grant? In addition, there is the related question: What needs to be done to improve NSF award efficiency and effectiveness?
Survey Design
The survey was designed to provide an opportunity for two key participants in the NSF grant process to answer these questions: (1) the Principal Investigators (PIs) who receive grants, and (2) the institutions that assist in preparing and administering NSF grants. In February and March 2002, surveys were conducted with each of these two groups by Mathematica Policy Research, Inc. (MPR). The study design was to (1) conduct a census of principal investigators who had received NSF grants in FY 2001, and (2) include a sample of the institutions these recipients represented. The results of this study describe the key issues NSF is considering from the perspective of these two key groups. This information will help guide NSF as it makes strategic decisions about the right amount of its grants and their duration.
Overview of Principal Investigator
Characteristics
In order to have a context for understanding the answers to the key questions described above, it is important to have some background information about the FY 2001 NSF grant recipients. While the PIs have numerous diverse characteristics, the following will provide a general profile of who received NSF grants in FY 2001. The professional age of these PIs—defined as the years since their last degree—is distributed as follows: 10 years ago or less (30%), 11 to 20 years (34%), and more than 20 years ago (36%). There is variation in the number of graduate students they supervise—with 40 percent currently working with two or fewer graduate students, 25 percent with three to four, and 27 percent with more than four. The PIs describe the focus of their NSF research project as follows: laboratory (44%), theoretical (37%), field (18%). PIs currently have various sources to fund their research. Nineteen percent have only this NSF grant, compared to 9 percent who have an additional NSF grant, 37 percent who have non-NSF grants, and 35 percent who have both an additional NSF grant and non-NSF support.
Level of Proposal Effort
One measure of the use of resources that could
be improved is the proposal-preparation effort: What is the extent of the
resources that PIs and institutions might save if NSF grant awards were more
efficient and effective? A PI who is on 100 percent soft money
provided a description of the level of proposal effort needed to get sufficient
support:
“For
a soft money researcher needing to cover 12 months of his/her time and current
budget standards pushing for 3 months (or less) coverage for senior personnel
per proposal (seemingly, this is only communicated obliquely) and an average of
three year research duration this results in obtaining four funded proposals
per three years. This would be a
limited time sink if success was 100% (4 weeks/year of proposal writing at
three weeks/proposal). At 50% it is
eight weeks/year writing proposals. At
33% success it is 12 week/year, and so on.
It seems that a better balance can be found. Consider that many programs have funding rates of 10-33 percent.”
Specifically, PIs’ estimate of the time it took for them and other people—for example, graduate assistants, budget administrators, and secretaries (not including time spent by institutional personnel)—to prepare their FY 2001 NSF grant submission was, on average, 157 hours, or about 19.5 days. It should be noted this is the time for just one proposal that was successful.
When PIs lack sufficient funding for their research and educational activities, they need to prepare and submit multiple grants. For example, 38 percent of the PIs reported that they divide their ongoing body of research and educational activities into several proposals and submit them to NSF; as described earlier, overall, about 8 of 10 PIs currently have funding in addition to the FY 2001 NSF grant identified for this survey.
Award
Duration and Funding Efficiency and Effectiveness
A key objective of the PI survey was to find out: What are the award durations and funding amounts that are the most efficient and effective in promoting NSF’s objectives? The answer to this question is not straightforward. NSF meets multiple objectives when it awards grants. These multiple needs increase the challenge of establishing criteria in order to achieve award efficiency and effectiveness. Adding to this already complex task is the nature of scientific inquiry as described by one PI:
“The very nature of research and
investigation of hitherto unexplored ideas makes it difficult to know or state
future requirements. More funding
generally means more personnel to help perform the intended research. However, often new and unforeseen
opportunities for new research arise as a consequence of new stimulus and ideas
in performing the currently-funded research.
This is always difficult to assess ahead of time. It is what makes research exciting!!”
Award Duration
The information from the PIs suggests a consensus on an increase in award duration. Currently, the average grant duration for all PIs is three years. When the additional years the PIs suggest they would like are added to each current award, the average is five years. Five year awards will give PIs the most effective period of time for their research and educational activities. This period has such benefits as continuity of employment for students, particularly graduate student staff, opportunities to explore other areas of inquiry that may develop as they conduct their research, higher quality of research, and less time spent pursuing additional funding.
Award Amount
There are multiple ways to consider the appropriate amounts needed to provide efficient and effective grant awards. The information provided by the PIs can be used by NSF to consider funding needs in a variety of ways. The following are four examples of possible ways to estimate award efficiency and effectiveness from the PIs’ perspective that have a range from an average of $40,000 to $230,000.
Option 1:
Award Effectiveness and Efficiency—Deviations from FY 2001 Grant Request
At
the most basic level, award effectiveness and efficiency can be defined as the
PIs’ actual experience of a reduction in funding and/or duration—from what they
requested to what NSF awarded. In
addition, it should be noted that the funding requests PIs make in their proposals
prior to receiving any reductions may already be at the lower limit of what
they need for their research and educational activities. For example, PIs who resubmit proposals that
were previously rejected, request less funding than those who are submitting
proposals for the first time.
In
FY 2001, 51 percent of the PIs had a 5 percent or greater decrease in
funding. If award efficiency and
effectiveness are defined as providing all PIs what they request, what would be
the additional funding that is needed?
In FY 2001, these PIs requested an overall average amount of $436,000
for an average of three years and received an average of $336,000 over three
years. A calculation for each PI that
subtracts the actual award from the request, and averages the differences,
finds $40,000 additional annual funding per grant would be required. The duration of the grant would remain the
same at three years using this same type of calculation for award duration.
Option 2: Award
Efficiency and Effectiveness—Percent of Research Being Funded
A second way to estimate award efficiency
and effectiveness is: What percentage
of the PIs’ ongoing body of research and educational activities is being funded
by the FY 2001 NSF grant, and what is
the amount needed to fund 100 percent?
Since PIs view their research and educational activities as being fluid,
to provide standard criteria to answer this question, the total of what they
would like to accomplish was established as 100 percent, and the time period
for this accomplishment was given as five years. Using these criteria, PIs
estimate that an overall average of 37 percent of the ongoing research and
educational activities that they would like to accomplish in the next five
years will be achieved with the FY 2001 NSF grant. Calculating this for each PI, the average additional funding
needed would be $181,000 per year.
Adding this to the FY 2001 average annual amount increases the possible
funding to $293,000.
Option
3: Award Efficiency and
Effectiveness—Additional Requirements
A third approach to estimating award
efficiency and effectiveness is: What
additional funding do the PIs estimate is needed to accomplish their key
goals? PIs were given the following
guidelines to develop their estimates: (1) the additional duration should not
include the years of funding they have in the FY 2001 NSF grant, and (2) the
additional funding estimate is based on their needs for the next five years and
should not include the current funding they have from NSF or from any other
sources. The average funding that PIs
project they will need is an additional $230,000 annually. It is important to emphasize that this
amount is not all the PIs require; it is an additional amount they would add to
the funding they currently have from NSF and non-NSF sources, which they anticipate
will enable them to achieve their five-year goals. When the current average annual award amount of $112,000 is added
to what the PIs would like to have in additional funding, the average annual
amount needed is $342,000.
Option
4: Award Efficiency And Effectiveness:
NSF’s Contribution
Another question
related to NSF modifications in award efficiency and effectiveness is: Should NSF be responsible for funding all
PIs research and educational activities?
As defined by the focus group participants, NSF grants are distinguished
from others because they do not have to be “mission-directed.” Therefore, PIs may not expect NSF to fund
all aspects of their research and educational agendas. PIs estimate of the amount of additional
funding they need in the next five years that is appropriate for NSF to fund is
an average of 67 percent. As described
in Option 3, the average annual amount of additional funding PIs require is
$230,000. PIs expect NSF to fund an
annual average amount of $135,000. This
amount is calculated by multiplying the additional funding each PI would like
over the next five years by the percentage each PI expects NSF to fund. If this additional amount is added to the
current FY 2001 amount, the PIs would require an annual award of $247,000.
It should be noted that, for each option, there can also be variation in the amount needed for award efficiency and effectiveness, depending on the particular characteristics of the PI, such as the type of research being conducted which NSF can consider. For example, over the next five years, PIs who have theoretical research projects expect to need less funding than those conducting field and laboratory research.
Although determining those changes that will best meet the needs of the country’s scientific community is complex, there are clearly benefits to making changes. As summarized by one of the PIs:
“As I said above, I think NSF is the
greatest. But the size of the
individual grants is just too small to really accomplish anything significant
in my field. At [name of institution]
200K per year will get you one post-doc and one student, and some materials and
supplies and that is it. So you need at
least three of these grants to run a decent sized group. This is a lot of grant-writing. I wish the NSF budget could be multiplied by
10. Then we might have something to
work with.”
Benefits of Changes in NSF Funding and Duration
How might NSF changes in their grant-funding levels and/or duration influence ongoing research and educational activities in this country? This question can be answered from two perspectives. First, by assessing the impacts reported by PIs whose FY 2001 NSF grant proposal was funded, but the amount and/or duration was altered; and, second, by PIs’ speculation on how their research and educational activities might be affected if NSF were to increase its grant funding and/or the duration of its grants.
Using a list of 18 areas where changes in funding and/or duration potentially could affect a research project, PIs identified what they are experiencing as a result of award cuts. Following are the key negative impacts they reported:
· Ability to achieve their research objectives within the specified time (67%)
· Ability to obtain quality personal (55%)
· Ability to pursue high-risk ideas (51%)
· Collaborate with researchers in their area of research (50%)
One PI summarized what happened as a result of a decrease in the FY 2001 award:
“A major impact of the shortened funded
period was the lack of continuity. It
had a dramatic effect on training students and attracting new students or
postdocs. One cannot plan or attract
good students and postdocs if you only have 1.5 yr of funding. The best training and research is
accomplished when you have a dynamic group of people interacting. The synergisms are incredible. To do this one needs to be able to
constantly recruit new people and this cannot be done without some guarantee of
support.”
Using the same 18 areas, the PIs speculated on the positive consequences if they were provided additional funding and duration. The following are the most broadly perceived benefits:
· Ability to pursue high-risk ideas (96%)
· Collaboration with researchers in area of research (92%)
· Ability to achieve research objectives within the specified time (92%)
When asked how they would spend additional funding for securing the resources needed to conduct high-quality research, “student support” is top among 16 resources presented to the PIs. These areas are where PIs are very likely to expend additional resources:
· Number and/or months of graduate students (78%)
· Number of experiments, tests, subjects (54%)
· Number and/or months of undergraduate students (50%)
· Number and/or months of postdoctoral associates (48%)
To summarize, support for students is a major focus of what PIs view as a benefit of increases in NSF funding and/or duration. A change in funding that gives PIs what they request in their grant proposals would provide an additional average amount of $40,000 per year. At many schools, this difference alone could support at least one additional graduate student. In addition to improving PI productivity, hiring additional students has the benefit of funding highly qualified students whose talents would be lost if they have to leave the sciences to support themselves in alternative careers.
From
the PIs’ perspective, it is not only the additional funding and duration that
is attractive, it is also NSF’s unique contribution to the development of
students.
“Clearly
more money helps, but it is the TYPE of funding that NSF provides that is
unique. So many other sources
(including federal agencies) have a short-term focus on technological or
economic deliverables More NSF funds opens up a spectrum of possibilities: working on
new ideas, allowing students to drive some of the research, building
infrastructure and a base from which more funds can be obtained from other
sources, more freedom to focus of graduate and undergraduate educational goals
rather than "research" goals.
The NSF is different than industry, and needs to stay that way; and it
needs to be a bigger part of academic funding.”
“My main concern with
NSF is the 3 year cap on funding for grants as I mentioned before This 3 year
cap is not enough to have a student finish a project. Even if the project is renewed, then the renewal often does not
come in time to support a graduate student’s stipend working on the original
project. As a result, the student has
to TA or find a scholarship or just borrow from other grants for obtaining
money to finish and graduate.
Consequently, I would be very much in favor of extending grants to 4
years rather than 3. If this reduces
the award amount slightly, so be it.
But I do not think that has to be true since people would just get fewer
grants but for longer
periods of time. Since NSF is in the
business of education scientists and engineers, I would think that NSF would be
most interested in obtaining continuous funding for our hard-working graduate
students. These students often
sacrifice personal rewards for the opportunity to do research and we shouldn’t
punish them by making them go get more money to finish out their thesis since
the NSF grant is out after 3 years.”
Benefits to the PI’s Field of Research
In addition to the individual benefits described above, PIs reported that increased NSF funding or duration had broader implications for their field of scientific research and education. Among the top four areas most likely to be affected if NSF provides additional resources, two focus on students. Following are the key changes expected in the PIs’ field:
· Decrease interruptions in funding (70%)
· Attract more graduate students (65%)
· Widen the focus of research in the field (63%)
· Attract better graduate students (62%)
Summary
The results of the surveys indicate that, at the present time, NSF is not meeting the needs of a diverse group of researchers and research and educational activities. PIs underscore the benefits to education that can be achieved by NSF award changes. In particular, awards of longer duration will provide continuity and avert the problems associated with the disruption of graduate student education. Additional funding can provide important benefits such as the PIs pursing their more innovative ideas and reducing their need to spend time preparing multiple proposals. These PIs send a clear message that there is a need to make changes to improve award efficiency and effectiveness. How that need can be met is more complex. Just as there is variation in the types of funding NSF currently provides, there are choices about how to maximize award efficiency and effectiveness.
In June 2001, the National Science Foundation (NSF) received instructions from the Office of Management and Budget (OMB) to prepare for the FY 2003 Budget by determining the right size grant for “the myriad types of research the agency funds.”[1] Subsequently, OMB and NSF recognized that the process of meeting OMB’s objective could not be completed in time for consideration in the FY 2003 Budget; so the goal is to provide information that can be used to inform the development of the NSF FY 2004 budget.
To provide information that will help answer OMB’s primary question of what constitutes the right size grant needed, as well as the related issue of what needs to be done to improve award efficiency and effectiveness, NSF contracted with Mathematica Policy Research, Inc. (MPR) to conduct surveys with two key participants in the NSF grant process: (1) the Principal Investigators (PIs) who receive grants, and (2) the institutions that assist in preparing and administering NSF grants. This report presents the results of those surveys.
Through the support of nearly 200,000 scientists, engineers, teachers, and students each year, NSF supports about 50 percent of federal nonmedical fundamental research at U.S. colleges and universities. Thus, the research productivity and research training of the U.S. scientific and engineering community depend heavily on NSF funding.
The concept of “fully-enabled” investment levels was used in the early discussions about research funding changes that took place in the focus groups and the cognitive pretest conducted for this study. The three key objectives for fully enabled awards were to: (1) provide researchers and institutions sponsoring the research with resources adequate to complete the work for which the grant was awarded, (2) address the efficiency of the proposal and award processes, and (3) increase the awards’ benefits and outcomes to the nation. These surveys were commissioned to assist NSF in understanding these objectives from the perspective of the communities that receive funding.
While the objectives of what NSF wants to achieve are clear, identifying a term that summarizes these objectives is more complicated. The PIs who were selected to assist in the development of the questionnaire agreed that the concept, fully enabled, did not accurately describe the goals of their grant awards. These PIs described their research as an “ongoing body of research,” not a finite project that had a specific product or end that could be fully enabled. In the survey, a PI summarized the difficulty of describing “fully enabled”:
“The very nature of
research and investigation of hitherto unexplored ideas makes it difficult to
know or state future requirements. More
funding generally means more personnel to help perform the intended research. However, often new and unforeseen
opportunities for new research arise as a consequence of new stimulus and ideas
in performing the currently-funded research.
This is always difficult to assess ahead of time. It is what makes research exciting!!”
Since the PIs assisting with the questionnaire rejected the concept of fully enabled, for the purpose of this report, the NSF goal for changes in the funding structure in both size and duration of awards will be referred to as award efficiency and effectiveness. This description of awards has also been referenced in other NSF reports. As noted in the Report to the National Science Board on the National Science Foundation’s Merit Review Process Fiscal Year 2001: “Larger awards increase the efficiency of the system by allowing scientists and engineers to devote a greater portion of their time to actual research rather than proposal writing and other administrative work” (page 8). And, “Longer award terms are important in increasing the effectiveness of principal investigators and graduate students” (page 9). Although this example uses these concepts to refer to the system, and not to individual grants, it can also be applied to the experiences of individual PIs. Throughout the report, the term “award efficiency” will be used to summarize the PIs’ experiences that are related to changes in NSF funding and “award effectiveness” that will be used to summarize the experiences related to grant duration.
As noted above, this research project included two surveys: the Principal Investigator FY 2001 Grant Award Survey (PI Survey) and the Institutional Survey. The research design for the PI Survey called for conducting a census of all PIs included in NSF’s data file of recipients of grants in FY 2001.
A questionnaire for
the PIs was developed that uses information from two focus group discussions
conducted by MPR in August 2001, with 23 NSF representatives. Cognitive pretesting of the initial draft
questionnaire was completed in December 2001, with eight principal
investigators from grants that year. Two
questionnaire modes were developed: a Computerized Self-Administered
Questionnaire (CSAQ) for use on the World Wide Web, and a mail
questionnaire. A copy of the
questionnaire is included in Appendix B.
Invitations to participate in the survey were sent by e-mail beginning
January 30, 2002, and PIs had an opportunity to respond until March 8, 2002.
The total number of PIs invited to participate was 5,793 (PIs with multiple awards were selected for their response to only one grant), and 5,221, or 91 percent, returned a questionnaire. Among these returns, fewer than 1 percent were completed by mail. The data analysis file includes 4,989 PI questionnaires that met the quality assurance completion criteria. These
criteria included such standards as providing a response to key questions and having appropriate information about the FY 2001 grant. Appendix A contains a detailed description of the survey process.
The Institutional Survey research design defined the universe as the 582 institutions that awarded NSF grants to PIs in FY 2001. The questionnaire for this survey was developed from individual interviews conducted with institutional representatives and with cognitive pretesting of the survey instrument. All institutional representatives were invited to complete a self-administered mail survey. As part of the study design, among all these institutions, a representative sample of 105 institutions was selected to be targeted for data collection and analysis. Among all 582 institutions—359, or 62 percent—returned questionnaires. In the sample—95, or 90 percent—completed questionnaires. The institutional analysis reported here is based on information from the sample. It should be noted that the results from the sample institutions have been statistically weighted to ensure that they are representative of the universe of institutions. The appendices provide a complete description of the survey methodology, the weighting procedure, the standard errors, and a copy of the questionnaire.
Information from the non-sample institutions is not included in the text of the report. Appendix C has information for the 264 non-sample institutions that participated in the survey.
It should be noted that information from NSF’s grant data file for FY 2001 for the PIs and the institutions is also used in this report.
In addition to this report, readers should be aware that Ruts in “The Royal Road” written by Deborah Shapley, also describes information from the PI survey. It provides additional information about the PIs’ qualitative responses to the open-ended questions and their perspectives on possible NSF award changes. It should be noted that the verbatim comments included in both reports are used with permission from the PIs.
The goal of this report is to inform NSF’s effort to improve award efficiency and effectiveness. Chapter II provides a profile of NSF grant receipt characteristics for FY 2001, as well as giving an overview of the grant experience from the perspective of the PI and the institution. This will give readers a context for the information that is reported in the following chapters. Chapter III describes the NSF grant proposal process for FY 2001 as experienced by the PIs and institutions. It provides insights into the level of effort needed to compete for an NSF grant. The multiple dimensions of award efficiency and effectiveness are outlined in Chapter IV. Chapter V summarizes the costs and benefits of potential changes in NSF awards.
Following are specific terms that will be used throughout the report:
PI: Refers to the Principal Investigators who received NSF grants in FY 2001.
Award efficiency and effectiveness: As described above, this refers to the broad NSF objective of identifying the changes that are needed to ensure grant awards provide adequate resources for PI research and educational activities. In addition, for this report, the term focuses specifically on PIs’ perceptions of the resources needed to achieve their key research and educational goals.
Institutions: Refers to the institutions that had PIs who received FY 2001 NSF grants.
In addition, it should be noted that the text and the tables in this report that refer to the NSF directorates do not include two that had a small number of PIs in the classification: (1) Education and Human Resources Directorate (16 PIs) and (2) the directorate (O/DD) that includes primarily the Office of Polar Programs (130 PIs).
Readers should note that percentages may not always add to 100 because they have been rounded to the nearest whole number and means are also rounded to the nearest whole number. Percentages are reported for all survey participants unless specific subgroups are referenced in the discussion. In addition, means[2] are used to report numeric results, and dollar amounts are reported to the nearest 1,000.
There are other considerations readers should be aware of as they review the information from these two surveys. The competitiveness of the grant process, the range of economic and administrative challenges being confronted by academe, and the salaries that need to be supported are examples of the situational factors that have an impact on PIs and institutions as they consider the issues addressed in this survey that are related to NSF reforms in award efficiency and effectiveness.
The information provided in this report is limited to those who received NSF grants in FY 2001. During FY 2001, NSF took action on 31,942 reviewed proposals and funded approximately 31 percent. NSF estimates that approximately $1.25 billion was requested for declined proposals that received at least as high as the average rating for an awarded proposal. (Report to the National Science Board on the National Science Foundation’s Merit Review Process Fiscal Year 2001.)
Higher education is experiencing many changes that can have an impact on both faculty and institutional resources. In particular, reduced budgets can result in institutions providing less support for research and graduate education, as well as an increase in the number of faculty expected to provide substantial portions of their own research funding. These types of dynamics increase the risk of cutbacks in research that can affect the scientific community.
When considering funding amounts for award efficiency and effectiveness, a key question is: How many professional staff and students can be supported? One approach to estimating the personnel costs for research and educational activities is to know the range of typical faculty salaries and fees for graduate students. According to the College and University Professional Association for Human Resources (CUPA-HR) salary survey for FY2000-FY 2001, the average faculty salary at a private institution is $58,700, and $59,123 at a public institution (Chronicle of Higher Education, July 27, 2001). However, depending on the specific discipline, it may be lower or higher. For example, the salary for bioengineering and biomedical engineering is $79,857 (private) and $72,250 (public), chemical engineering is $82,878 (private) and $82,254 (public), and for the social sciences the average faculty salary is $49,894 (private) and $54,560 (public).
Stipends for graduate assistants vary (Chronicle of Higher Education, September 28, 2001). However, a typical NSF fellowship of $21,000 provides a standard to use when calculating the fees associated with staffing an NSF grant project in FY 2001. Concerns about having enough funding to support a research staff and the consequences are summarized by a PI:
“As I said above, I think NSF is the
greatest. But the size of the
individual grants is just too small to really accomplish anything significant
in my field. At [name of
institution], 200K per year will get you one post-doc and one student, and
some materials and supplies and that is it.
So you need at least three of these grants to run a decent sized
group. This is a lot of grant-writing. I wish the NSF budget could be multiplied by
10. Then we might have something to
work with.”
NSF is known for providing awards that meet the needs of principal investigators (PIs) with diverse backgrounds and that support a range of varied research and educational activities. We use information from the FY 2001 NSF grantee database and from the survey to give a description of the PIs’ characteristics and an overview of the NSF grant awards for FY 2001 and other funding currently supporting PIs’ research. A discussion of these characteristics will assist in understanding the different experiences that can influence perceptions of award efficiency and effectiveness.
PIs have a range of experience in being the primary author on peer-reviewed articles. Over the past five years, 31 percent have published more than 13 peer-reviewed articles; 30 percent have published 7 to 13 articles; and 36 percent report being the author of 6 or fewer. PIs conduct their research in the following types of institutions: 25 percent are from one of the top 20 NSF-funded institutions, 26 percent from the top 21 to 50 NSF-funded institutions, and 22 percent from the top 51 to 100 NSF-funded institutions. It should be noted that these categories of funding are based on the total amount an institution receives from NSF, not just the amount from these individual grant awards.
TABLE II.1
PROFILE OF FY 2001 NSF GRANT PRINCIPAL INVESTIGATORS
(Percentages)
N =
4,989
Professional Age: Date of Last Degree |
|
10 years or less |
30 |
11 to 20 years |
34 |
More than 20 years |
36 |
Publication
Experience: Number of Articles in the
Past 5
Years |
|
Low (6 or less) |
36 |
Medium (7-13) |
30 |
High (more than 13) |
31 |
PI Student Supervision |
|
Undergraduates |
|
Low (1 or less) |
41 |
Medium (2) |
18 |
High (more than 2) |
24 |
Graduates |
|
Low (2 or less) |
40 |
Medium (3-4) |
25 |
High (more than 4) |
27 |
Postdoctural |
|
Low (none) |
33 |
Medium (1) |
25 |
High (more than 1) |
22 |
Type of Institution |
|
NSF funded top 20 |
25 |
NSF funded top 21 to 50 |
26 |
NSF funded top 51 to 100 |
22 |
Other Ph.D. |
18 |
Non-Ph.D. |
5 |
Non-Academic |
4 |
Gender |
|
Male |
83 |
Female |
17 |
Race |
|
White |
85 |
Non-white |
15 |
Most PIs supervise undergraduate and graduate students, as well as postdoctoral fellows. The average number of undergraduate students assisting PIs with their current research projects is two, graduate students four, and postdoctoral fellows one (Figure II.1). More than half of the PIs supervise three or more graduate students, compared to 22 percent of PIs who have more than one postdoctoral student working with them. With respect to individual characteristics, 8 of 10 PIs are white and male.
Seventy-one percent
of the FY 2001 awards were reported as first-time submissions of this grant to
NSF. Among the NSF directorates, the largest percentage of the awards (26
percent) were funded in Mathematical and Physical Sciences (Table II.2). We asked the PIs to classify the research
for which they were funded in one of the following categories:
Theoretical research can be accomplished with minimal physical resources beyond the investigator’s institutional research library, computing capability, and office space.
Laboratory research requires an equipped laboratory, for example, research often found in chemistry, biology or engineering university laboratories requiring research and/or testing equipment, plumbing.
Field research requires fieldwork, specimen collection, sample survey, location of sensors, etc. away from the principal investigator’s institution, for example, some science activities in geosciences, biology, social sciences.
Among these classifications, 44 percent of the PIs categorized their FY 2001 NSF grant as laboratory research, 37 percent as mostly theoretical, and 18 percent as field research.
Table II.3 shows the proportion of grants in each research category for the different directorates at NSF. Not surprisingly, more than half of the grants awarded in the Computer and Information Science and Engineering (CISE) directorate and the Mathematical and Physical
TABLE II.2
PROFILE OF FY 2001 NSF GRANT AND OTHER FUNDING
(Percentages)
N=4,989
|
|
TABLE II.3
TYPE OF FY 2001 NSF GRANT BY DIRECTORATE
(Percentages)
|
Type of Research |
||||
|
Theoretical Research |
Laboratory Research |
Field Research |
Total |
(N) |
Total |
37 |
44 |
18 |
99 |
(4,989) |
Directorate |
|
|
|
|
|
Biological Sciences |
7 |
76 |
17 |
100 |
(819) |
Computer and Information Science and Engineering |
62 |
32 |
5 |
99 |
(602) |
Engineering |
35 |
61 |
3 |
99 |
(646) |
Geosciences |
28 |
30 |
41 |
99 |
(803) |
Mathematical and Physical Sciences |
55 |
40 |
5 |
100 |
(1,290) |
Social, Behavioral, and Economic Sciences |
37 |
29 |
33 |
99 |
(683) |
Sciences (MPS) directorate were classified as theoretical research. In the Biological Sciences (BIO), Engineering (ENG), and Education and Human Resources (EHR) directorates, the majority of grants were classified as laboratory research; while in the Geosciences (GEO) and the Social, Behavioral, and Economic Sciences (SBE) directorates, the grants were closer to being equally divided among all three research categories. When asked about international research facilities—such as an accelerator, light source, ship, major telescope, or supercomputer—about one of five PIs require the use of such facilities.
Based on the NSF grantee database, the average NSF grant for the year FY 2001 was $336,000, and the average award duration was three years. A third of the PIs received total awards of $162,000 or less, another third received awards of more than $162,000 but less than $333,000, and the final 33 percent received more than $333,000 (Table II.2).
Overall, NSF did not request any changes of 5 percent or greater in the proposed award amount for nearly half (47%) of the funded proposals (Table II.2). However, 51 percent of the grants had a decrease of 5 percent or greater in the amount of funding awarded, compared to the amount of funding requested, while 2 percent had an increase of this amount. Chapter IV describes these funding changes in greater detail.
When awarding a grant, NSF is much less likely to change the requested duration of the proposed grant for research and education. Close to 9 of 10 PIs (88%) reported no change in the grant duration they requested for conducting their research and educational activities, compared to 10 percent who reported a decrease of one year or greater and 2 percent who reported an increase of one year or greater in the duration of their award (Table II.2).
In addition to
their FY 2001 NSF grant, most PIs report other funding for their research and
educational activities. For 19 percent
of the PIs, the FY 2001 NSF grant is the only source of current funding; 9
percent of the PIs have this grant and other NSF funding, 37 percent have
additional non-NSF funding, and 35 percent have other NSF and non-NSF funding
(Table II.2). Among the 44 percent of
the PIs who have other NSF grants, based on the resources they currently have
(which may not be the total amount and duration of these other grants), the
average number of other NSF grants is two, and the current amount is
$207,000. Among the 72 percent who now
have non-NSF grants, the current average number is two, and the average amount
is $199,000.
The Institutional Survey complements the Principal Investigator FY 2001 Grant Award Survey by providing information on how institutions manage the grant process. It should be noted that the process of applying for and administering NSF grants varies greatly from institution to institution. In our discussions with institutions while developing the questionnaire, it was clear that there is no “typical” administrative organization or grant process. For example, depending on the institution, there is variation in how many or what type of grant activities are centralized, versus their location in various different institutional units and departments. The responses to this survey are from the contact person nominated by that person’s institution as “the most knowledgeable about the overall grant process from the proposal phase to grant administration. And, who has final administrative responsibility for this process.”
We asked these institutional representatives about the way they are organized in general to handle the various aspects of grant management, and specifically about the use of resources for a “typical” FY 2001 NSF grant. The representatives provided information on the administrative offices and individuals who are assigned to grant proposals, grant proposal revisions, and the administration of grant awards. The number of administration offices involved in these three types of grant activities are as follows: grant proposal—an average of 2; grant proposal revisions—an average of 2; and grant administration—an average of 2 (Table II.4). The individuals assigned to each of these activities are: grant proposal an average of 6; grant proposal revisions—an average of 5; and grant administration—an average of 8.
As a context for the level of institutional management used for NSF grants, compared to the total number of grants administered in FY 2001, we asked institutions what percentage of the total grant awards and total award dollars are from NSF grants. For the total number of grants, 16 percent is the average. This estimate is similar for the total FY 2001 budget, with 18 percent the average percentage of the funding that comes from NSF. Another dimension of the level of effort expended is the institution’s number of NSF proposal awards and declines. Using the institutional information from the FY 2001 NSF grant data file, the average number of FY 2001 NSF grant awards is 12, compared to an average of 32 proposals that were declined. Chapter III goes into detail about the estimated number of hours institutions devoted specifically to FY 2001 NSF grants.
Forty-four percent are PH. D granting institutions that are not among the top 100 funded by NSF in FY 2001. They also include 5% who are the 20 top FY 2001 NSF-funded institutions, 3% rank in the top 21 to 50, and 9% are in the top 51 to 100 (Table II.4).
TABLE II.4
PROFILE OF INSTITUTION LEVEL OF GRANT EFFORT
N =
95
|
Mean* |
Number of FY 2001 NSF Grant Awards |
12 |
Number of FY 2001 NSF
Grant Declines |
32 |
Percent of FY 2001 Grants from NSF |
16 |
Percent of FY 2001 Total Grant Dollars
from NSF |
18 |
Individuals Assigned to: |
|
Grant
proposals Grant
proposal revisions Grant
administration |
6 5 8 |
Administrative Offices Assigned to: |
|
Grant
proposals Grant
proposal revisions Grant
administration |
2 2 2 |
................................................................................................................................ |
|
Type of Institution (Percentages) |
|
NSF funded top 20 |
5 |
NSF funded top 21 to 50 |
3 |
NSF funded top 51 to 100 |
9 |
Other Ph.D. |
44 |
Non-Ph.D. |
28 |
Non-Academic |
11 |
* Standard errors for the means are in the Appendix
The level of effort the PI expends to submit an application to NSF is an important factor to consider in assessing the funding and duration of NSF grants. While there may be some benefits to the PI in using the NSF proposal process as an opportunity to clarify research and educational goals and objectives, PIs are more likely to view the process as one in which resources are expended in preparing a proposal that may or may not be funded. Both the time and the money needed to prepare proposal submissions could be used to support research and educational activities. It should be noted that the questionnaire did not explicitly ask the PIs to evaluate the costs and benefits of the proposal process. However, among the PIs who volunteered verbatim comments about the proposal process, those who discussed the cost of time and resources expended were more numerous than those who mentioned the benefits. The study did, however, obtain information with which to determine the extent of the resources PIs and institutions use in submitting proposals. This information provides an estimate of the amount that could be saved if NSF grant awards were more efficient and effective. This chapter summarizes the level of effort PIs and institutions use to obtain grant funding.
One measure of proposal preparation effort is the total hours PIs estimate they needed to prepare their FY 2001 NSF grant submissions. This estimate includes the time spent by the PI and other people (such as graduate assistants, budget administrators, and secretaries) preparing the submission. The measure does not include time spent by institutional personnel. On average, PIs estimate 157 hours, or about 19.5 days (assuming an 8-hour day) for total proposal preparation. Assuming 235 working days in a full year, on average, about 8 percent of days in a working year is estimated as needed to prepare this one proposal. Overall, 33 percent of the PIs estimated hours for this proposal preparation in the low range of less than 80 hours; 29 percent are in the medium range of 80 to 150 hours; and 29 percent are in the high range of more than 150 hours used for proposal preparation (Table III.1). As summarized by one of the PIs, this level of effort can detract from other activities:
“…Raising funding is almost a full time job for me. If I didn’t have to do that, I could spend
more time mentoring women and minority students, and I could do my “dream
projects”—those that are either too high risk, or too expensive to do in the
current funding environment.”
Another
PI, who is on 100 percent “soft money” provides a description of the level of
proposal effort that is needed to get sufficient support:
“For a soft money
researcher needing to cover 12 months of his/her time and current budget
standards pushing for 3 months (or less) coverage for senior personnel per
proposal (seemingly, this is only communicated obliquely) and an average of
three year research duration this results in obtaining four funded proposals
per three years. This would be a
limited time sink if success was 100% (4 weeks/year of proposal writing at
three weeks/proposal). At 50% it is
eight weeks/year writing proposals. At
33% success it is 12 week/year and so on.
It seems that a better balance can be found. Consider that many programs have funding rates of 10-33
percent. (You tell me, I only hear
these numbers, perhaps they are a worst case).”
Contrary to what might be expected, PIs who submitted a revision of a previously declined NSF proposal (36%) are more likely than those with a first-time submission (26%) to be in the high range of proposal preparation hours (Table III.1). In addition, PIs who describe the research funded by this grant as laboratory (35%) and field (33%) are more likely to report hours for proposal preparation in the high range than those who describe their grant as theoretical (19%).
Overall, most grant preparation time is spent on the intellectual content of the proposal, compared to proposal mechanics. On average, PIs report that 69 percent of the time was devoted to intellectual content, compared to 31 percent on the mechanics of proposal preparation. The
TABLE III.1
PI LEVEL OF PROPOSAL EFFORT
(Percentages)
|
Low 80 or Less Hours |
Medium 81 to 150 Hours |
High More Than 150 Hours |
Don’t Know |
Total |
(N) |
Total |
33 |
29 |
29 |
8 |
99 |
(4,989) |
Grant Request |
|
|
|
|
|
|
First time submission |
36 |
28 |
26 |
9 |
99 |
(3,521) |
Revision of previously declined submission |
25 |
31 |
36 |
8 |
100 |
(1,449) |
Type of Research |
|
|
|
|
|
|
Theoretical |
43 |
27 |
19 |
10 |
99 |
(1,863) |
Laboratory |
25 |
32 |
35 |
8 |
100 |
(2,186) |
Field research |
31 |
28 |
33 |
7 |
99 |
(902) |
ranges for the percentage of time spent on preparation of intellectual content are: low, 65 percent or less (35%), medium, 66-80 percent (47%), and high, more than 80 percent (18%). In comparison, the range for mechanics is: low, 20 percent or less (37%), medium, 21-35 percent (30%), and high, more than 35 percent (32%). This suggests that a majority of the preparation time is expended by the PIs, since they are the main contributors to the intellectual content of the proposal.
Another level of proposal cost and effort is the PIs interaction with NSF to develop the grant proposal. Overall, about half of the PIs got advice from NSF about the amount of funding (57%), the substance of the grant (51%), and the grant duration (48%). Consultation with NSF resulted in 27 percent of the PIs decreasing the grant amount requested in their proposal, and 4 percent making an increase. PIs were less likely to change their duration request. Six percent decreased, and 3 percent increased, the duration of their grant request.
PIs who do not have sufficient funding for their research and educational activities need to prepare and submit multiple grants. For example, 38 percent of the PIs reported that they divide their ongoing body of research and educational activities into several proposals and submit them to NSF. More PIs conducting field research (44%) and laboratory research (40%) than PIs doing theoretical research (32%) reported dividing their work and submitting multiple proposals.
To summarize, just for the grant that the PIs submitted to NSF, proposal writing itself took about 20 days, and the key person in preparing the grant is the PI. As noted by some PIs, this time they might have used more effectively for research and educational activities.
For this discussion of the level of effort required for proposals, it is important to review the extent to which PIs prepared submissions other than for their FY 2001 NSF proposal that was described in Chapter II. In addition to submitting multiple grants to NSF, the PIs reported funding from other sources where proposals were submitted. To summarize, overall, about 8 of 10 PIs currently have funding, in addition to the FY 2001 NSF grant identified for this survey (Table III.2). It should be noted that depending on characteristics such as their type of research, some PIs spend more time on additional proposals than others.
With some PIs reporting that they have been awarded an average of four additional grants, an estimate can be made of the extent of PI time spent annually on proposals. Assuming the same number of hours used for each grant proposal that PIs estimated for the FY 2001 NSF grant, there could be an estimated 628 additional hours, or 79 days spent on preparing successful proposals. However, it should be noted that PIs may also be spending time preparing unsuccessful proposals, and this effort is not included in these estimates.
The level of effort devoted to proposal preparation can have an impact on award efficiency and effectiveness, as described by this PI:
“The trend of NSF to fund longer time periods is a great and
useful change. It reduces the time
spent preparing proposals, increases the time doing research and provides
stability for ongoing research programs.”
In addition to the hours spent by the PIs and their proposal assistants, there is proposal preparation time expended by the institutions where the PIs do their research and conduct their educational activities. From the institution’s perspective, this level of effort includes both proposals that are funded and those that are not. Among the institutions, there is variation in the
TABLE III.2
TOTAL FUNDING PROFILE
(Percentages)
|
Only FY
2001 NSF Grant |
FY 2001 NSF Grant and Other NSF Funding |
FY 2001 NSF Grant and Non-NSF Funding |
FY 2001 NSF Grants, Other NSF and Non-NSF Funding |
Total |
(N) |
Total |
19 |
9 |
37 |
35 |
100 |
(4,989) |
Type of Research |
|
|
|
|
|
|
Theoretical
research |
27 |
9 |
36 |
28 |
100 |
(1,863) |
Laboratory
research |
14 |
8 |
40 |
39 |
101 |
(2,186) |
Field
research |
13 |
12 |
34 |
41 |
100 |
(902) |
Additional Funds to
Accomplish Goals |
|
|
|
|
|
|
Low
($300,000 or less) |
27 |
11 |
38 |
23 |
99 |
(1,616) |
Medium
($300,000+ to $750,000) |
16 |
10 |
37 |
37 |
100 |
(1,377) |
High (more
than $750,000) |
6 |
7 |
37 |
50 |
100 |
(1,352) |
Do not know |
29 |
7 |
34 |
30 |
100 |
(644) |
Research Divided Into
Several Proposals |
|
|
|
|
|
|
Yes |
8 |
12 |
23 |
56 |
99 |
(1,895) |
No |
25 |
7 |
46 |
22 |
100 |
(3,069) |
Directorate |
|
|
|
|
100 |
|
Biological
Sciences |
19 |
8 |
46 |
27 |
100 |
(819) |
Computer and Information,
Science, and
Engineering |
16 |
12 |
31 |
41 |
100 |
(602) |
Engineering |
8 |
7 |
38 |
47 |
100 |
(646) |
Geosciences |
12 |
11 |
26 |
52 |
101 |
(803) |
Mathematical
and Physical
Sciences |
29 |
8 |
39 |
24 |
100 |
(1,290) |
Social,
Behavioral, and Economic
Sciences |
23 |
8 |
45 |
24 |
100 |
(683) |
level of institutional proposal involvement. For the purpose of describing the hours spent on proposal preparation, only those hours spent by the initial office are included. This may be an underestimate because as noted earlier, for some institutions, more than one office is involved in the proposal effort; but one office is typical. For a typical FY 2001 NSF proposal, the institutions estimate an average of 6 hours for preparation (Table III.3). Using the institutional information from the NSF data file, 44 is the average number of accepted (12), and declined (32) NSF proposals; the total average number of hours spent on proposal activity by the institutions can be estimated at 264 hours, or about 33 days (assuming an 8-hour day).
There is an additional level of effort when the institutions are included in the proposal revision process—that is, when NSF does not accept the original proposal, and the institution is involved in preparing the revision. Overall, from the institutions’ perspective, the average number of hours for each proposal revision is three.
Changes in award efficiency and effectiveness, which reduced the number of proposals needed for preparation, would have multiple benefits as described by one institution:
“This
could change the number of proposals submitted which would require less time
for the PI Department and proposal review and submission from the
Administrative Offices. This would help
the grant administration as the same amount of time per year would be required,
yet there would be additional funds.”
TABLE III.3
INSTITUTIONAL LEVEL OF EFFORT FOR TYPICAL FY 2001 NSF
GRANT ACTIVITIES
(Based
on Hours Per Grant)
|
Mean* |
Grant Proposal |
|
Number
of hours |
6 |
Grant Proposal Revision |
|
Number
of hours |
3 |
Grant Administration |
|
Number
of hours |
21 |
Grant Report Requirements |
|
Number
of hours |
6 |
*Standard errors for the means are in the Appendix.
A key objective of the PI survey is to find out: What are the award duration and funding amounts that are most efficient and effective in promoting NSF’s objectives? The answer is not straightforward. NSF has multiple objectives when it awards grants. As identified in the focus groups used to develop the survey, NSF grants contribute to the enhancement of the country’s scientific community by: (1) providing the only funding source for some scientific fields; (2) giving researchers the freedom to pursue unique ideas that reflect their own interests and expertise; (3) providing support for fundamental research; (4) playing a central role in establishing the careers of young researchers; (5) supporting the education and training of students; and (6) providing intellectual and scientific benefits through the review process. PIs who received the FY 2001 NSF grant awards are carrying out these diverse objectives. This diversity underscores what was said in the initial focus group discussions—that, at NSF, one size does not fit all. It also increases the challenge of establishing criteria to achieve award efficiency and effectiveness. This chapter summarizes the information that can be used to meet this challenge.
As described in the PI cognitive group discussion,
asking about the extent of additional funding and duration needed to support
ongoing research and educational activities is like “trying to measure a
dream.” Because the question of what is
the “appropriate” funding amount for award efficiency and effectiveness is
critical, the questionnaire used multiple approaches to get the PIs’
perspectives. In the following, we
describe each measure of award efficiency and effectiveness (Table IV.1). It
should be noted that, for the discussion of the amounts of funding
PROFILE OF AWARD EFFICIENCY AND EFFECTIVENESS ATTRIBUTES
(Means)
|
Award Request |
Duration Request |
Award Amount |
Award Duration |
Additional Funding for Next 5 Years |
Additional Duration for Next 5 Years |
Percent of Additional Funding From NSF |
Total |
$436,000 |
3 |
$336,000 |
3 |
$1,149,000 |
3 |
67 |
Type of Research |
|
|
|
|
|
|
|
Theoretical |
$373,000 |
3 |
$276,000 |
3 |
$740,000 |
2 |
71 |
Laboratory |
$507,000 |
3 |
$390,000 |
3 |
$1,190,000 |
3 |
63 |
Field |
$395,000 |
3 |
$331,000 |
3 |
$1,839,000 |
3 |
71 |
Directorate |
|
|
|
|
|
|
|
Biological Sciences |
$585,000 |
3 |
$436,000 |
3 |
$1,135,000 |
3 |
67 |
Computer and Information Science and Engineering |
$635,000 |
3 |
$447,000 |
3 |
$1,342,,000 |
2 |
66 |
Engineering |
$378,000 |
3 |
$315,000 |
3 |
$1,255,000 |
2 |
56 |
Geosciences |
$309,000 |
3 |
$270,000 |
3 |
$1,871,000 |
3 |
71 |
Mathematical and Physical Sciences |
$459,000 |
3 |
$350,000 |
3 |
$751,000 |
3 |
70 |
Social, Behavioral and Economic Sciences |
$229,000 |
2 |
$177,000 |
2 |
$715,000 |
3 |
71 |
and or duration needed for award efficiency and effectiveness, the
means include those for single variables and those for constructed variables as
described in Appendix A.
At the most basic level, award effectiveness and efficiency can be defined as the PIs’ actual experience of a reduction in funding and/or duration from what they requested to what NSF awarded. In FY 2001, 51 percent of the PIs had a 5 percent or greater decrease in funding, and 10 percent had a one year or greater decrease in duration. If award efficiency and effectiveness are defined as providing all PIs what they request, what would be the additional funding that is needed?
In FY 2001, these PIs requested an average amount of $436,000 for three years, and received $336,000 over three years. Defining award efficiency and effectiveness as providing PIs the funding they requested in their original proposal, a calculation for each PI that subtracts the actual award from the request and averages these differences finds $40,000 additional annual funding per grant would be required. The duration of the grant would remain the same at three years using the same type of calculation (Table IV.1 and Figure IV.1).
Using the same definition, there are variations in award efficiency and effectiveness among different types of PIs. For example, from the funding perspective, PIs conducting laboratory research and education would require $48,000 additional funding compared to those doing theoretical ($34,000) or field ($29,000) research and education (Figure IV.2).
There may be a caveat when these estimates are used to calculate award effectiveness and efficiency. The amounts the PIs request in their proposals may already have been adjusted to a lower amount because of advice the PIs receive when they submit or re-submit their budgets or because of perceptions of the amount of funding NSF is expected to support.
Another way to estimate award efficiency and effectiveness is to consider what percent of PIs’ ongoing body of research and educational activities is being funded by their FY 2001 NSF grant, and the amount needed to fund them 100 percent. Since PIs view their research and educational activities as being fluid, to provide standard criteria to answer this question, the total of what they would like to accomplish was established as 100 percent, and the time period for this accomplishment was given as the next five years.
Using these criteria, PIs estimated an average of 37 percent of the ongoing research and educational activities they would like to accomplish in the next five years will be achieved with the FY 2001 NSF grant (Table IV.2). Calculating the amount that would equal 100 per cent and subtracting the PIs current funding, there is an average difference of $904,000 additional funding. Over a five-year period, this equals $181,000 per year (Figure IV.1).
PIs conducting field research and education (34%) have a lower average estimate of what the current NSF grant can contribute in the next five years, compared to those doing theoretical (39%) and laboratory (37%) research and education. However, using this definition of award efficiency and effectiveness, the actual funding amounts required are greater for PIs conducting laboratory research and field research than for those doing theoretical research (Figure IV.2).
A third approach to estimate award efficiency and effectiveness is: What additional duration and funding do the PIs estimate is needed to accomplish their key goals? PIs were given the following guidelines to develop their estimates: (1) the additional duration should not include the years of funding they have in the FY 2001 NSF grant, and (2) the additional funding estimate is based on their needs for the next five years, and should not include the current funding they have from NSF or from any other sources. Using these criteria, PIs estimates that they need an
AWARD EFFICIENCY AND EFFECTIVENESS
FUNDING AMOUNT TO ACCOMPLISH A 100 PERCENT IN FIVE YEARS
(Means)
|
Award Dollar Amount |
Percent Achieved with FY 2001 Grant |
Dollar Amount for 100%* |
5-Year Additional Dollar Amount Needed for 100%* |
Total |
336,000 |
37 |
1,240,,000 |
904,000 |
Type of Research |
|
|
|
|
Theoretical Research |
276,000 |
39 |
946,000 |
671,000 |
Laboratory Research |
390,000 |
37 |
1,443,000 |
1,055,000 |
Field Research |
331,000 |
34 |
1,315,000 |
985,000 |
Directorate |
|
|
|
|
Biological Sciences |
436,000 |
39 |
1,561,000 |
1,122,000 |
Computer and Information Science and Engineering |
447,000 |
36 |
1,680,000 |
1,222,000 |
Engineering |
315,000 |
36 |
1,211,000 |
900,000 |
Geosciences |
270,000 |
30 |
1,076,000 |
812,000 |
Mathematical and Physical Sciences |
350,000 |
45 |
1,124,000 |
782,000 |
Social, Behavioral, and Economic Sciences |
177,000 |
32 |
790,000 |
608,000 |
*Constructed variables
created for each PI and then averaged.
See Appendix A for additional information on the calculation.
average of three additional award years. When the additional years the PIs suggest they would like are added to the years for each current award, five years is the average estimate to accomplish their key goals.
The questionnaire asked the PIs to focus on their needs for the next five years. The additional average funding of PIs’ project that they need in the next five years is $1,149,000 (Table IV.1) or an annual amount of $230,000 (Figure IV.1). It is important to remember that the $230,000 is not all that the PIs require; they would add this amount to their current funding from NSF and non-NSF sources to enable them to achieve their five-year goals.
The diversity of types of research funded by NSF is underscored by the need for additional duration and funding described by the PIs (Table IV.1). For example, using the means, PIs conducting laboratory and field research and education want three additional years, compared to those conducting theoretical research and education, who want an average of two years (Table IV.1). Over the next five years, PIs conducting theoretical research and education expect to need an average of about $148,000 per year, which is less than the $238,000 average for laboratory research and the $368,000 average for field research and education (Figure IV.2).
Another question related to NSF modifications in award efficiency and effectiveness is: Should NSF be responsible for funding all the PIs’ research and educational activities? As defined by the focus group participants, NSF grants are distinguished from others because they do not have to be “mission-directed.” Therefore, PIs may not expect NSF to fund all aspects of their research and educational agendas. PIs gave an estimate of the amount of additional funding they need in the next five years which they consider appropriate for NSF to fund.
The average percentage of funding that PIs suggest for NSF to fund is 67 percent. As described in Option 3 above, the annual amount of additional funding that PIs require is $230,000. The annual average amount each PI expects from NSF is $135,000 (Figure IV.1). This amount is calculated by multiplying the additional funding each PI would like over the next five years by the percentage that each PI expects NSF to fund and computing the mean.
Not all PIs agree on the percentage of funding appropriate for NSF. For example, PIs conducting theoretical (71%) and field (71%) research and education, expect NSF to pay a higher average percentage of the additional funds they require over the next five years, than for those doing laboratory (63%) research and education. These differences probably are based on PIs’ perceptions of NSF’s mission and of the typical type of research project NSF funds. The PIs conducting theoretical research and education projects ($94,000) expect the lowest amount of additional average annual funding from NSF, compared to laboratory ($140,000) and field ($199,000) research and education (Figure IV.2).
These four examples demonstrate the different choices to be made in determining the appropriate funding and duration for improving NSF award efficiency and effectiveness. Information from the PIs suggests that there is a greater consensus on award duration than on the amount. An average of five-year awards will, PIs feel, give them the most effective period of time for their research and educational activities. These awards would have such benefits as continuity of employment for students, particularly graduate student staff, opportunities to explore other areas of inquiry that may develop as they conduct their research, higher quality of research, and less time spent pursuing additional funding. However, NSF has a greater number of choices in deciding about additional dollars per year needed to fund grants that are efficient and effective. The previous examples of four possible ways to estimate award efficiency and effectiveness from the PIs’ perspective have averages that range from $40,000 to $230,000 per year (Figure IV.1). In addition, there is variation in these amounts, depending on particular characteristics of the PI, such as the type of research being conducted. NSF can use this information to identify which funding approach best meets the needs of the country’s scientific community. In Chapter V, we go into detail on the benefits of NSF improving award efficiency and effectiveness.
As described above, the requests for additional funding do not include their FY 2001 NSF grant or any other current funding. Figure IV.3 illustrates the projected total annual amounts when the calculations for the different options are added only to the average FY 2001 NSF grant amount. Overall, possible changes in annual funding amounts could range from $152,000 to $342,000 for an average NSF grant.
The four approaches to estimating award efficiency and effectiveness, focus on contributions expected from NSF, based on the FY 2001 grant identified for this survey. However, additional calculations could be made if NSF wanted to explore supplementing or replacing other PI funding sources. Obtaining the necessary funding from a single or few sources frees up PIs’ time, enabling them to conduct more research and to work more with students.
The survey included information on two additional current funding sources: (1) other NSF funding and (2) current non-NSF funding. As described in Chapter II, 8 of 10 PIs currently have other funding. Among those who have other NSF funding (44%), the current annual average amount is $207,000 (Figure IV.4). Currently 72 percent of the PIs have non-NSF funding that has an annual average of $199,000 (Figure IV.5). Depending on NSF’s decision on such criteria as the types of proposals they want to fund and what they want to achieve with award efficiency and effectiveness, these are additional estimates of funding that can be used to develop a budget for future awards.
The diversity of needs was reinforced when PIs were asked about their preferences for their own research. Thinking about what they specifically would like to accomplish, for about half of the PIs, more funding would have the greatest impact on their ongoing body of research and educational activities, while, for one-third, a longer duration would provide the more important resource. One of 10 PIs did not respond to this question because they did not make a request for a specific amount of additional funding or duration.
As NSF considers ways in which to improve award efficiency and effectiveness, resources may be used in three key ways, to: (1) increase the amount of funding, (2) increase the length of time per award, and (3) increase the total number of awards per year. For this question, the PIs were asked to change their focus and think about their general field of research, rather than their own specific grant. From the PIs’ perspective, there is no consensus as to which of these actions NSF should take if more funding is available. Among the three choices for their field of research, more PIs consider additional funding (40%) as their first choice, while an almost equal percentage prefer increasing the total number of awards (36%), and 24 percent would increase only the duration of the award (Table IV.3). The variation in PIs’ selection of choices underscores the multiple research and educational needs that NSF grants meet, which may require different types of resources. For example, PIs who describe their area of research and education as theoretical (35%) are less likely than those who do field work (39%) or laboratory research (43%) to select an increase in the award amount as their first priority for their field.
TABLE
IV.3
MOST IMPORTANT NSF AWARD REFORM FOR PI AREA OF RESEARCH
(Percentages)
|
Increase Only the Amount of Funding Per Award |
Increase Only the Length of Time Per Award |
Increase Only the Total Number of Awards Per Year |
Total |
(N) |
Total |
40 |
24 |
36 |
100 |
(4,989) |
Change in 2001 NSF Grant Funding |
|
|
|
|
|
5% or greater increase |
32 |
20 |
47 |
99 |
(123) |
5% or greater decrease |
44 |
24 |
31 |
99 |
(2,533) |
All others |
35 |
24 |
40 |
99 |
(2,333) |
Change in 2001 NSF Grant Duration |
|
|
|
|
|
1 year or greater increase |
43 |
22 |
34 |
99 |
(92) |
1 year or greater decrease |
40 |
29 |
30 |
99 |
(485) |
All others |
39 |
23 |
36 |
98 |
(4,412) |
Type of Research |
|
|
|
|
|
Theoretical |
35 |
26 |
38 |
99 |
(1,863) |
Laboratory |
43 |
25 |
31 |
99 |
(2,186) |
Field research |
39 |
17 |
42 |
98 |
(902) |
Additional Funds to Accomplish Goals |
|
|
|
|
|
Low (less than $300,000) |
32 |
24 |
43 |
99 |
(1,616) |
Medium ($300,000 + to $750,000) |
40 |
26 |
33 |
99 |
(1,377) |
High (greater than $750,000) |
52 |
22 |
26 |
100 |
(1,352) |
Do not know what amount is needed |
31 |
23 |
45 |
99 |
(644) |
How might NSF reforms in its grant funding levels and/or duration influence ongoing research and educational activities in the United States? We looked at this question from two perspectives. First, we looked at the impacts reported by PIs when the FY2001 NSF grant they had requested was awarded but the funding level or duration had been altered. Second, we asked the PIs to speculate on how their own research might be affected if NSF increased the funding and/or duration of their grant awards. The results of these questions are described below.
As described earlier in this report, among the FY2001 NSF grantees, almost half (47%) were funded within 5 percent of the amount requested. Of the remaining 53 percent, 51 percent of the grantees had their funding cut by 5 percent or more, while 2 percent had their funding increased by more than 5 percent or more. Changes to the grant duration were even less frequent, with 88 percent remaining unchanged. Of the 12 percent with changes, 10 percent had their duration decreased by more than one year, and 2 percent had it increased by more than one year.
The questionnaire listed 18 areas where changes in funding or duration could potentially affect a research project. PIs whose NSF grants for FY2001 included changes in funding or duration were asked whether, for each area, the change had proven to be positive, of no consequence, or negative. Not surprisingly, most PIs reported some type of negative impact. As shown in Table V.1, approximately half (55%) said the change had affected their ability to obtain quality personnel, and another 50 percent said it had limited their ability to collaborate with researchers in their area of research. Close to half (51%) felt that the change had restricted their ability to pursue innovative or “high-risk” ideas. Again, not surprisingly, about two-thirds (67%)
NEGATIVE IMPACT OF FUNDING AND/OR DURATION
AWARD CHANGES
(Percentages)
|
|
Negative Impact |
|
|
|
Total (N=4,989) |
5% or Greater Decrease in Funding (N = 2,533) |
1. |
Achieve the research objectives within the
specified time |
34 |
67 |
2. |
Obtain quality personnel |
28 |
55 |
3. |
Collaborate with researchers in your area of research |
26 |
50 |
4. |
Pursue high-risk ideas |
26 |
51 |
5. |
Pursue innovative ideas |
23 |
45 |
6. |
Collaborate with researchers in different areas of research |
21 |
41 |
7. |
Establish mentoring or other research-based education activities |
21 |
41 |
8. |
Develop instrumentation or other enhancements for the research and
education infrastructure |
20 |
40 |
9. |
Integrate research activity into your teaching and training |
18 |
36 |
10. |
Broaden participation of under-represented groups in the research
activity |
18 |
34 |
11. |
Access state-of-the-art equipment |
17 |
33 |
12. |
Disseminate research findings |
17 |
33 |
13. |
Develop partnerships with industry, other educational
institutions, or national laboratories |
13 |
25 |
14. |
Improve public understanding of the project |
11 |
20 |
15. |
Access facilities |
10 |
20 |
16. |
Nurture connections between research activity and its potential
for: health benefits, economic
benefits, and national security benefits |
10 |
20 |
17. |
Develop programs with K‑12 teachers and/or students |
7 |
13 |
18. |
Obtain other funding |
6 |
12 |
indicated that the change had negatively affected their ability to “achieve their research objectives within the specified time.”
PIs were given the opportunity to describe any other impacts they had experienced due to changes in their award. Overall, 48 percent of those who had a grant change provided a verbatim response. Table V.2 contains a summary of these comments. It should be noted that these percentages are based on the number of responses given, not on the individual PIs. The key impact described in 35 percent of the responses was the negative impact on building a project team. Among the responses on this topic, a main concern was the ability to recruit and retain staff (22%). The second most mentioned consequence of award changes was the impact on meeting the goals and objectives of what had been proposed. Nine percent of the comments about goals and objectives were focused on a reduction in the scope of the project. Thirteen percent of the responses mentioned an impact on aspects of the project process, with 8 percent focused specifically on the increased time spent seeking funding, rather than working on their research and educational activities.
Several PIs summarized their experience resulting from the decreases in their FY2001 awards:
“A major impact of the
shortened funded period was the lack of continuity. It had a dramatic effect on
training students and attracting new students or postdocs. One cannot plan or
attract good students and postdocs if you only have 1.5 yr of funding. The best
training and research is accomplished when you have a dynamic group of people
interacting. The synergisms is incredible. To do this one needs to be able to
constantly recruit new people and this cannot be done without some guarantee of
support.”
“The question before addressed well the
negative impact that the decrease in funding has on my research program. Having
funding for 5 years instead of three would have allowed me to have a broader
focus and to be somewhat more ambitious in my long term research goals. Also, a
considerable amount of effort goes into the preparation of a grant proposal,
and so longer periods between submissions could benefit the research output. It
is clear that there is a positive side to the writing of grant proposals, the
need
TABLE
V.2
IMPACT OF CHANGE IN AWARD FUNDING AND/OR DURATION
(Percentages)
Response Categories |
Total Responses (1,835) |
Negative impact on team building – staff, student, collaborators |
35 |
Reduction in meeting goals and objectives |
27 |
Negative impact on project process |
13 |
No impact or impact not known |
8 |
Limited research tools |
7 |
Comments on positive impact |
6 |
Other comments |
4 |
TOTAL |
100 |
to
clarify one's ideas and to plan ahead with scientific vision, but this is also
well achieved through the annual progress reports.”
It is useful to contrast the experiences of PIs who had a decrease with those of the small number of PIs who received increases in funding and/or duration.
“Increased funding level has allowed increased flexibility in how we approach our research, which should have a positive impact on our productivity and which is having a very positive effect on graduate student training.”
Switching from the limitations that a reduction in funding or duration might bring, the PIs were asked to describe the impact of receiving additional funding and/or time on their ongoing body of research and educational activities. This leads to a key question: If funding and duration increases are implemented, what are the potential benefits? When PIs who specified a specific additional amount of funding and/or duration were asked this question with respect to 18 areas related to research goals, outcomes, processes or team-building, about half of them indicated a positive impact for all but 1 of the 18 areas (Table V.3). The most broadly perceived benefit, indicated by 96 percent, was the freedom to “pursue more innovative ideas.” Moreover, 92 percent also felt that additional support would facilitate greater collaboration among researchers, as well as help them achieve their research objectives within a specified time frame.
About half (47%) of the PIs took the opportunity to describe in their own words the impact of receiving an award that would provide them with what they needed to accomplish their research and educational goals (Table V.4). Thirty-five percent of the responses described improvements and increases in the applications and outcomes of their research and educational activities; among these 9 percent were a description of a specific scientific advance that would
POSITIVE IMPACT OF ADDITIONAL AWARD FUNDING
AND/OR DURATION
(Percentages)
|
|
Positive Impact |
|
|
|
Total (N = 4,989) |
Gave Funding/ Duration Amount (N = 4,489) |
1. |
Pursue innovative ideas |
87 |
96 |
2. |
Collaborate with researchers in your area of research |
83 |
92 |
3. |
Achieve the research objectives within the specified time |
83 |
92 |
4. |
Collaborate with researchers in different areas of research |
76 |
84 |
5. |
Obtain quality personnel |
76 |
85 |
6. |
Pursue high-risk ideas |
76 |
85 |
7. |
Disseminate research findings |
74 |
82 |
8. |
Integrate research activity into your teaching and training |
73 |
81 |
9. |
Establish mentoring or other research-based education activities |
71 |
79 |
10. |
Broaden participation of under-represented groups in the research
activity |
62 |
69 |
11. |
Develop partnerships with industry, other educational
institutions, or national laboratories |
62 |
69 |
12. |
Develop instrumentation or other enhancements for the research and
education infrastructure |
61 |
67 |
13. |
Access state-of-the-art equipment |
60 |
67 |
14. |
Improve public understanding of the project |
58 |
64 |
15. |
Obtain other funding |
54 |
60 |
16. |
Access facilities |
49 |
54 |
17. |
Nurture connections between research activity and its potential
for: health benefits, economic
benefits, and national security benefits |
48 |
54 |
18. |
Develop programs with K‑12 teachers and/or students |
32 |
35 |
TABLE
V.4
IMPACT OF ADDITIONAL FUNDING AND/OR DURATION
(Percentages)
Response Categories |
Total Responses (3,857) |
Improved and increased applications and outcomes |
35 |
Enhanced goals and objectives |
23 |
Improved team building with staff and students |
21 |
Process for funding improved |
11 |
Research tools enhanced |
6 |
No impact |
3 |
Other comments |
2 |
result from having additional resources. Twenty-three percent of all responses were related to enhanced goals and objectives; these included 13 percent who described the ability to explore new, high-risk ideas and the chance to be more innovative. Comments on improved team-building (21%) included 13 percent who specifically mentioned enhanced opportunities for students.
When asked how they would spend additional funding for securing the resources needed to conduct high-quality research—personnel, equipment, travel, the size and quality of experiments or tests—the PIs indicated that they would be most likely to spend additional resources on hiring students. Among the 16 resources listed in Table V.5, three of the top five choices indicated that PIs would spend additional funding on increasing the number and/or time of graduate students, undergraduates, or postdoctoral associates—in that order of preference. More specifically, 78 percent said they would “very likely” increase the number of graduate students hired or the duration of their students’ work period. Increasing the quality and number of experiments or tests rounds out the other two top choices.
PIs who indicated that additional funding would benefit their work were asked the extent to which they felt this additional funding would affect the quality, duration, or number of experiments they performed or their ability to recruit the highly qualified labor needed to conduct high-quality research. As shown in Table V.6, more than half said that additional funding would help them in their efforts to recruit postdoctoral and graduate students “a great deal.” Close behind, nearly half thought the additional support would improve their research by allowing for a substantial increase in the number of experiments, tests, or subjects.
ADDITIONAL NSF AWARD FUNDING AND/OR
DURATION:
POTENTIAL INCREASES IN RESEARCH AND
EDUCATIONAL ACTIVITIES
(Percentages)
|
|
Very Likely |
|
|
|
Total (N = 4,989) |
Gave Funding/ Duration Amount (N = 4,489) |
1. |
The number and/or months of graduate
students |
70 |
78 |
2. |
The number of experiments, tests, subjects |
49 |
54 |
3. |
The number and/or months of undergraduate students |
45 |
50 |
4. |
The number and/or months of post doctoral associates |
43 |
48 |
5. |
The quality of the experiments or tests |
36 |
40 |
6. |
The number of equipment purchases |
33 |
36 |
7. |
The size of the experiments or tests |
30 |
33 |
8. |
The number and/or months of senior personnel |
29 |
32 |
9. |
The number of trips |
29 |
32 |
10. |
The quality of equipment purchases |
28 |
31 |
11. |
Participant support |
21 |
23 |
12. |
Computer/publication costs |
16 |
18 |
13. |
The number and/or months of technicians |
15 |
17 |
14. |
The number and/or months of programmers |
7 |
8 |
15. |
Consultant services |
6 |
6 |
16. |
The cost per trip |
3 |
4 |
ADDITIONAL NSF AWARD FUNDING AND/OR
DURATION:
INCREASED ABILITY FOR RESEARCH AND
EDUCATIONAL ACTIVITIES
(Percentages)
|
|
Increased a Great Deal |
|
|
|
Total (N = 4,989) |
Gave Funding/ Duration Amount (N = 4,489) |
1. |
Recruit graduate students |
56 |
62 |
2. |
Recruit post-doctoral associates |
47 |
52 |
3. |
Conduct more experiments, tests or subjects |
42 |
47 |
4. |
Have higher-quality experiments or tests |
31 |
34 |
5. |
Provide adequate support for a graduate student to shorten time to
degree |
29 |
32 |
6. |
Recruit undergraduate students |
27 |
30 |
7. |
Provide stability for technicians |
17 |
19 |
8. |
Duration of experiments |
17 |
19 |
9. |
Provide stability for programmers |
8 |
8 |
As Table V.7 illustrates, these PIs, in addition to the individual benefits they felt their own research would gain, felt that increased NSF funding or duration had broader implications for scientific research. Overall, more than 60 percent said increased NSF funding and duration would “very likely” “widen the focus” of research in their field, while also attracting “more” and “better graduate students.” Seventy percent agreed that additional support would “very likely” “decrease interruptions in funding,” and almost half (46 percent) thought that such support would very likely increase the number of proposals containing innovative ideas.
As described in Chapter II, for their current research projects, PIs support an average of two undergraduates, four graduate students, and one postdoctoral fellow (Figure II.1). As noted, PIs reported being “very likely” to increase the number or duration of these students and postdoctoral fellows if NSF provided additional grant support. Moreover, as stated earlier, among those who were awarded an NSF grant in FY2001, the average difference between the level of funding requested and the level awarded was $33,000 a year. At many schools, being able to provide this difference could itself mean supporting an additional graduate student. In addition to improving PI productivity, hiring additional students has the added benefit of supporting highly qualified students whose talents are lost when they must leave the sciences to support themselves in alternative careers.
From the PIs’ perspective, it
is not only the additional funding and duration that is attractive, but also NSF’s unique contribution to the
development of students:
ADDITIONAL NSF AWARD FUNDING AND/OR
DURATION:
POTENTIAL FOR CHANGES IN FIELD OF RESEARCH
(Percentages)
N = 4,989
|
|
Very Likely |
1. |
Decrease interruptions in funding |
70 |
2. |
Attract more graduate students |
65 |
3. |
Widen the focus of the research in your field |
63 |
4. |
Attract better graduate students |
62 |
5. |
Increase the number of proposals to NSF with innovative ideas |
46 |
6. |
Increase the number of proposals to NSF with high-risk ideas |
37 |
7. |
Attract more established researchers to apply for NSF funding |
37 |
8. |
Improve access to facilities and databases |
36 |
9. |
Decrease the amount of time to answer research questions |
31 |
“Clearly more money helps, but it is the
TYPE of funding that NSF provides that is unique. So many other sources (including federal agencies) have a
short-term focus on technological or economic deliverables. More NSF funds opens up a spectrum of
possibilities: working on new ideas, allowing students to drive some of the
research, building infrastructure and a base from which more funds can be
obtained from other sources, more freedom to focus of graduate and
undergraduate educational goals rather than "research" goals. The NSF is different than industry, and
needs to stay that way; and it needs to be a bigger part of academic funding.”
“The NSF funding provides the "next
generation" of technologies that are needed to take this project to its
optimal conclusion. It is amazingly
difficult to get funding for this basic research any other way. Also, the continuity of NSF multi-year
funding is critical to the continuity of my students. Without it, promising funding for a PhD student can be risky, as
many of the other grants can have "lag time" between them.”
“My main concern with NSF is the 3 year cap on funding for grants as I mentioned before. This 3 year cap is not enough to have a student finish a project. Even if the project is renewed, then the renewal often does not come in time to support a graduate student’s stipend working on the original project. As a result, the student has to TA or find a scholarship or just borrow from other grants for obtaining money to finish and graduate. Consequently, I would be very much in favor of extending grants to 4 years rather than 3. If this reduces the award amount slightly, so be it. But I do not think that has to be true since people would just get fewer grants but for longer periods of time. Since NSF is in the business of education scientists and engineers, I would think that NSF would be most interested in obtaining continuous funding for our hard-working graduate students. These students often sacrifice personal rewards for the opportunity to do research and we shouldn’t punish them by making them go get more money to finish out their thesis since the NSF grant our out after 3 years.”
The institutional representatives were asked several questions that gave them an opportunity to describe in their own words what they thought would be the most significant change for their institution if NSF increased either the average duration of the awards or the average award amount. The greatest percentage of comments on possible changes that would occur if NSF increased the average award duration were about how it would impact the institution’s grant process (35%). Thirteen percent who noted this change said there would be a decrease in time and effort (Table V.8). These comments also noted that there would be an improvement in the quality and efficiency of the research (18%), more stable funding (15%), and there would be staffing changes (15%) which included comments on the positive impact on PIs (6%) and more student involvement (4%).
The institutional representatives had somewhat similar comments about the significant changes that would occur if NSF increased the award amount. More than half of the responses describing the impact of increasing the average dollar amount described improvements in the quality and quantity of the research (27%) and possible staffing changes (24%) such as increased student involvement (16%) (Table V.9). The impact on the institution’s grant process (28%) was also noted with 9 percent describing a decrease in the institution’s time and effort and 5 percent speculating that there would be an increase in time and effort.
There were also suggestions on how NSF could reduce the amount of time and resources used by the institution to manage NSF grants. More than half (55%) of these comments described possible changes in the grant process. Specifically, 16 percent described positive experiences with FastLane and 14 percent offered suggestions for possible FastLane improvements.
F. ADDITIONAL COMMENTS ABOUT THE NSF GRANT
PROCESS
At the end of the questionnaire, the PIs were given an opportunity to write in any additional comments they had about the NSF grant process. Half of them provided additional information (Table V.11). Since the question was very broad, the PIs commented on a variety of topics. Responses related to the review process made up 23 percent of these comments. Two main themes in this category were the observation that the peer review process is satisfactory and
INSTITUTION CHANGES IF NSF INCREASED THE
AVERAGE DURATION PER GRANT
|
Total
Responses* |
|
Grant Process General
comments (18) Increase
time and effort (4) Decrease
time and effort (13) |
35 |
|
Research Changes Improved
quality/efficiency |
18 |
|
Award Duration Improvements More
stable funding; fewer no-cost extensions |
15 |
|
Staffing Changes General
comments (5) More
student involvement (4) Positive
PI impact (6) |
15 |
|
No Changes |
6 |
|
No Comment/No Response |
12 |
|
TOTAL |
101 |
*Weighted number of responses is 839.
INSTITUTION CHANGES IF NSF INCREASED THE
AVERAGE DOLLAR AWARD PER GRANT
|
Total Responses* |
Grant Process General
comments (7) Increase
time and effort (5) Decrease
time and effort (9) Increase
number of applications (7) |
28 |
Research Changes More
conducted, improved quality |
27 |
Staffing Changes General
comments (4) More
student involvement (16) More
faculty involvement (4) |
24 |
Award Amount More
stable funding; more budget flexibility |
11 |
No Changes |
7 |
No Comment/No Response |
4 |
TOTAL |
101 |
*Weighted number of responses is 967.
SUGGESTIONS FOR NSF CHANGES TO REDUCE
INSTITUTION TIME AND RESOURCES
|
Total Responses* |
Grant Process |
55 |
General
comments (21) Reduce
budget revisions, requests (4) Comments
on FastLane improvements (14) Positive
experience with FastLane (16) |
|
General Comments on Award Amount and Duration |
5 |
No Suggestions |
8 |
Experience with NSF Staff No Comments |
4 27 |
TOTAL |
99 |
*Weighted number of responses is 681.
TABLE
V.11
PRINCIPAL INVESTIGATORS: OTHER COMMENTS
ON THE NSF GRANT PROCESS
(Percentages)
Response Categories |
Total Responses (5,056) |
Review process |
23 |
General award comments |
17 |
Proposal process |
12 |
Award size |
11 |
Award administration |
10 |
Award duration |
8 |
Overall satisfaction with NSF grant |
8 |
Comments on the questionnaire |
3 |
Communication between NSF and principal investigators |
2 |
Other comments |
5 |
TOTAL |
99 |
award decisions were risk adverse. Seventeen percent of the comments were general remarks about the grant awards, with many PIs singling out certain aspects of NSF grants—for example, not being able to do the research without NSF, and NSF grants provide more flexibility than other funding organizations. These general comments also covered the proposal process (12%), the award size (11%), the administration of the award (10%), and the award duration (8%).
The institutional representatives were given the opportunity to write in “any other comments they had about their institution’s experiences with the NSF grant process.” Twenty-three percent of the comments were about FastLane and the use of technology in the grant process (Table V.12); among these comments, 15 percent described positive experiences with FastLane. There were also comments about working with NSF staff (11%) and the level of effort involved in the grant process (16%).
INSTITUTIONS: OTHER COMMENTS ON THE NSF GRANT PROCESS
|
Total Responses* |
NSF Staff |
11 |
-
Positive experiences (9) -
Other comments (2) |
|
Technology/Fast Lane |
23 |
Level of Effort for Grant Process |
16 |
Award Duration and Amount |
6 |
Other Comments |
3 |
No Comments |
42 |
TOTAL |
101 |
*Weighted number of responses is 666.
[1]“Document the efficiency of the research process. With the assistance of U.S. academic research institutions, NSF should develop efficiency measures of the research process and determine what is the right grant size for the myriad types of research the agency funds. These metrics and grant size determinations should be developed for consideration in the FY 2003 Budget”. June 7, 2001, letter from OMB to Dr. Rita Colwell.
[2] The use of means to describe values was decided after careful consideration. As a general comparison, the mean uses more information than the median since the exact scores are used in the computation, while the median uses the relative position of the scores. However, the mean is effected by extreme values while the median is generally unaffected by extremes. For some items there are calculated means which are described in detail in Appendix A.