Appendix | Methodology

Statistical tests can be used to determine whether differences observed in sample survey data are “real” differences in the population. Differences that are termed statistically significant are likely to occur in the target population. When Indicators reports statements about differences on the basis of sample surveys, the differences are statistically significant at least at the 10% level. This means that, if there were no true difference in the population, the chance of drawing a sample with the observed or greater difference would be no more than 10%.

A statistically significant difference is not necessarily large, important, or significant in the usual sense of the word. It is simply a difference that is unlikely to be caused by chance variation in sampling. With the large samples common in Indicators data, extremely small differences can be found to be statistically significant. Conversely, quite large differences may not be statistically significant if the sample or population sizes of the groups being compared are small. Occasionally, apparently large differences are noted in the text as not being statistically significant to alert the reader that these differences may have occurred by chance.

Numerous differences are apparent in every table in Indicators that reports sample data. The tables permit comparisons between different groups in the survey population and in the same population in different years. It would be impractical to test and indicate the statistical significance of all possible comparisons in tables involving sample data.

As explained in the section About Science and Engineering Indicators, Indicators presents indicators. It does not model the dynamics of the S&E enterprise, although analysts could construct models using the data in Indicators. Accordingly, Indicators does not make use of statistical procedures suitable for causal modeling and does not compute effect sizes for models that might be constructed using these data.