Chapter 7.
Reporting the Results of Mixed Method Evaluations

The final task the evaluator is required to perform is to summarize what the team has done, what has been learned, and how others might benefit from this project’s experience. As a rule, NSF grantees are expected to submit a final report when the evaluation has been completed. For the evaluator, this is seen as the primary reporting task, which provides the opportunity to depict in detail the rich qualitative and quantitative information obtained from the various study activities.

In addition to the contracting agency, most evaluations have other audiences as well, such as previously identified stakeholders, other policymakers, and researchers. For these audiences, whose interest may be limited to a few of the topics covered in the full report, shorter summaries, oral briefings, conference presentations, or workshops may be more appropriate. Oral briefings allow the sharing of key findings and recommendations with those decisionmakers who lack the time to carefully review a voluminous report. In addition, conference presentations and workshops can be used to focus on special themes or to tailor messages to the interests and background of a specific audience.

In preparing the final report and other products that communicate the results of the evaluation, the evaluator must consider the following questions:

• How should the communication be best tailored to meet the needs and interests of a given audience?
• How should the comprehensive final report be organized? How should the findings based on qualitative and quantitative methods be integrated?
• Does the report distinguish between conclusions based on robust data and those that are more speculative?
• Where findings are reported, especially those likely to be considered sensitive, have appropriate steps been taken to make sure that promises of confidentiality are met?

This chapter deals primarily with these questions. More extensive coverage of the general topic of reporting and communicating evaluation results can be found in the earlier User-Friendly Handbook for Project Evaluation (NSF, 1993).

Ascertaining the Interests and Needs of the Audience

The diversity of audiences for which the findings are likely to be of interest is illustrated for the hypothetical project in Exhibit 19. As shown, in addition to NSF, the immediate audience for the evaluation might include top-level administrators at the major state university, staff at the Center for Educational Innovation, and the undergraduate faculty who are targeted to participate in these or similar workshops. Two other indirect audiences might be policymakers at other 4-year institutions interested in developing similar preservice programs and other researchers. Each of these potential audiences might be interested in different aspects of the evaluation's findings. Not all data collected in a mixed method evaluation will be relevant to their interests. For example:

• The National Science Foundation staff interested in replication might want rich narrative detail in order to help other universities implement similar preservice programs. For this audience, the model would be a descriptive report that traces the flow of events over time, recounts how the preservice program was planned and implemented, identifies factors that facilitated or impeded the project’s overall success, and recommends possible modifications.
• Top-level administrators at the university might be most interested in knowing whether the preservice program had its intended effect, i.e., to inform future decisions about funding levels and to optimize the allocation of scarce educational resources. For this audience, data from the summative evaluation are most pertinent.
• Staff at the Center for Educational Innovation might be interested in knowing which activities were most successful in improving the overall quality of their projects. In addition, the Center would likely want to use any positive findings to generate ongoing support for their program.

Exhibit 19. Matrix of stakeholders
		Intended impacts of the study
Potential audience for the study findings	Levels of audience involvement with the program	Assess success of the program	Facilitate decision-making	Generate support for the program	Revise current theories about preservice education	Inform best practices for preservice education programs
National Science Foundation	Direct	X	X		X	X
Top-level administrators at the major state university	Direct	X	X		X
Staff at the Center for Educational Innovation	Direct	X	X	X	X	X
Undergraduate faculty targeted to participate in the workshops	Direct	X			X
Policymakers at other 4-year institutions interested in developing similar preservice programs	Indirect	X	X			X
Other researchers	Indirect	X			X	X

In this example, the evaluator would risk having the results ignored by some stakeholders and underutilized by others if only a single dissemination strategy was used. Even if a single report is developed for all stakeholders (which is usually the case), it is advisable to develop a dissemination strategy that recognizes the diverse informational needs of the audience and the limited time some readers might realistically be able to devote to digesting the results of the study. Such a strategy might include (1) preparing a concise executive summary of the evaluation’s key findings (for the university's top-level administrators); (2) preparing a detailed report (for the Center for Educational Innovation and the National Science Foundation) that describes the history of the program, the range of activities offered to undergraduate faculty, and the impact of these activities on program participants and their students; and (3) conducting a series of briefings that are tailored to the interests of specific stakeholders (e.g., university administrators might be briefed on the program's tangible benefits and costs). By referring back to the worksheets that were developed in planning the evaluation (see Chapter 5), the interests of specific stakeholders can be ascertained. However, rigid adherence to the original interests expressed by stakeholders is not always the best approach. This strategy may shortchange the audience if the evaluation - as is often the case - pointed to unanticipated developments. It should also be pointed out that while the final report usually is based largely on answers to summative evaluation questions, it is useful to summarize salient results of the formative evaluation as well, where these results provide important information for project replication.

Organizing and Consolidating the Final Report

Usually, the organization of mixed method reports follows the standard format, described in detail in the earlier NSF user-friendly handbook, that consists of five major sections:

• Background (the project’s objectives and activities);
• Evaluation questions (meeting stakeholders’ information needs);
• Methodology (data collection and analysis);
• Findings; and
• Conclusions (and recommendations).

In addition to the main body of the report, a short abstract and a one-to four-page executive summary should be prepared. The latter is especially important because many people are more likely to read the executive summary than the full document. The executive summary can help focus readers on the most significant aspects of the evaluation. It is desirable to keep the methodology section short and to include a technical appendix containing detailed information about the data collection and other methodological issues. All evaluation instruments and procedures should be contained in the appendix, where they are accessible to interested readers.

Regardless of the audience for which it is written, the final report must engage the reader and stimulate attention and interest. Descriptive narrative, anecdotes, personalized observations, and vignettes make for livelier reading than a long recitation of statistical measures and indicators. One of the major virtues of the mixed method approach is the evaluator’s ability to balance narrative and numerical reporting. This can be done in many ways: for example, by alternating descriptive material (obtained through qualitative techniques) and numerical material (obtained through quantitative techniques) when describing project activities, or by using qualitative information to illustrate, personalize, or explicate a statistical finding.

But - as discussed in the earlier chapters - the main virtue of using a mixed method approach is that it enlarges the scope of the analysis. And it is important to remember that the purpose of the final report is not only to tell the story of the project, its participants, and its activities, but also to assess in what ways it succeeded or failed in achieving its goals.

In preparing the findings section, which constitutes the heart of the report, it is important to balance and integrate the descriptive and evaluative reporting section. A well-written report should provide a concise context for understanding the conditions in which results were obtained and identifying specific factors (e.g., implementation strategies) that affected the results. According to Patton (1990),

For the hypothetical project, most questions identified for the summative evaluation in Exhibit 16 can be explored through the joint use of qualitative and quantitative data, as shown in Exhibit 20. For example, to answer some of the questions pertaining to the impact of faculty training on their students’ attitudes and behaviors, quantitative data (obtained from a student survey) are being used, together with qualitative information obtained through several techniques (classroom observations, faculty focus groups, interviews with knowledgeable informants.)

Exhibit 20. Example of an evaluation/methodology/matrix
Project goals	Summative evaluation study questions	Data collection techniques (see codes below)
		a	b	c	d	e	f
Changes in instructional practices by participating faculty members	Did the faculty who experienced the workshop training change their instructional practice?	X	X		X	X
	Did the faculty who experienced the workshop training use the information regarding new standards, materials, and practices?	X	X		X	X
	What practices prevented the faculty who experienced the workshop training from implementing the changes?	X	X
Acquisition of knowledge and changes in instructional practices by other (nonparticipating) faculty members	Did participants share the knowledge acquired through the workshops with other faculty?	X	X			X
	What methods did participants use to share the knowledge acquired through the workshops?	X
Institution-wide changes in curriculum and administrative practices	Were changes made in curriculum?	X					X
	Were changes made in examinations and other requirements?	X					X
	Were changes made in expenditures for libraries and other resource materials?	X					X
Positive effects on career plans of students taught by participating teachers	Did students become more interested in their classwork?	X	X			X
	Did students become more active participants?	X	X		X	X
	Did students express interest in teaching math after graduation?					X
	Did students plan to use new concepts and techniques?					X
a = indepth interviews with knowledgeable informants b = focus groups c = observation of workshops		d = classroom observations e = surveys of students f = documents

Formulating Sound Conclusions and Recommendations

In the great majority of reporting activities, the evaluator will seek to include a conclusion and recommendations section in which the findings are summarized, broader judgments are made about the strengths and weaknesses of the project and its various features, and recommendations for future, perhaps improved, replications are presented. Like the executive summary, this section is widely read and may affect policymakers' and administrators' decisions with respect to future project support.

The report writer can include in this section only a limited amount of material, and should therefore select the most salient findings. But how should saliency be defined? Should the "strongest" findings be emphasized, i.e., those satisfying accepted criteria for soundness in the quantitative and qualitative traditions? Or should the writer present more sweeping conclusions, ones which may be based in part on impressionistic and anecdotal material?

It can be seen that the evaluator often faces a dilemma. On the one hand, it is extremely important that the bulk of evaluative statements made in this section can be supported by accurate and robust data and systematic analysis. As discussed in Chapter 1, some stakeholders may seek to discredit evaluations if they are not in accord with their expectations or preferences; such critics may question the conclusions and recommendations offered by the evaluator that seem to leap beyond the documented evidence. On the other hand, it is often beneficial to capture insights that result from immersion in a project, insights not provided by sticking only to results obtained through scientific documentation. The evaluator may have developed a strong, intuitive sense of how the project really worked out, what were its best or worst features, and what benefits accrued to participants or to institutions impacted by the project (for example, schools or school systems). Thus, there may be a need to stretch the data beyond their inherent limits, or to make statements for which the only supporting data are anecdotal.

We have several suggestions for dealing with these issues:

• Distinguish carefully between conclusions that are based on "hard" data and those that are more speculative. The best strategy is to start the conclusions section with material that has undergone thorough verification and to place the more subjective speculations toward the end of the section.
• Provide full documentation for all findings where available. Data collection instruments, descriptions of the study subjects, specific procedures followed for data collection, survey response rates, refusal rates for personal interviews and focus group participation, access problems, etc., should all be discussed in an appendix. If problems were encountered that may have affected the findings, possible biases and how the evaluator sought to correct them should be discussed.
• Use the recommendations section to express views based on the total project experience. Of course, references to data should be included whenever possible. For example, a recommendation in the report for the hypothetical project might include the following phrase: "Future programs should provide career-related incentives for faculty participation, as was suggested by several participants." But the evaluator should also feel free to offer creative suggestions that do not necessarily rely on the systematic data collection.

Maintaining Confidentiality

All research involving human subjects entails possible risks for participants and usually requires informed consent on their part. To obtain this consent, researchers usually assure participants that their identity will not be revealed when the research is reported and that all information obtained through surveys, focus groups, personal interviews, and observations will be handled confidentially. Participants are assured that the purpose of the study is not to make judgments about their performance or behavior, but simply to improve knowledge about a project's effectiveness and improve future activities.

In quantitative studies, reporting procedures have been developed to minimize the risk that the actions and responses of participants can be associated with a specific individual; usually results are reported for groupings only and, as a rule, only for groupings that include a minimum number of subjects.

In studies that use qualitative methods, it may be more difficult to report all findings in ways that make it impossible to identify a participant. The number of respondents is often quite small, especially if one is looking at respondents with characteristics that are of special interest in the analysis (for example, older teachers, or teachers who hold a graduate degree). Thus, even if a finding does not name the respondent, it may be possible for someone (a colleague, an administrator) to identify a respondent who made a critical or disparaging comment in an interview.

Of course, not all persons who are interviewed in the course of an evaluation can be anonymous: the name of those persons who have a unique or high status role (the project director, a college dean, or a school superintendent) are known, and anonymity should not be promised. The issue is of importance to more vulnerable persons, usually those in subordinate positions (teachers, counselors, or students) who may experience negative consequences if their behavior and opinions become known.

It is in the interest of the evaluator to obtain informed consent from participants by assuring them that their participation is risk-free; they will be more willing to participate and will speak more openly. But in the opinion of experienced qualitative researchers, it is often impossible to fulfill promises of anonymity when qualitative methods are used:

The evaluator may also find it difficult to balance the need to convey contextual information that will provide vivid descriptive information and the need to protect the identity of informants. But if participants have been promised anonymity, it behooves the evaluator to take every precaution so that informants cannot be linked to any of the information they provided.

In practice, the decision of how and when to attribute findings to a site or respondent is generally made on a case-by-case basis. The following example provides a range of options for revealing and disclosing the source of information received during an interview conducted for the hypothetical project:

• Attribute the information to a specific respondent within an individual site: "The dean at Lakewood College indicated that there was no need for curriculum changes at this time."

• Attribute the information to someone within a site: "A respondent at Lakewood College indicated that there was no need for curriculum changes at this time." In this example, the respondent's identity within the site is protected, i.e., the reader is only made aware that someone at a site expressed a preference for the status quo. Note that this option would not be used if only one respondent at the school was in a position to make this statement.

• Attribute the information to the respondent type without identifying the site: "The dean at one of the participating colleges indicated that there was no need for curriculum changes at this time." In this example, the reader is only made aware of the type of respondent that expressed a preference for the status quo.

• Do not attribute the information to a specific respondent type or site: "One of the study respondents indicated that there was no need for curriculum changes at this time." In this example, the identity of the respondent is fully protected.

Each of these alternatives has consequences not only for protecting respondent anonymity, but also for the value of the information that is being conveyed. The first formulation discloses the identity of the respondent and should only be used if anonymity was not promised initially, or if the respondent agrees to be identified. The last alternative, while offering the best guarantee of anonymity, is so general that it weakens the impact of the finding. Depending on the direction taken by the analysis (were there important differences by site? by type of respondent?), it appears that either the second or third alternative 2 or 3 represents the best choice.

One common practice is to summarize key findings in chapters that provide cross-site analyses of controversial issues. This alternative is "directly parallel to the procedure used in surveys, in which the only published report is about the aggregate evidence" (Yin, 1990). Contextual information about individual sites can be provided separately, e.g., in other chapters or an appendix.

Tips for Writing Good Evaluation Reports

Start early. Although we usually think about report writing as the final step in the evaluation, a good deal of the work can (and often does) take place before the data are collected. For example, a background section can often be developed using material from the original proposal. While some aspects of the methodology may deviate from the original proposal as the study progresses, most of the background information (e.g., nature of the problem, project goals) will remain the same throughout the evaluation. In addition, the evaluation study questions section can often be written using material that was developed for the evaluation design. The evaluation findings, conclusions, and recommendations generally need to wait for the end of the evaluation.

Because of the volume of written data that are collected on site, it is generally a good idea to organize study notes as soon after a site visit or interview as possible. These notes will often serve as a starting point for any individual case studies that might be included in the report. In addition, as emphasized in Chapter 4, preparing written text soon after the data collection activity will help to classify and display the data and reduce the overall volume of narrative data that will eventually need to be summarized and reported at the end of the study. Finally, preparing sections of the findings chapter during the data collection phase allows researchers to generate preliminary conclusions or identify potential trends that can be confirmed or refuted by additional data collection activities.

Make the report concise and readable. Because of the volume of material that is generally collected during mixed method evaluations, a challenging aspect of reporting is deciding what information might be omitted from the final report. As a rule, only a fraction of the tabulations prepared for survey analysis need to be displayed and discussed. Qualitative field work and data collection methods yield a large volume of narrative information, and evaluators who try to incorporate all of the qualitative data they collected into their report risk losing their audience. Conversely, by omitting too much, evaluators risk removing the context that helps readers attach meaning to any of the report's conclusions. One method for limiting the volume of information is to include only narrative that is tied to the evaluation questions. Regardless of how interesting an anecdote is, if the information does not relate to one of the evaluation questions, it probably does not belong in the report. As discussed previously, another method is to consider the likely information needs of your audience. Thinking about who is most likely to act upon the report's findings may help in the preparation of a useful and illuminating narrative (and in the discarding of anecdotes that are irrelevant to the needs of the reader).

The liberal use of qualitative information will enhance the overall tone of the report. In particular, lively quotes can highlight key points and break up the tedium of a technical summation of study findings. In addition, graphic displays and tables can be used to summarize significant trends that were uncovered during observations or interviews. Photographs are an effective tool to familiarize readers with the conditions (e.g., classroom size) within which a project is being implemented. New desktop publishing and software packages have made it easier to enhance papers and briefings with photographs, colorful graphics, and even cartoons. Quotes can be enlarged and italicized throughout the report to make important points or to personalize study findings. Many of these suggestions hold true for oral presentations as well.

Solicit feedback from project staff and respondents. It is often useful to ask the project director and other staff members to review sections of the report that quote information they have contributed in interviews, focus groups, or informal conversations. This review is useful for correcting omissions and misinterpretations and may elicit new details or insights that staff members failed to share during the data collection period. The early review may also avoid angry denials after the report becomes public, although it is no guarantee that controversy and demands for changes will not follow publication. However, the objectivity of the evaluation is best served if overall findings, conclusions and recommendations are not shared with the project staff before the draft is circulated to all stakeholders.

In general, the same approach is suggested for obtaining feedback from respondents. It is essential to inform them of the inclusion of data with which they can be identified, and to honor requests for anonymity. The extent to which other portions of the write-up should be shared with respondents will depend on the nature of the project and the respondent population, but in general it is probably best to solicit feedback following dissemination of the report to all stakeholders.

References

Miles, M.B., and Huberman, A.M. (1994). Qualitative Data Analysis, 2nd Ed. Newbury Park, CA: Sage.

National Science Foundation. (1993). User-Friendly Handbook for Project Evaluation: Science, Mathematics, Engineering, and Technical Education. NSF 93-152. Washington, DC: NSF.

Patton, M.Q. (1990). Qualitative Evaluation and Research Method, 2nd Ed. Newbury Park, CA: Sage.

Yin, R.K. (1989). Case Study Research: Design and Method. Newbury Park, CA: Sage.