Evaluating the Results of a Quality Reporting Project

The purpose of results- or outcome-oriented evaluation goes beyond answering the “did it work” question. To evaluate results, however, you have to be clear about what you wanted to achieve.

• What consumer audience were you trying to reach?
• What changes did you want to see in their behavior or attitudes?
• What about other audiences? Who were they and what changes were you looking for?

You also have to be realistic about timing, that is, when it is likely that you will observe various results.

To the extent that report sponsors can demonstrate and document that they are making a difference—that their efforts, over time, are bearing fruit—they can expect more support, including financial support as well as legitimacy and recognition.

The Purpose of a Summative Evaluation

Evaluators refer to efforts to judge the worth and value of a project that has been fully implemented and refined as a summative evaluation. A summative evaluation may not be feasible for a couple of years or more after you get going. The audience for a summative evaluation includes those doing the work as well as funders and the involved stakeholders. A summative evaluation should focus on results, but it’s helpful to get enough information on processes to enable you to figure out why you got the results you did.

You might undertake a summative evaluation to determine the following:

• Should you continue with your current report design, dissemination, and promotion strategies or fundamentally change them? If so, how?
• Would it be a good return on investment to expand your efforts (and your budget)? You might expand in terms of the measures you report, the region you cover, or the audiences you target.
• If you have done different kinds of reports, what are the benefits of each kind? What problems are associated with each kind?

Example: Evaluation of “California’s Quality of Care Report Card”

Title: Evaluation of California’s HMO Report Card.
Sponsor: California Office of the Patient Advocate
URL: http://www.ucdmc.ucdavis.edu/chpr/downloads/ReportText.pdf.

In 2005, the Center for Health Services Research in Primary Care, University of California, Davis, conducted an evaluation of the 2003-2004 "California’s Quality of Care Report Card" on behalf of the State of California’s Office of the Patient Advocate (OPA). Questions addressed by this evaluation included:

• Are California health care consumers using California’s Quality of Care Report Card?
• How useful to consumers are the quality indicators that are currently included in the Quality Report Card?
• Does the Quality Report Card impact the participating HMOs and medical groups?

Source: Rainwater JA, Lowey-Ball A, Enders SR, et al. Evaluation of California’s HMO Report Card. A Report for the California Office of the Patient Advocate. Davis, CA: UCD Center for Health Services Research in Primary Care; 2005. Available at http://www.ucdmc.ucdavis.edu/chpr/downloads/ReportText.pdf.

Questions To Assess Results

An evaluation of results could focus on a variety of questions. To learn which methods would be most appropriate to answer questions about results, go to Data Collection Methods: Matching Questions to Methods.

• Did the consumer/patient audience you were targeting become aware of the report?
  • Did they remain aware of it over time?
  • Did they seek out the report? Did they find it?
• Did the consumer/patient audience understand the report?
• Did consumers/patients use the report?
  • If they used it, how? To make plan and provider decisions? To initiate a discussion with their physician?
  • Did they share the report’s content with family and friends?
• Over time, did the availability of a report card make a difference in the market share of plans and providers? Did those with higher scores attract more customers?
• How did providers and plans, especially (but not only) the ones you measured, respond to the report?
  • Did it guide or intensify their quality improvement efforts?
  • With what results?
• How did purchasers respond to the report?
  • How did they use it?
  • With what results?
• Did policymakers use the report?
  • If so, how?
  • Did the report lead them to take any actions to improve health care quality?
• How has this effort affected the direction and support for future reporting efforts?

You may get different responses from different audience segments, so you may need to refine the questions above to learn who did and did not become and remain aware, seek out the report, understand it, use it, and so on.

Also in "What You Can Evaluate"

• Evaluating the Process
• Evaluating Results
• Evaluating Cost-Effectiveness