How Will You Test Aspects of Your Health Care Quality Report?

Given all the time, money, and effort you are investing in developing and distributing a quality report, you want to be sure that your report is as effective as possible. In other words, you want every aspect of the report to work as well as it can for your audience.

The best way to do this is to test each aspect of your report and your dissemination strategy with your audience and other stakeholders during the developmental phase of the project. This testing is especially important when you are introducing information that is new and complex.

This kind of testing is distinct from evaluating your report and your dissemination strategy after it has been carried out. To learn more about that aspect of a project, go to Assess Your Health Care Quality Reporting Project.

Why Test Materials With Your Audience?

The main reason to test various aspects of your quality report is to see the information from your audience's perspective. The testing of materials and strategies can reveal useful information about the audience’s needs, attitudes, and capabilities.

What Testing Can Tell You

At its best, testing at the developmental stage (sometimes referred to as formative testing or pretesting) can offer valuable insights into how your audience thinks about the issues addressed by your report and the reasons for certain attitudes or misunderstandings. You can use these insights to:

• Determine which kinds of health care organizations they care about and what they really want to know.
• Refine your messages and materials to communicate clearly and effectively.
• Identify language that will be meaningful to your audience.
• Determine whether people are interpreting the comparative information correctly.
• Uncover issues that may be culturally sensitive.
• Discover ways of reaching the people who most need and want the information.

Why You Can’t Know All This Yourself

While you may know a lot about the issues in your report, you cannot rely solely on your own judgment or that of your colleagues. Also, while you may have much in common with your audience—whether consumers or the health care organizations that are the subject of your report—you are far too close to the work of quality measurement, reporting, and improvement to see things as they do.

Here are some reasons why:

• Your understanding of the issues and terminology isn't representative of your audience’s understanding.
• Details that seem inconsequential to you may be important to your audience or vice versa.
• Your messages may be misinterpreted or create misconceptions.
• You may not be aware of whether your materials are inconsistent with the cultural norms of certain groups within your audience.

The Basics of the Audience Testing Process

Define Your Objectives

Before you initiate testing with your target audience, form a clear statement of your objectives:

• What is it that you want to learn through testing?
• How will you use what you learn to improve the report?

For example, you may want to know whether the language you used to explain the quality data in your report made sense to your audience. If it didn’t, you need to be prepared to rewrite the text with terminology and phrasing that your audience will find more understandable. Similarly, you may want to learn whether the audience is interpreting the data displays accurately. If they’re not, will you redo your data displays? Consider whether you have someone ready to make the necessary changes and whether they’ll have sufficient time and resources to do that work.

Focus on What You Need To Know

To help decide how to focus your testing efforts, consider the following questions:

• What do you already know about the audience for this report?
• What do you know about your audience’s views on health care quality, either in general or regarding the quality of a particular kind of provider?
• Have you conducted or do you have access to related research that you can use to anticipate how your audience might respond to the design and content of a quality report?

Use your answers to these questions to figure out what else you might want to know.

What To Test

You can use interviews with representatives of your audience to find out how they respond to different characteristics of your report: the messages, the text and graphics, design and navigation, and usability. Here are some issues to explore when you test a report card:

• Are people interpreting the information in the way it is intended?
• Do they think that the information is:
  • Understandable?
  • Useful?
  • Attractive?
  • Personally relevant?
  • Persuasive?
  • Helpful?
• What do they like and dislike about the report?
• Is the level of detail appropriate?
• Do they trust the accuracy of the information and its source?
• Do they think they’d use the information? When and how?
• What would they change about the report?

How To Test

To answer your questions, you need to give members of your intended audience an opportunity to react to existing materials or drafts of new materials. No matter how hard you try, you will not be able to predict how your audience will respond to the information you give them. Let them tell you what's appropriate, what works well, and what doesn’t.

The most effective way to do this is a technique called cognitive interviewing, where you probe a small number of respondents—typically no more than 10—on their understanding and interpretation of the language and other elements of your report. For these interviews, you can give them one version to respond to, or you can let them compare versions of the design, the visuals (such as photos or graphs), a web page, or the copy to see if they like one better than another.

This kind of testing will help you identify problems, but not necessarily solutions, so it is not a one-time event. If at all possible, plan on multiple rounds of testing to refine the different aspects of your report. For example:

• In the first round of testing, you may learn that respondents are confused by a table that shows how several medical groups performed on a given set of quality measures.
• In the second round, you could test a redesigned table that uses different symbols.
• In a third round, you may let respondents compare the revised table with one that has a different design.

Both time and resources are usually limited, so do what you can. Some testing is almost always better than none at all.

Learn more about The Purpose and Process of Cognitive Testing.