Development of the CAHPS Health Information Technology Item Set

Nearly all health care providers have incorporated various forms of health IT into their practices. The pace of change has been driven in large part by pressure from public and private purchasers to adopt technologies that promise to improve the quality and efficiency of care.

To assess the impact of these technologies on patients, AHRQ asked the CAHPS Consortium to develop an item set that would enable survey sponsors to capture both the positive and negative effects of health IT in the practice setting from the patients' perspective. For example, how do electronic medical records and personal health records affect the timeliness of care, physician-patient communication, coordination of care, and the accessibility of information? Does health IT change the patients' perceptions of involvement in, and sense of control over, their health?

The development process for the CAHPS Health Information Technology Item Set included several steps:

Stakeholder meetings. In June 2006, AHRQ hosted a meeting with health IT and CAHPS survey experts, as well as stakeholders representing health care organizations and consumers. This group discussed the use of health IT by physician practices, the aspects of patient care likely to be affected by health IT, the areas that consumers would be best suited to comment on, and sources of related information, including potential items.

Learn more in this summary of the Health Information Technology CAHPS^® Stakeholders Meeting, June 28, 2006 (PDF, 293 KB).

Stakeholders convened again in September 2008 to provide feedback on the item set domains, options for field testing, and issues with the adoption and use of the item set.

Literature review. The CAHPS team completed a literature review of interactions between health IT, health care providers, and patients.

Focus groups. The CAHPS team conducted focus groups with health IT users. The goals of the focus group sessions were to explore:

• What health care functions are important to patients;
• How patients view health IT’s role in facilitating those functions;
• What patients believe they are able to report on regarding health IT; and
• What patients regard as the most important roles for health IT in a physician practice.

Item inventory. Based on the expert meeting, literature review, and focus groups, the grantees developed an inventory of items, sorted them by domains, and set priorities for the domains. Those priorities determined which items were selected for cognitive testing.

Site interviews. The team conducted in-depth interviews with health plans and providers that use health IT. The interviews explored issues around electronic and personal health record architecture, functions most frequently used by providers and patients, and the percentage of patients registered to use personal health records.

Cognitive testing. The draft items were cognitively tested with consumers to ensure that they can accurately understand and interpret the questions and response options. After two rounds of cognitive testing, the team incorporated the findings into an item set that was used for field testing.

Field testing. The Health IT Item Set was field tested in three health care organizations: Beth Israel Deaconess Medical Center in Massachusetts, Group Health Cooperative in Washington, and Kaiser Permanente in Southern California. Using data from those three sites, the CAHPS Team analyzed the reliability, validity, and other psychometric properties of the items and composite measures.

Composite label testing. The composite measure labels were tested with consumers to ensure that consumers understood the labels correctly.

Public release. In 2011, AHRQ released the CAHPS Health Information Technology Item Set for use with the original CAHPS Clinician & Group Survey. These supplemental items were revised slightly in 2012 and 2013 to align the questions and placement instructions with the 2.0 version of the Clinician & Group Surveys and related supplemental items. In 2017, the items and placement instructions were modified to fit with the 3.0 version of the Clinician & Group Survey.

The CAHPS Consortium would like to acknowledge the following organizations for their support of the development of this item set: Beth Israel Deaconess Medical Center, Group Health Cooperative, and Kaiser Permanente Southern California.