Patient safety is a public health problem in the United States and abroad. In the late 1990s, up to 42 percent of the respondents in a Louis Harris Poll reported that they or a close friend or relative had experienced a medical mistake.4 A more recent public opinion poll in Massachusetts found that nearly one in four adults had personally experienced a medical error in the past 5 years (or someone close to them had), and half of them reported that the medical error resulted in serious health consequences.5 Most of the reported errors (75 percent) occurred in hospitals. The 2014 National Quality and Disparities Report showed that more Americans are getting safer care now than in the past, but it also indicated that the problem persists: there were 121 adverse events per 1,000 hospitalizations in 2013.6
Researchers have confirmed that 5 to 10 percent of hospitalized patients suffer injuries,7-13 many of which could be prevented by implementing patient safety-related activities such as those focused on reducing infections, implementing surgical checklists, and introducing health information technology (IT) to improve medication safety.14-16 To guide such improvement initiatives and to provide for accountability, the Institute of Medicine (IOM) report To Err is Human: Building a Safer Healthcare System recommended establishment of a nationwide reporting system for collecting standardized information about serious adverse events, which would be used to identify issues that require additional analyses or a broad-based response and would allow for public investigation and remediation of serious events.17 A subsequent IOM report called for standardization of reporting and the development of a common taxonomy.2 This report listed the critical “domains” that should be included in patient safety event reports, including discovery (by whom and how); attributes (what, when, where, who, why, and an assessment of risk); a narrative account including contributing factors; and, if appropriate, a causal analysis with lessons learned. However, the primary focus of the IOM recommendations was on the reporting of patient safety events by health care professionals. Most adverse event reporting systems (with a few exceptions) are designed to be used by health care providers rather than consumers.
Value of Patient Reporting
In recognition of the fact that some patients have the capacity to serve as “vigilant partners,” patient participation in their own safety has been identified as a potentially important safety improvement strategy.18 At least three attributes account for the potential power of patients’ participation: intimate knowledge of their symptoms and treatments, motivation to ensure favorable outcomes, and proximity to care.19 In a series of studies, patients endorsed their willingness to perform error-prevention activities such as asking questions about medications and medical care, helping to mark a surgical site, and reporting an error to the medical staff.18, 20-25
Building on the concept of the vigilant partner, researchers have shown that patients are able to recognize medical errors, some of which are not otherwise identified by existing health care monitoring systems,26 and are willing and able to report this information reliably.27-33 As demonstrated in the United Kingdom (UK), these reports can strengthen the ability of health care organizations to detect systemic problems in care.
In an early study, Weingart and colleauges31 found that 8 percent of inpatients reported adverse events, and 4 percent experienced “near misses.” Importantly, none of these events were documented in the hospital’s adverse event reporting systems. Other groups have used similar methods to elicit patient reports in the emergency department and in a children’s hospital.34-36 In the largest study of its kind, Weissman and colleagues1 compared patient reports with medical records and found that 23 percent of the study patients had at least one adverse event detected by interview, and 11 percent had at least one adverse event identified by medical record review. Two-thirds of the adverse events were detected by patient interview alone, demonstrating that patients could identify adverse events of which the hospital was unaware.
This principle applies in ambulatory settings as well. Weingart and colleagues29 found that 73 percent of adverse events occurring in primary care practices were identified by patient report only, 9 percent by chart review only, and 19 percent by both. Similarly, Weingart and colleagues, studying adverse event reports from ambulatory oncology patients, reported that 20 percent of patients identified a concern about safety.32, 37
These researchers succeeded in eliciting patient safety reports through patient interviews; however, patient reports frequently require recoding or classification to enable the extraction of the key content, resolve discrepancies or inconsistencies, and assess the attribution of symptoms or complications to care rather than to the natural history of disease. To achieve scale would require the development and deployment of consumer reporting systems that can routinely collect such information, analyze these reports efficiently, and generate results that can motivate patient safety improvement actions within and across health care organizations. In the absence of such simplified data collection approaches, widespread implementation of a consumer-oriented reporting system would be difficult.
Consumer Reporting Systems: Realizing the Potential
Patient reporting remains a promising but unrealized approach for identifying patient safety hazards in health care operations. As reported by Research Triangle Institute (RTI) and Consumers Advancing Patient Safety (CAPS) for AHRQ,38 multiple patient safety reporting systems exist, but only a minority of them permit consumer reporting. These include the Joint Commission; the Institute for Safe Medication Practices’ Medication Event Reporting Program (MERP); the Food and Drug Administration Safety Information and Adverse Event Reporting Program (MedWatch); and the United Kingdom’s National Reporting and Learning System (NRLS). Three systems focus exclusively on patient reports (a Kaiser Permanente system and medication reporting systems in the Netherlands and Australia). These systems, typically focused on communities or patients, have had mixed success in improving the quality, volume, representativeness, and utility of patient safety reports. For example, the UK National Patient Safety Agency recorded only 12 reports in its 3-year operation.39 In contrast, the Danish Cancer Society’s patient reporting system provided enough information and value to encourage the creation of a national patient safety reporting system.40 However, no well-established model exists for eliciting consumer-identified patient safety events on the scope envisioned by AHRQ.
Although consumer reporting has the potential to increase knowledge about system failures amenable to analysis and remediation, patient input about care received is currently solicited mainly through patient experience surveys (such as the Consumer Assessment of Healthcare Providers and Systems [CAHPS] and Press Ganey), which tend to focus on interpersonal aspects of care, communication, and access. These surveys do not explicitly include questions on adverse events, and they typically feature closed-ended questions that do not allow patients to provide narrative information.
In addition to the need for technical infrastructure, logistic considerations, and analytic capabilities for consumer-specific event reporting systems, measures must be taken to ensure that event reporting is acceptable to patients. At Dana-Farber Cancer Institute, for example, a proposal to collect patient event reports was initially met with concern about putting patients in the position of criticizing caregivers.41 The Danish Cancer Society’s consumer reporting system40 invites both adverse event reports and positive anecdotes. (This is not to suggest that one effort got it wrong and another got it right—both projects went through various iterations before settling on an approach.)
These past efforts illustrate the importance of design decisions to the success of consumer reporting systems. One important design decision is whether to use “prompted” methods or “passive” strategies to generate reports. For example, an application called PatientSite sent patients an electronic message 10 days after they received a new or changed prescription, inquiring about problems with the medication.42 In an evaluation, 50 percent of 267 respondents reported problems filling their prescriptions, 12 percent noted problems with drug effectiveness, and 10 percent described a medication-related symptom. Prompted reporting may lower the threshold for reporting without reducing the severity of reported incidents.
A second design decision is whether to link patient reports with provider reports. Doing so may have some advantages. Patients may be able to provide unique information that is not already collected by health care organizations and reported to a PSO for the conduct of patient safety activities. A PSO, authorized under the Patient Safety and Quality Improvement Act of 2005, can collect, aggregate, and analyze confidential information reported by health care providers. The law provides Federal privilege and confidentiality protections for information that is assembled and reported by providers to a PSO (“patient safety work product [PSWP]”).b Information obtained from clinician reports can further understanding of patient reports and vice versa. Consumer-reported information may spark safety improvement actions that might not be identified otherwise. Connecting risk management to the patient complaint and grievance process required by the Centers for Medicare & Medicaid Services (CMS) is a well-established practice, and it is therefore not difficult to envision that providers will glean useful information from consumer safety reports.
A third design decision is whether to allow patients and family members to report anonymously—and whether the advantage of having identifiable reports outweighs the possibility that some patients, fearing retribution against themselves or their health care providers, will not report at all, meaning their patient safety concerns will remain unaddressed.
In summary, existing research has identified the opportunity to engage consumers in reporting about health-related safety events, but it also points to many difficult challenges, including the need to design systems that: (1) are acceptable to patients, (2) elicit structured and narrative reports using strategies designed to maximize the volume of reports, (3) address care in a variety of practice settings, (4) take advantage of electronic technologies, and (5) allow for coding and classification of preliminary patient reports in a way that ensures quality control and utility to providers and other stakeholders seeking to improve the quality and safety of care.
The Charge from AHRQ
In an effort to explore the untapped potential of having health care consumers provide important information about patient safety events, AHRQ initially awarded a contract to RTI and CAPS to identify key design elements of a consumer reporting system. The researchers collected and summarized information about a variety of patient safety reporting systems around the world and highlighted a unique and critical role for consumer reporting systems. Their report states, in part:
Several reasons can be provided to substantiate the importance of consumer reporting systems. First, not all patient safety events are known (or knowable) in the absence of consumer reporting; many events may not be noticed or detected if … not reported by a consumer. This may particularly apply to … events that occur in outpatient settings, where there is less likely to be surveillance than in hospitals. Also, even if an event is identified without consumer reporting, consumers are likely to be able to provide additional important information…. the consumer perspective is a unique source of information for understanding the contributing factors associated with patient safety events, the response of health care providers and systems to these events, and the subsequent impact of events on patients and their families. In many cases, information from health care professionals may not be sufficient to understand a patient safety event … Further … providing consumers with the opportunity to report events allows them to be active participants in the pursuit of improvements in patient safety. Beyond positive impacts of this role for consumers, involvement of consumers … may increase the level of vigilance among providers and organizations, and may increase motivation to produce system change and thereby improvements in patient safety (RTI International, 2010, pp. 6-1 and 6-2).38
Using an iterative, consensus-building process, RTI and CAPS identified six recommendations (and sub-recommendations) for developing an “ideal” consumer reporting system (see Appendix A). These recommendations provided general guidance on the type of information to be collected, the source of the reporting, the types of mechanisms that could be used, the level of operations (local, regional, national, international), the type of infrastructure and analytic functionality needed, modalities and formatting of reports, and whether such data should be linked to provider reports. No implementation recommendations were made other than strongly encouraging pilot development and testing.
AHRQ subsequently sought to determine the feasibility of creating a robust and scalable patient safety reporting system. On June 14, 2011, AHRQ requested proposals through its Accelerating Change and Transformation Through Organization and Networks (ACTION) II contracting mechanism to develop and test a prototype reporting system for patient safety events. As stated in the Request for Task Order, the purpose of the project was to:
- Design and develop a consumer reporting system for patient safety events, using previously determined recommendations for such a system.
- Test the prototype consumer reporting system in a variety of settings and utilizing a variety of methods for patient reporting.
- Collect and analyze consumer reports to determine patterns in events and root causes of such events.
- Compare and contrast those events reported by consumers with those reported by health care professionals, noting the differences and similarities.
- Develop plans for any needed modification and recommendations for expansion of such a system (or systems) nationwide, based on the results of the prototype testing.
The contract was awarded to the RAND Corporation and its ACTION II teaming partner, ECRI Institute, along with collaborators at Tufts Medical Center and Brigham and Women’s Hospital in Boston.
Focus of This Research
The research team undertook a project to design, pilot test, and evaluate a prototype for collecting narrative and structured data about concerns that patients have about the safety of their health care, including errors and adverse events. In developing the prototype, called the Health Care Safety Hotline, the research team sought to address several requirements:
- Patients (and other potential reporters, such as family members and other caregivers) had to be able to report online or over a toll-free phone number with a human interface.
- The prototype had to include a formal relationship with a PSO or PSOs.
- The collection of data was to focus on adverse events, near-miss events, and unsafe conditions as perceived by patients to describe the risks and hazards in the delivery of care across the continuum of care.
- The AHRQ Common Formats were to be used to describe patient safety events.
The contract also specified the need for an outreach and information campaign for recruiting health care delivery organizations and for marketing the hotline to consumers. The prototype was to be designed to enable feedback of the narrative and structured data to health care organizations and PSOs.
Project Phases
In the first phase of the project (September 2011 to September 2013), the research team designed and developed the prototype. In the second phase (September 2013 to September 2015), the team implemented the hotline in the pilot communities, tested and refined the prototype, evaluated the types of reports entered by patients and caregivers, and documented the experience of the health care organizations and professionals using the patient event report data to improve care. A multi-stakeholder technical expert panel (TEP) reviewed and provided feedback on each step of the design, implementation, and evaluation process.
Overview of Methods
The research team employed a variety of methods to design, develop, and evaluate the hotline prototype. First, we conducted a comprehensive environmental scan and literature review to identify promising models of consumer patient safety reporting systems, instruments used to collect data, and lessons learned to date. We then used focus groups to identify important domains and language to be included in the event reporting form and used cognitive interviews to create and test the wording of individual items. Using the results of the scan and literature review and employing the new event reporting form, the research team designed the architecture and infrastructure required to operationalize a Web- and phone-based process for data collection, refinement, classification, and reporting. The proposed model was reviewed by the TEP, which included consumer representatives, patient advocates, patient safety experts, health IT experts, physicians, patient and consumer reporting experts, and survey research and reporting experts. Modifications were made in response to the TEP review. In addition, the prototype model was published in the Federal Registerc for public comment, in accordance with requirements of the Paperwork Reduction Act of 1980,d under direction of the Office of Management and Budget (OMB). Two opportunities for public comment were provided as part of the OMB process, resulting in several hundred comments that led to additional modifications of the prototype. The team conducted several live “webinars” to demonstrate the final Web-based prototype to the TEP, AHRQ officials, and candidate health care organizations.
After the design and development phase was completed, the team recruited two health care delivery organizations within a single community to pilot-test the prototype. The team worked with the two organizations to clarify the legal and regulatory framework for operation of the prototype, developed and implemented an outreach and marketing plan, refined protocols and tested the prototype, and began hotline operations in January 2014.
During the operations period, the research team collected and analyzed Web traffic statistics, analyzed patient-reported events, standardized event descriptions, and revised the reporting instrument. As a part of the evaluation of implementation, the team also conducted a site visit to each of the health care delivery organizations and also met with their Patient and Family Advisory Councils (PFACs) to solicit advice and feedback.
Purpose and Outline of This Report
This report provides findings from all phases of the research project. Chapter II provides additional detail on hotline design and development, including the design of the patient report form. Chapter III describes hotline implementation and refinement. Chapter IV describes the aims, methods, and results of the evaluation. The goal of the evaluation was to test whether the prototype could be used to collect meaningful information about patient safety concerns across a range of settings and to understand challenges in triaging and sharing that information with health care organizations and the public. Chapter V summarizes some of the challenges and lessons learned. Chapter VI presents conclusions and recommendations to AHRQ. The report also contains four appendixes. Appendix A includes the RTI recommendations for an ideal consumer reporting system. Appendix B lists the members of our TEP and their professional affiliations. Appendix C is the Operations Manual, which contains the consumer event reporting forms, surveys, Web content, telephone scripts, specifications for operating the Web site, and other materials. Finally, Appendix D includes the interview discussion guide for the site visits to our community partner organizations.
b The law has specific requirements for PSOs, including the participation of multiple entities and the use of standard reporting formats. For a brief description, see archive.ahrq.gov/news/newsroom/press-releases/2008/psoact.html, accessed August 21, 2015.
c “Agency for Healthcare Research and Quality, Agency Information Collection Activities (Proposed Collection; Comment Request),” Federal Register 77:175 (September 10, 2010) p. 55475. Available from Federal Register.gov, accessed August 30, 2015.
d The Paperwork Reduction Act of 1980 (Pub. L. No. 96-511) is a federal law designed to reduce the total paperwork burden the federal government imposes on businesses and individual citizens. The Act imposes procedural requirements on federal agencies (such as AHRQ) that wish to collect information from the public. The statute authorized the White House Office of Management and Budget (OMB) to establish policies on the collection of information and to oversee the implementation of the requirements.
