With funding from the Agency for Healthcare Research and Quality (AHRQ), researchers at the RAND Corporation, its ACTION II Teaming Partner ECRI Institute, Tufts Medical Center, and Brigham and Women’s Hospital designed, developed, implemented, and evaluated a prototype consumer reporting system for patient safety, called the Health Care Safety Hotline. The prototype was intended to enable collection of reports from consumers (i.e., patients and their proxies, including families, friends, and other caregivers) about patient safety problems with their health care. During the first phase of the project, the research team designed and assembled the key building blocks of the hotline prototype, including a patient-oriented event reporting form, a Web-based data collection platform and content, and protocols for data collection, data processing, and data sharing. In the second phase of the project, the hotline prototype was implemented and evaluated in a carefully selected pilot community, with the participation of two local health care delivery organizations.
Development of the Hotline
The patient reporting form in the prototype was based on existing resources, including a previously published survey developed by the project team members,1 minimum dimensions for reporting patient safety events laid out by the Institute of Medicine,2 and the AHRQ Common Formats,a which includes definitions of adverse events and forms for adverse event reporting. The team conducted an environmental scan of all existing consumer-enabled reporting systems and identified 27 relevant systems. These systems were then analyzed by event type, reporting mode, key terms used, and other criteria. Analysis of these instruments highlighted several gaps and also a lack of consensus on key issues, such as the optimal terminology to be used for consumer reporting and consumer expectations about what might occur as a result of a submitted report.
To fill gaps in existing knowledge and to address unresolved issues, the research team conducted two focus groups, one in English and one in Spanish. The focus groups helped us to determine whether patients and caregivers could recognize and were willing to report adverse events, to identify the preferred terminology to be used in reporting, and to understand consumer concerns and expectation about reporting.
The patient reporting form was revised in response to the focus group findings, and the revised form was tested in cognitive interviews with patients and family members. Using insights obtained in the cognitive interviews, the research team refined question content and wording, shortened the reporting form dramatically, replaced some of the terminology, and added open text boxes to encourage narrative reporting.
The team next developed protocols for handling consumer-reported patient safety observations. An intake protocol was designed for assessing the confidentiality and anonymity preferences of the patient or caregiver and for screening the reports on the basis of inclusion and exclusion criteria (such as completeness, coherence, and type of report).
A process was developed for following up with the consumers to clarify details (as appropriate and with permission from the consumers) and to enter the report into a database. If a patient gave permission for the information to be shared with the relevant health care organization, a copy of the report was provided to that organization. This enabled health care organizations to match and supplement the consumer report with any relevant information collected from other sources within their organizations (such as an existing adverse event reporting system for health care professionals). The information provided by the health care organization was entered into a second database, a patient safety organization (PSO) database, within which the report was protected as a “patient safety work product” under the Patient Safety and Quality Improvement Act of 2005.3
After the reporting form and protocols for handling reports were developed, the research team initiated the prototype review process. They first presented the prototype to a technical expert panel (TEP). The TEP members provided multi-stakeholder expertise on relevant dimensions of the project, including patient safety, reporting systems, patient and consumer perspectives, and survey methodology. In response to feedback from the TEP, the researchers revised the patient event reporting form and other aspects of the prototype before submitting it to the Office of Management and Budget (OMB) for public comment (required under the Paperwork Reduction Action of 1980). Additional revisions were made in response to the public comments, and the final versions were submitted to OMB for review. The team then demonstrated the prototype to project officers at AHRQ, the TEP, and organizational leaders in the candidate pilot community. RAND received OMB approval to deploy the prototype on August 31, 2013.
Implementation and Evaluation of the Hotline
In the next phase of the project, the research team implemented, refined, and evaluated the hotline prototype to assess its feasibility, yield, and potential scalability and to recommend any necessary modifications. Two highly integrated health care organizations in a single community—each of which already had a relationship with a PSO—were recruited. Together with these health care organizations, the research team developed and implemented an outreach and marketing plan to alert consumers to the availability of the hotline.
The Health Care Safety Hotline was launched in January 2014, and staffed by members of the research team, who kept in close contact with the two health care organizations over the 17 months of hotline operations. Health care leadership, patient safety and risk management staff, and Patient and Family Advisory Councils provided guidance and feedback. Because of OMB requirements, only minor adjustments to the patient event reporting form and Web content were made during the operation period.
The evaluation of the prototype was designed to analyze how resources related to the marketing and promotion of the hotline and the modes used to provide access to it affected the flow of its operations and the numbers and types of reports received. The research team also examined the experiences of health care organizations and health care professionals who received and used consumer reports to improve patient safety in their institutions and identified remaining technical and content challenges that would need to be addressed prior to scaling up from the prototype. Finally, recognizing that perhaps a single hotline could not meet all of the identified goals (patient safety improvement, public monitoring and accountability, research on adverse events), the researchers identified a range of opportunities for the hotline (and its components) to be deployed in other settings and for other uses.
Challenges and Lessons Learned
This project highlighted several challenges inherent in involving consumers in reporting about safety concerns. Safety-related events do not surface in a predictable way, and their causation is complex. This makes standardized data collection approaches for health safety events difficult to design and implement. Further, the reporting process must be acceptable to the person making a report. Current systems for classification and coding of events may be useful to clinicians and patient safety managers, but they may not be easily adaptable to the unique perspective that consumers bring to health care safety improvement. In addition, the optimal timing for soliciting reports may vary, depending on a patient’s health status, experience, and expectations. A patient safety reporting system must also respect current legal and regulatory requirements regarding confidentiality and data protection that come into play when soliciting, storing, and transmitting safety-related information.
The project demonstrated that “we can build it,” that is, it is possible to put into place a high-quality patient reporting system with the help of willing partner health care organizations, but that does not mean that “they will come.” In addition, the outreach strategy used to bring consumers to the Web site did not produce the hoped-for volume of reports. Although the outreach strategy was refined and expanded, with help from the health care partners and guidance from our TEP throughout the 17 months of hotline operation, the volume of reporting remained suboptimal for testing the prototype.
A related issue was whether the prototype was perhaps trying to “serve too many masters.” The AHRQ-sponsored design report that gave rise to the development of the prototype recommended that patient reports be gathered locally but also communicated to a centralized (national) level to be aggregated, analyzed, and triaged or distributed to State and local levels for action. Based on our experience with the hotline, we now question whether improving patient safety at the local level, aggregating data for public monitoring and accountability, and conducting research on patient-reported errors are compatible objectives that can be addressed by a single consumer reporting system without undue burden on either the consumers or the health systems involved. Attempting to address all of these goals within one prototype resulted in an instrument and a process that were much more cumbersome than they needed to be to achieve any one objective (for example, patient safety improvement within a single health system), and this may have contributed to the lower-than-hoped-for response rates.
Given the many challenges revealed by the pilot study, the takeaway conclusions are summarized here:
- Patient safety reporting is both desirable and feasible. The hotline created a mechanism, accessed via a Web-based platform or a toll-free phone number, that patients and families used to report meaningful clinical information about perceived errors and physical and emotional injuries.
- The hotline yielded information that was previously unavailable. The hotline provided information that the sophisticated health care organizations would not have had despite the existence of mechanisms such as patient experience surveys and complaint departments and information about patient safety concerns from staff (e.g., adverse event reporting systems).
- The hotline was readily incorporated into existing patient safety systems and was not disruptive to health system operations. Although health system leadership initially worried that there would not be sufficient bandwidth to respond appropriately to all reported patient concerns, they ultimately tried to elicit more reports from patients. Further, the hotline was not disruptive; instead, its implementation reinforced organizational commitment to safety, quality, patient engagement, and transparency.
- Legal and regulatory obstacles to soliciting, storing, analyzing, and sharing event reports were manageable through the use of PSOs. These obstacles are likely to be even less problematic if hotlines are adopted by individual health care systems for their own use.
- The option of proxy reporting was an important component of the hotline. Proxies (such as family members and friends of patients) submitted about half of the total number of reports.
- The prototype has a number of components that can be utilized separately, depending on the specific goals of the end user. Which components are employed and how they are adapted would depend on the individual project’s purposes.
Given the many positive findings associated with the hotline, how should the discrepancy between its potential high value and the low response rate among patients and caregivers in the pilot project be addressed? It appears there were three principal problems. First, although a variety of marketing vehicles were employed (brochures, hospital Web sites, pamphlets included in discharge materials), it was assumed that, when given the opportunity and invitation, patients would reach out to institutions to report problems. However, this passive outreach strategy (simply making a Web site or a toll-free phone number available) did not generate a large volume of reports. Second, Web traffic data showed that more individuals accessed the Web site than actually made reports—many potential users dropped off after reading the introductory materials or completing only the narrative portion of the report. This suggests that, despite the time and care taken in developing the Web site, there also most likely were problems with the platform itself. Third, the presence of alternative mechanisms for reporting at the hospital level (e.g., discharge surveys) or the community level (e.g., State regulatory agencies) could have depressed the number of reports.
Conclusion
The design and development of the Health Care Safety Hotline represents a significant effort on the part of the research team and AHRQ to move promising research findings “from bench to bedside.” The effort yielded important information about the feasibility of consumer reporting of safety events and also some critical constraints. Overall, the number of consumer reports was disappointing, given the potential reservoir of adverse events and the yield of research-oriented methods. Nevertheless, the data the reports provided were rich compared with the data available from standard data collection methods.
The reports revealed events that were largely unknown to the well-informed health care organizations and prompted them to reach out to those patients and to directly address their concerns through patient safety improvement mechanisms. The hotline demonstrated that telephone and online reporting are feasible methods for collecting patient safety information and that consumers are largely willing to share this information with the organizations that delivered the care they found problematic. While the logistics of managing patient-reported safety information poses certain challenges to confidentiality, timeliness, and peer review, these potential barriers can be addressed in a way that advances patient engagement in patient safety. The hotline prototype developed in this project, with some modifications, is scalable to diverse settings and can be a valuable piece of the patient safety improvement puzzle for years to come.
On the basis of our experience in designing, developing, operating, and evaluating the Health Care Safety Hotline prototype, we make the following recommendations to AHRQ:
- Sponsor further development work on the prototype. While we are not at this time recommending a full scale-up of the prototype, we urge AHRQ to consider follow-on work to answer the primary question raised in this report: whether a “prompted” outreach strategy will succeed in generating a reasonable volume of reports. As a corollary, we also recommend a full usability study of the Web platform and content.
- Put all of the hotline materials, including the Operations Manual (which has all the forms, surveys, and Web site specifications) into the public domain and encourage their use. AHRQ has considerable influence on organizations involved in patient safety reporting, monitoring, and improvement. Encouraging health care organizations, regulatory agencies, and others to use the prototype (or components of it) would increase the likelihood that the tools developed here will be employed for a variety of research, patient safety improvement, and public reporting purposes.
This report offers further specifics about the project, including a detailed description of the design, development, and implementation of the prototype; a review of the patient reporting form, the data collection platform and content, and the protocols for data collection, data processing, and data sharing; the findings, challenges, and lessons learned during all stages of the project; and conclusions and recommendations for AHRQ’s next steps. The report also includes detailed appendixes, including the full Operations Manual, with copies of the event reporting forms, surveys, Web content, Web specifications, and instructions.
a The AHRQ Common Formats provides precise definitions of patient safety events, examples of patient safety reports, forms to guide development of data collection instruments, and a metadata registry. Information on the Common Formats is available at the Patient Safety Organization Privacy Protection Center (PSOPPC) at www.psoppc.org/web/patientsafety.
