The goal of the evaluation was to test the capability of the hotline prototype to collect meaningful information about patient safety concerns across a wide range of settings and to understand challenges in triaging and sharing that information with professionals who can improve the safety of health care and (as appropriate) with the public. Therefore, the evaluation effort had three principal aims:
- To characterize the hotline reports entered by patients and caregivers. This part of the evaluation effort was designed to analyze how resources related to the marketing and promotion of the hotline and the modes used to provide access to it affected the flow of its operations and the numbers and types of reports received. Once we realized that the volume of reports was going to be low, we also looked at web traffic metrics to try to identify problems in the event reporting form or the web design.
- To document and report on the experience of health care organizations and health care professionals who received and used the patient and caregiver reports to improve patient safety in their institutions. Lessons learned from the implementation of the prototype will help to identify needs for further refinement and circumstances under which implementation can be successful.
- To identify remaining technical and content challenges that would need to be addressed prior to scaling up the prototype.
Methods
Web Traffic Reports
During the implementation phase, the research team developed a monthly Web traffic report to inform the research team about the number of people who were visiting the Web site but not completing the patient event reporting form or not completing the entire form. Statistics reported included date, time, Internet protocol (IP) address, referring URL, and pages clicked.
Analysis of Patient-Reported Events
The research team abstracted and tabulated the following kinds of information from the hotline reports: the mode of reporting (Web or phone), whether the information was complete or incomplete, whether it was provided by the patient or a proxy, patient demographic information, the type(s) of safety concerns reported, whether the event involved a mistake and/or a negative effect, and contributing factors. The team also tabulated how frequently patients and caregivers agreed to receive a followup clarification call from the hotline team and how frequently the clarification call was completed.
The way the physicians on the team classified the patient safety events using the AHRQ Common Formats was also reviewed. As described in Chapter II, before and after additional information is elicited through the patient or caregiver clarification process, a physician reviews the event report and classifies the event(s) described within it by event type, harm scale, and duration of harm. (Refer to the classification form in the Operations Manual, Appendix C.)
Finally, we reviewed data on what the health care delivery organizations reported they did with the consumer reports they received (e.g., whether an RCA was conducted, what contributing factors were identified, whether a matching report was found in the staff adverse event reporting system), and what the organizations learned as a result. This information was collected as part of the provider supplementation process that occurred 45 days after a report was made. The analysis was performed to determine whether hotline-reported events provided information that was otherwise unavailable to the organizations and whether the information was a factor in subsequent actions or improvements.
Interviews with the Participating Health Care Delivery Organizations
To understand organizational context and readiness, the research team conducted site visits at the two health care delivery organizations that were implementing the prototype in June 2013—approximately 6 months before the hotline went live. We specifically sought to understand the features of the organizations that made the prototype project attractive to them, sources of potential concern or resistance, and how the organizations had prepared for hotline deployment.
During each site visit, members of the research team conducted 45- to 60-minute interviews with the organization’s hotline implementation team, communications leaders, quality and safety leaders, patient-relations leaders, legal staff, risk management staff (at one organization only), and Institutional Review Board leaders.
A semi-structured interview guide was developed (Appendix D) to elicit information on the organization’s reason for deciding to participate in the pilot, challenges encountered throughout the first year of participation, anticipated challenges moving forward, the value added by the hotline, and opportunities to increase the volume of reports submitted. A note-taker took notes during the interviews, except for two interviews that were digitally recorded and transcribed. We then used a variation of content analysis to summarize stakeholder comments on each interview topic, noting differences by type of stakeholder and by health care delivery organization.
In May 2015 (after the 15th month of hotline operation), the research team convened a webinar with both participating health care delivery organizations. The discussion included updates from the two organizations, a summary of reported events, a summary of provider supplementation data (Module 8 in the Operations Manual, Appendix C), a review of Web traffic data, and a discussion of lessons learned and possible next steps.
Feedback from Web Designers
We asked a Web design team at RAND that had not been involved in the design of the Web platform or content to review the prototype and provide expert opinions on access and technical and content challenges that may have affected the volume of reports received.
Feedback from the TEP
In June 2015, the hotline team convened the third—and final—in-person meeting of the TEP. The discussion at the meeting included a recap of modifications made to the hotline since the last TEP meeting, results of the Web traffic and patient event reporting analysis to date, a summary of input from the health care organizations, and a discussion of lessons learned and next steps.
Evaluation Results
Aim No. 1: Characterize Hotline Reports from Patients and Caregivers
Web Traffic
Between October 1, 2014 (when the research team began tracking Web metrics) and June 30, 2015, the hotline home page received a monthly average of 170 visitors with a unique IP address (Table 4.1). The majority of the visitors (70 percent) accessed the hotline home page through Google or another search engine. The remaining 30 percent accessed the hotline home page from the Web site of one of the two health care delivery organizations.
The most frequently visited hotline Web page (after the home page) was the FAQ page, which had an average of 72 visitors per month during this period.
Table 4.1. Average Number of Visitors Per Month to the Hotline via the Web
| Page | Average Number of Visitors per Month |
|---|---|
| Home | 170 |
| FAQs | 72 |
| First page of reporting form | 45 |
| Second page of reporting form | 44 |
| Third page of reporting form | 12 |
The first page of the patient event reporting form (which consumers accessed by selecting “Click here” on the home page) received an average of 45 visitors per month (25 percent of the visitors to the home page). If each of these 45 visitors had completed the event reporting form, the hotline would have received 516 reports via the Web per year. However, only 20 individuals over a 17-month period ultimately used the hotline Web site to complete an event report; an additional 17 individuals used the toll-free phone number to complete a report.
The first page of the event reporting form requires potential reporters to indicate whether they are 18 years of age or older. Seven visitors indicated that they were under 18 years of age and so were not allowed to complete a report. The initial Web page also offers the option of creating a password so that the reporter may finish the report at a later date. A total of seven visitors created passwords.
The second page of the reporting form provides information about the hotline, including assurance that the reported information will not be shared with anyone outside of the research team unless the patient or caregiver gives permission. The page requires potential reporters to indicate that they accept a series of statements, (e.g., that they have read the background information on the Web page, and that they will provide accurate information). Only one visitor declined the request and was not allowed to complete a report.
The third page of the reporting form asks the first set of questions about the patient or caregiver’s safety concern. An average of 12 visitors per month visited this page. If each of these visitors had completed the reporting form, the hotline would have received 144 reports via the Web each year.
As noted above, the largest drop-off was from the home page to the first page of the reporting form. On average, only 25 percent of the 170 monthly visitors to the home page continued to the first page of the reporting form.
Patient Event Reports
During the 17 months of operation from February 2014 through June 2015, only 37 reports of safety occurrences (RSOs) were submitted to the hotline. This is an average of 2.3 completed reports per month (Figure 4.1). Of the 37 completed reports, 20 were submitted via the Web, and 17 were submitted by phone. All reports were made in English.
For comparison, spontaneous reporting from hospital-based adverse event reporting systems for health care professionals yields about one report per 10,000 admissions per year. Research studies using patient surveys or chart reviews produce roughly five to ten reports per 100 admissions; sometimes as many as 25 reports per 100 admissions. We had hoped for at least one report per 10,000 admissions.
Figure 4.1. Safety Concerns Reported by Month
Seventeen additional reports were started on the hotline Web site but were not completed. Of the 17 incomplete reports, 13 contained no information after Module 1 (Introduction), and 3 contained no information after Module 2 (Description of your safety concern).
Of the 37 completed reports, 14 were from patients, and 23 were from caregivers (11 by a parent and 12 by a spouse, domestic partner, or other family member). Four people entered multiple reports, accounting for a total of 12 reports.
Demographics
Of the 37 completed reports, 25 pertained to female patients, 11 pertained to male patients, and one did not indicate the patient’s sex. The majority of reports (n=19) involved adult patients 18 to 64 years of age. The majority of patients (n=22) identified as white, four identified as black or African American, and one identified as Hispanic, Latino, or Spanish origin. Fifteen reports involved patients with private insurance through an employer, three involved patients with Medicare, and three involved patients with Medicaid.
Reported Patient Safety Concerns
The reports addressed a wide range of safety concerns,l as shown in Table 4.2.
Table 4.2. Safety Concerns Reported via the Hotline
| Type of Safety Concerns | Number of Reports |
|---|---|
| Communication | 9 |
| Environmental | 6 |
| Care coordination | 6 |
| Process-related* | 6 |
| Documentation | 4 |
| Treatment-related | 4 |
| Discharge-related | 3 |
| Medication errors | 1 |
| Failure to diagnose | 1 |
| Infection | 1 |
| Physical assault | 1 |
Note: The number of concerns (42) does not equal the number of reports (37) because some reporters indicated more than one concern.
* Process-related events include, for example, staff not washing their hands before examining a patient or failing to remove an IV during discharge.
In 20 of the 37 completed reports, at least one medical mistakem was reported. The mistakes involved prescription drugs (n=1); tests, procedures, or surgery (n=1); pregnancy or childbirth (n=1); diagnosis or advice from a doctor, nurse, or other health care provider (n=5); poor cleanliness or poor hygiene (n=2); and something else or more than one mistake (n=10).
Fifteen of the 20 safety concerns involved both mistakes and negative effects.n Seven reports involved negative effects only (for a total of 22 reports of negative effects.) Three categories of negative effects were reported: only physical (n=3), only emotional (n=5), and both physical and emotional (n=14). In addition to these negative effects associated with a mistake, patients and caregivers categorized six safety concerns as having a negative effect that did not involve a mistake. Seven of the safety concerns involved negative effects only.
When patients or caregivers were asked to identify where the mistake or negative effect occurred, 18 were reported to have occurred in the hospital, seven in the emergency department, and three in a doctor’s office or clinic.
Patients and caregivers were also asked to identify the individual they told about the mistake or negative effect. Most shared it with a doctor, nurse, or other health care provider (n=23), a family member or friend (n=17), a health care administrator or manager (n=15), a lawyer (n=5), or someone else (n=4), such as a licensing agency.
Three respondents said that as a result of the event, the patient switched to a different health care provider; three said the patient transferred to a different hospital; four said they did something else; and 17 said there was no change in the care provider.
The list of contributing factors was organized into six categories for the patient event reporting form: communication with doctors, nurses, or other health care providers; responsiveness of staff; coordination of care; access; verification; and other. The most commonly identified contributing factors were communication (n=21) and care coordination issues (n=19), followed by access (n=14) and responsiveness of staff (n=13).
Patients/caregivers had the opportunity to select more than one cause within a contributing factor. Table 4.3 shows the most commonly identified causes within a contributing-factor category. Seventeen patients and caregivers reported that staff did not listen to the patient, 17 reported that staff ignored what the patient told them, and 12 reported that staff did not spend enough time with the patient. Twelve patients and caregivers reported that doctors, nurses, or other health care providers did not seem to work well together as a team, and 11 reported that they lacked followup.
Processing of the Patient and Caregiver Reports
In 34 of the 37 reports, the respondent agreed to be contacted for a followup call if the research team wished to ask any clarifying questions. Twenty-two individuals completed a clarification call; the other reports did not require clarification.
Thirty-four of the patient or caregiver respondents consented to have the report shared with a named health care organization or clinician. Fourteen of the 34 shared reports were completed by the patient who experienced the concern (rather than a caregiver); 13 of these patients agreed to include their names and contact information with the shared report. Twenty-three of the 37 shared reports were completed by a caregiver; 18 of these caregivers agreed to share their names and contact information with the shared report.
Table 4.3. Contributing Factors Identified by Patients and Caregivers, Overall and by Type
| Area of Concern | Contributing Factor | Number of Reports |
|---|---|---|
| Communication | Total reports | 21 |
| Doctors, nurses, or other health care providers did not listen to the patient. | 17 | |
| Doctors, nurses, or other health care providers ignored what the patient told them. | 17 | |
| Doctors, nurses, or other health care providers did not spend enough time with the patient. | 12 | |
| Doctors, nurses, or other health care providers did not provide a clear explanation of the diagnosis or care plan. | 6 | |
| Doctors, nurses, or other health care providers did not explain follow-up care instructions. | 3 | |
| Doctors, nurses, or other health care providers did not explain things to the patient in the patient’s language. | 2 | |
| Doctors, nurses, or other health care providers used terminology the patient could not understand. | 2 | |
| Doctors, nurses, or other health care providers spoke with an accent that was hard to understand. | 2 | |
| Coordination | Total reports | 19 |
| Doctors, nurses, or other health care providers did not seem to work well together as a team. | 12 | |
| There was a lack of followup by the doctors, nurses, or other health care providers. | 11 | |
| Doctors, nurses, or other health care providers were not aware of care that took place someplace else. | 2 | |
| Access | Total reports | 14 |
| The patient did not get help or advice he or she needed. | 10 | |
| The patient was not able to get the tests or treatments that he or she believed necessary | 6 | |
| The patient was not able to get the tests or treatments that a provider believed necessary. | 2 | |
| Responsiveness | Total reports | 13 |
| Was it because of not getting help as soon as the patient needed it? | 11 | |
| Was it because of not getting care as soon as the patient needed it? | 9 | |
| Verification | Total reports | 7 |
| Was it because someone did not have the most recent and up-to-date information about the patient? | 4 | |
| Was it because someone did not correctly identify the patient? | 3 |
Classification of the Reports
A physician on the research team reviewed each completed report and classified the event(s) according to the AHRQ Common Formats classification scheme for event type, harm, and duration of harm.o Of the 37 reported events, 23 were classified as “incidents,” that is, patient safety events that reached a patient.p Some incidents result in patient harm, and some do not.q
As shown in Table 4.4, about half of the reported incidents (n=12) resulted in mild harm,r about a quarter (n=6) resulted in no harm, and the other quarter (n=5) resulted in moderate harm,s as classified by the physician reviewers.t None of the incidents resulted in severe harmu or death. Table 4.4 also shows the duration of the harm associated with these reported events.
Only one of the 37 reported events was classified as a near miss—that is, an event that did not reach a patient. Eight of the events were classified as an unsafe condition.v The remaining four reported events were service complaints (e.g., concerns about non-clinical aspects of care such as food, parking, or long wait times in the doctor’s office).
For 28 of the reported events, a clarification call with the patient or caregiver was successfully completed; however, no changes were made to the classification of the type of event, harm, or duration of harm. While the clarification calls provided useful information about the context of the patient safety concern and the sequence of events, they offered no additional value for classifying reported events.
Provider Actions in Response to the Hotline Reports (Module 8)
As described earlier, if an individual consented to having the report shared with a named health care organization, the research team contacted the health care organization within 45 days of the report to ask a series of questions about what the organization did with the information provided in the report. As of June 30, 2015, this step had been completed for 32 reports.
Table 4.4. Physician Classification of Reported Patient Safety Events
| Event Type and Harm | Number of Reports | Duration of Harm |
|---|---|---|
| No harm | 6 | N/A (no harm) |
| Mild harm | 12 | <1 year: 8 ≥1 year: 0 Unknown: 4 |
| Moderate harm | 5 | <1 year: 3 ≥1 year: 1 Unknown: 1 |
| Severe harm | 0 | No reported events |
| Death | 0 | No reported events |
| Near miss | 1 | N/A (no harm) |
| Unsafe condition | 8 | N/A (no harm) |
| Service complaint | 4 | N/A (no harm) |
| Unclassified | 1 | |
| Total | 37 |
No match was found for 26 of the 32 patient event reports. The health care organizations were able to identify only six patient-reported events in their internal databases (referred to here as “able to match”). When they did find a match, however, they were able to identify the patient and the incident with a high degree of confidence (Table 4.5).
Table 4.5. Ability to Match Hotline Report with Internal Information
| Was the health care provider or facility able to match the patient safety concern? | Number of Reports |
|---|---|
| Matched with the provider. | 1 |
| Matched with the patient. | 6 |
| The mistake or negative effect was matched with a high degree of confidence. | 5 |
In addition to trying to identify whether a patient safety event was already known, the organizations undertook a variety of actions in response to hotline reports (Table 4.6). In half of the cases, the health care organization followed up with the patient involved.
Table 4.6. Actions Taken as a Result of a Hotline Report
| Action Taken | Number of Reports |
|---|---|
| Department quality review, such as peer review, morbidity and mortality conference, or “tumor board.” | 21 |
| Contact made with the patient involved. | 16 |
| Contact made with the provider involved. | 5 |
| Institutional level review. | 5 |
Note: Other possible actions were RCA, reporting to PSOs, and reporting to national or State regulatory agencies.
The primary contacts at the health care organizations were also asked to identify what they felt were contributing factors to the events reported through the hotline. The most commonly identified contributing factors were a lack of team coordination and environmental factors (Table 4.7). Organizations also had the opportunity to select more than one issue within a contributing factor. The most common contributing factor identified by the health care organizations was a lack of staff-to-patient communication (n=12) and a lack of communication among staff or team members (n=9). Other contributing factors included the behavior of individual staff (e.g., adherence to policy, protocols, and orders) and deficits in the environment (e.g., lack of equipment).
In six of the reports, staff identified specific quality improvement opportunities for health care organizations, including the need for better communication between staff and patients and their family members, the need for a discharge checklist, the need for a handoff communication tool, and the need to monitor low-volume, high-risk complications associated with certain medical conditions. One event associated with a low-volume/high-risk complication resulted in an increase in staff education concerning the medical condition, and it may result in a performance improvement project including simulation training and creation of a rapid-response protocol for this type of condition.
Table 4.7. Contributing Factors Identified by Health Care Organization, Overall and by Type
| Category | Contributing Factor | Number of Reports |
|---|---|---|
| Team coordination | Overall | 16 |
| Communication: staff to patient | 12 | |
| Communication: among staff or team members | 9 | |
| Communication: supervisor to staff | 3 | |
| Clinical supervision | 1 | |
| Heavy workload | 1 | |
| Operating environment | Overall | 6 |
| Equipment/device availability | 2 | |
| Housekeeping | 2 | |
| Physical surroundings (e.g., lighting, noise) | 1 | |
| Workflow/task | Overall | 4 |
| Bed capacity | 1 | |
| Data availability | 1 | |
| Data accuracy | 1 | |
| Management of test results | 1 | |
| Staff/individual | Overall | 3 |
| Adherence to policy, protocols, or orders | 3 | |
| Patient/resident | Overall | 2 |
| Agitated/aggressive | 1 | |
| Impaired hearing or speech | 1 | |
| Management/organization | Overall | 1 |
| Culture of safety management | 1 | |
| Other | Other | 2 |
Usability Surveys
Four of the 37 hotline reports were accompanied by a completed post-submission survey about the usability of the Web site—a sample that is too small for the data to be meaningful. Three individuals accessed the first page of the post-submission survey but did not complete it.
Aim No. 2: The Experience of the Participating Health Care Delivery Organizations
We sought to understand the features of health care organizations that made the hotline an attractive proposition—and by contrast, the sources of potential concern or resistance—and how our partners prepared for and responded to hotline deployment.
The Decision to Participate
At one of the two participating organizations, several departments, interdepartmental committees, and leaders (including the board and management council) provided input to the decision to participate. A leader in the safety department learned about the project, vetted the idea of participation with the vice president of safety, and then discussed it with multiple stakeholders within the health care system. Input was also solicited from communications/outreach, risk management, clinical chairs, and patient relations. The chief executive officer made the final decision. The safety department leader described her role in the organization’s decision to participate as “key influencer.”
At the other organization, the vice president of quality and safety was primarily responsible for the decision to participate. This executive shared information about the project with stakeholders but was ultimately the decisionmaker. In both organizations, the individual who learned about the opportunity and brought it to the attention of the rest of the organization went on to lead the implementation team.
When describing their organization’s decision to participate, stakeholders from both health systems emphasized the importance of the alignment of the goals of the hotline with the organization’s mission, priorities, and current practices. One stakeholder explained, “This particular decision was easy because we’re doing this type of thing already.” Another interviewee noted, “[There were] a lot of linkages to what we’re currently doing in our safety program.” Stakeholders also identified the following reasons for participating in the pilot project:
- Linking to patient-centered care and preventable-harm reduction strategies.
- Promoting patient and family engagement as a method of building safe health care.
- Improving patient and family experience.
- Validating that the organization provides high-quality, safe care.
- Identifying any blind spots.
- Communicating the organization’s commitment to transparency.
- Advancing the use of Web-based technology as an alternative method of communicating with patients and families.
- Providing an additional avenue for educating the community about patient safety.
When asked if there was anything unique about the organization that influenced the implementation and integration of the hotline into the workflow, one stakeholder noted that the organization was created recently, stating, “We’re unusual in that we’re relatively young so there aren’t a lot of bureaucratic pitfalls. If it’s rational and reasonable, it’s easy to do.”
Although both organizations ultimately decided to participate in the pilot project, they identified several risks associated with participation. The two main risks raised by quality and safety leaders were that the project might “open the floodgates,” and that the organizations might not have sufficient staff to manage events effectively and respond quickly. A member of the legal team asked, “How do we know that we’re going to get the patient’s name? How do we know we could get back to the patient? What would our process look like in terms of timeliness of referrals?” Leaders of the communications and external affairs teams, in comparison, were most concerned about reputational risk and protection of the brand. These teams noted that individuals who submit a report through the hotline might experience survey fatigue and be less likely to respond to the organization’s patient experience survey. A member of the implementation team also noted the risk that the Web site might not work properly when launched.
Deciding How to Process Patient Event Reports from the Hotline
Both organizations had an existing mechanism for addressing safety concerns determined by the organizations to be grievances. Reports of grievances are processed by the patient- and family-relations department, which follows up promptly with individuals who provide their names and contact information. Both organizations decided to treat reports received through the hotline in the same way they treat grievances.
Challenges Encountered Prior to Launch
Stakeholders at both organizations indicated that one challenge prior to the launch was the amount of staff effort required for participating in meetings with the research team and the other delivery organization, reviewing and revising outreach materials, and ensuring that the organization was ready to process any reports received from the hotline. The implementation team at one organization felt that this was a significant challenge: “[There were] a lot of meetings, a lot of calls with all of you, integrated calls with [organization name], marketing calls, a lot of communication that had to take place, a lot of coordination and facilitation. It took our administrative assistant and me a lot of time.” In comparison, the implementation team at the other organization took steps to limit the draw on staff time, “I was … worried about people getting over-involved. Let’s blend it into our normal processes.”
Challenges Encountered After Launch
The primary challenge identified by the stakeholders at both organizations was the much-lower-than-expected volume of reports. Stakeholders wondered if this signified limited awareness of the hotline, a low incidence of safety events, or reluctance of patients and caregivers to report, perhaps due to features of the prototype (the Web interface design, introductory materials on the Web site, difficulty getting through to the toll-free number). Several stakeholders, especially those on the outreach and marketing teams, suspected that limited awareness might be the primary contributor and wondered what more could be done to engage patients and families. By contrast, staff at ambulatory offices at one of the health care organizations pushed back against circulating outreach materials, asserting that “there is too much stuff that doctors are trying to hand out: discharge papers, public health information, and the business cards about the hotline.” Others were concerned about a lack of direct feedback to either the patient or the facility when hotline reports did not include the patient’s name.
Anticipated Challenges to Scale-Up
Stakeholders at both organizations identified two sets of challenges that will need to be addressed in any effort to scale up the hotline: the need to encourage more patients and families to report their patient safety concerns (to make the amount of effort involved in implementing the hotline worthwhile), and the difficulty health care organizations may have in allocating an appropriate level of resources to manage the hotline on an ongoing basis. Specific comments made with regard to engaging more consumers include the following:
- The hotline is reaching people at the time of inpatient admission to a hospital or at the time of service in an ambulatory clinic. This may not be the time when a mistake has happened (or a patient realizes that a mistake has happened). The current outreach method may not be reaching the patient or caregiver at the right moment.
- There is information overload and competition for patient reporting. The hotline is competing against the statewide hotline for safety concerns, Health Grades, the Joint Commission, and other efforts.
- There is currently no “buy-in” from clinical staff—in fact, few were aware of the hotline. One stakeholder added, “In general, the hotline is a huge positive, but the natural reaction among clinical staff is not to take reports from patients seriously.”
Comments made with regard to allocating sufficient resources to manage the hotline include:
- The staff effort required to prepare for launching the hotline was unexpectedly large.
- The [prototype’s] followup processes and procedures are currently required for all hotline reports, regardless of an event’s severity. A tiered approach might be more appropriate; that is, resources invested should be aligned with the severity of the event. For example, a sentinel event might be followed by a concerted effort, whereas fewer resources would be allocated to identifying contributing factors in a report about the lack of a urinal at the bedside.
- Initially, leaders at one organization were worried that they would not be able to keep up with the volume of reports; later, they worried about whether the low volume of reports made the investment worthwhile.
Value of the Hotline
Participants had mixed views about a business case for the hotline, given the low volume of reports received in the pilot project. However, one member of the implementation team endorsed the hotline reports as more valuable than reports made to the State or another entity. The same person noted that the comprehensive nature of hotline reports might speed up the process of learning in some hospitals because the reports provide a better understanding of what a patient has experienced and what additional information might be needed from a patient before reaching out to him or her. Finally, this person stated that the organization is currently engaged in a “deep analysis” of an issue identified in one of the reports it received from the hotline. A member of the implementation team from the other health system stated that the hotline reports provide “appropriate, detailed enough information so that [the organization] knows where to send the report and process it through patient relations.” This individual also characterized the reports as “extremely complete.”
Lessons Learned
The research team and the two participating health care delivery organizations held an evaluation webinar in May 2015 for a final discussion of lessons learned. Representatives of one of the organizations identified aspects of the hotline that were “useful” and “less useful.” The most useful aspect from the point of view of one implementation team leader was the elicitation of patient concerns of which the organization was not previously aware; the staff had prior knowledge of only one or two of the events that were reported through the hotline.
A less useful aspect was that the involvement of an intermediary (RAND) led to a delay of 72 hours in receiving and being able to address patient concerns, when the organization standard was to reach out to a patient/family within 24 hours of a reported grievance. While the intermediary enabled several different options for reporting (e.g., anonymous, de-identified), the back-end review and processing that led up to sharing a report delayed the organization’s receipt of the concern. Another participant noted that the low volume of reports raised the question of whether the hotline added value but said that the reports did contain such “granular information” that they were useful.
Aim No. 3: Identify Technical and Content Challenges to Address Prior to Scale-Up
We asked RAND Web designers to look at the prototype and make suggestions about changes that should be made prior to scale-up. They identified two problems: (1) the content of the Web site may have been insufficient to make patients and caregivers see the benefit of using the hotline rather than other available options; and (2) technical problems caused the Web-based version of the hotline to be somewhat difficult to use. We begin with the suggestions concerning content.
Report a Safety Concern
The Web team pointed out that because filling out and submitting the form are the primary actions we want users to perform when they go to the hotline Web site, the link/button to submit the form should be very prominent on the page. Therefore, all unnecessary barriers to completing this action should be removed.
Hotline users will have been exposed to outreach materials before they come to the Web site and will therefore have a basic understanding of its purpose, so most other content, although important, should be considered secondary to getting users to report a legitimate patient safety concern.
What’s in it for me?
A key challenge for Web site producers and content developers is keeping the visitor in mind and remembering that his or her attention span is often quite short. When visitors access the home page of the hotline Web site, they should be presented with key content and a call to action as quickly as possible.
Unfortunately, the home page of the hotline Web site is heavy on text but light on explaining the benefits of submitting a report. The one FAQ that discusses the benefit of the hotline is buried and does not offer a particularly strong call to action:
Why should I participate? You can share your experiences and help make health care safer for people in your community. We need to hear from many people. We need to hear about many health care experiences and concerns.
To encourage use, this information should be front and center on the home page, with a stronger call to action. If the call is strong enough, a Web site can convince people to perform, even if incentives are nonexistent. When a Web site requires more time to use than an alternative option, it is especially important for the call to action to be strong.
There are established sites that already collect patient safety complaints, notably www.jointcommission.org, but these sites allow users to input large blocks of text that must then be read and coded, whereas the hotline uses structured form fields that can be processed more easily and accurately. Therefore, a key element of the call to action for the hotline Web site should include the benefits of collecting structured data. The site could also emphasize the benefits of using a form that asks specific questions, rather than only requesting open-ended feedback. A patient or caregiver using an open-ended form may forget to mention or not realize the importance of some aspects of the complaint; the hotline form ensures that all key issues are addressed, even if the user chooses not to answer specific questions.
How long will this take?
Rather than warning users that the form will take 20 to 25 minutes to complete, mention that the time required to write up a detailed narrative can far exceed 20 minutes. In other words, the site can save users time when submitting a concern.
What happens after I use the form?
If the prototype is to be scaled up, the Web designers strongly recommend that it be rebranded as something other than a hotline. Many people think of hotlines as a way of getting a quick response to a problem or as a forum for asking questions about a product or service; by contrast, the FAQs point to submissions as being suggestions for the health system to consider.
The home page call to action should more strongly address the question of what will be done with the user’s submission. This may require working with the participating health systems to offer a guarantee that they will respond to submissions that are not anonymous (not just that they “may call you”). If possible, users who submit a concern anonymously should be provided with a submission ID number so they can check on what has happened since the complaint was reported. Finally, if it is feasible to share concerns with other groups outside of the health care institution, this could be stated to inform users that their use of the hotline Web site and the concerns they submit are not simply being logged as part of a research effort.
Why do I need to create an account?
The need to create an account may be considered burdensome by users of this site because they may not need to visit it frequently, and the account creation process is often clunky from a user’s perspective. However, if enabling users to save their work and return to the site later is seen as valuable, it may be better to work this functionality into the submission process, rather than placing it on the first page. One option is to add a “Save for later” (or “Save and exit”) button. Different from the “Save and continue” option, this button would prompt the account creation process and would be selected only when a user realizes he or she will not be able to complete the form in one sitting. This might remove one of the initial barriers to completing the form.
The designers also tested the Web site using Chrome and Chrome Vox (a screen reader) and made technical suggestions concerning general bugs, usability and accessibility issues, and the need to make sure that the Web page responds to the available space in the browser window (so-called responsive Web design) to make it easier for users to access on mobile devices. The technical suggestions are included in the Operations Manual (Appendix C).
Summary of TEP Input
After reviewing the findings from the pilot project, several of the TEP members noted the value the hotline had offered to the health care delivery organizations that participated, particularly the provision of detailed, actionable information on safety concerns about which the organizations had not previously been aware.
Despite the fact that the hotline had provided useful information to the participating organizations, and as good as the prototype was, several TEP members noted that the hotline was probably capturing only a small percentage of the safety concerns occurring at these organizations. Continuing barriers to reporting include fear of retaliation, concern that reporting will not result in change, the desire to avoid “getting anyone in trouble,” and patients not wanting to revisit a difficult or traumatic experience.
Some members noted the difficulty of obtaining high response rates and advocated more active recruitment methods. Others thought a better approach would be for AHRQ to advocate that questions about safety concerns be added to the CAHPS survey (an option that the research team and partner organizations had explored but found infeasible, given time limitations and a lack of enthusiasm among the vendors).
Given the research team’s finding that most of the “story” of the safety concern was described in response to the form’s first few open-ended questions, one TEP member suggested that a future project could implement a dramatically shortened form—perhaps one using only three questions. Since completing such a form would be less time-consuming than completing the current form, the questions could potentially be asked of each patient before discharge. Benefits of this approach include employing a more prompted recruitment strategy and capturing safety concerns in nearly real time.
One TEP member suggested that the current hotline effort had too many objectives, which muddled the messaging to patients and caregivers. Different opinions were offered about whether a consumer-oriented reporting system should ideally provide publicly available hospital-level data, publicly available aggregated data, or no public data (instead providing data only to health care delivery organizations to use for internal quality improvement). The group discussed successful examples of each type of effort.
Finally, several TEP members emphasized the value of early and ongoing patient input, such as the information we received through PFACs.
Summary
Building a consumer-oriented patient safety reporting system using a Web portal and a toll-free phone number is a feasible approach. However, a passive strategy for bringing consumers to the hotline did not produce a high volume of reports. Several explanations for this are possible. For example, the outreach and marketing strategy may have been insufficient to draw consumers to the hotline, or they may have had difficulty navigating the system once they found it. More research is needed to understand how best to facilitate consumer reporting.
l The term “safety concern” is used throughout the reporting form because it tested well in focus groups and cognitive interviews during the development phase of the project. Safety concerns include both medical mistakes and negative effects. Safety concerns might arise during a visit to a doctor’s office, at a pharmacy, or in the hospital.
m Medical mistake” is defined as something that was done (or not done) by a health care provider that would be considered incorrect at the time that it happened. Medical mistakes can result in harm or injury to the patient, but not necessarily in every case.
n The term “negative effect” tested well in focus groups and cognitive testing, in contrast to “adverse event,” which many patients and caregivers did not understand. Negative effects can be physical or emotional, and they may include infections, drug reactions, or other complications.
o The IOM has also created a classification system for safety events. The system distinguishes between adverse events that are preventable, adverse events that are not preventable, and near misses.17
p “The AHRQ Common Formats characterize “reaching a patient” as “any action by a health care practitioner or worker or health care circumstance that exposes a patient to harm.”
q An “adverse event” in the IOM classification system is equivalent to an “incident” in the AHRQ Common Formats. A “near miss” that reaches a patient but does not cause harm is equivalent to an incident with no harm, and a “near miss” that does not reach a patient is equivalent to a “near miss.”
r The AHRQ Common Formats define “mild harm” as “minimal symptoms or loss of function, or injury limited to additional treatment, monitoring, and/or increased length of stay.”
s The AHRQ Common Formats define “moderate harm” as “bodily or psychological injury adversely affecting functional ability or quality of life, but not at the level of severe harm.”
t When a reported event described multiple harms, the physician classified the event according to the most serious harm.
u The AHRQ Common Formats define “severe harm” as “bodily or psychological injury (including pain or disfigurement) that interferes significantly with functional ability or quality of life.”
v The AHRQ Common Formats define an “unsafe” condition as “any circumstance that increases the probability of a patient safety event; includes a defective or deficient input to or environment of a care process that increases the risk of an unsafe act, care process failure or error, or patient safety event.”
