Demonstration Grants Final Evaluation Report

The seven demonstration projects focused on three broad approaches to improving patient safety and reducing medical liability: (1) improving communication, (2) preventing harm through the use of “best practices,” and (3) exploring alternative methods of settling claims.

Improving Communication

Four demonstration grantees—New York State Unified Court System, University of Illinois Medical Center at Chicago, University of Texas, and University of Washington—provided training on error disclosure and implemented disclosure and resolution programs (DRPs) loosely based on a model developed at the University of Michigan Health System.² Under DRPs, health care professionals and institutions disclose adverse outcomes to patients and families; investigate and explain what happened; use that knowledge to improve patient safety and prevent the recurrence of such incidents; and, when appropriate, apologize and offer fair financial compensation. The goal of a DRP is to help physicians, risk managers, and other staff communicate with patients to acknowledge medical errors and to make early offers of compensation to avoid costly litigation.

New York State Unified Court System

This demonstration project was implemented through a partnership between the New York State Department of Public Health (the State regulatory agency responsible for patient safety) and the New York State Unified Court System (the court system operating in the five boroughs of New York City). The grantees engaged five of the city’s largest teaching hospitals—which provide services to some of the most economically disadvantaged and medically underserved populations in the city—in two different interventions: a judge-directed negotiation program (to be described later in this report) and a disclosure and resolution program. Through the disclosure and resolution program, which the grantee called communication and resolution, this grant aimed to enhance the patient safety culture in participating hospitals, prevent and address adverse events affecting general surgery patients, prevent the filing of malpractice claims through disclosure and early offer, and, over time, lower malpractice costs and premiums.

Overall, the hospitals implemented the disclosure part of the model with some success but experienced greater difficulty in implementing the apology and compensation component. Among the reasons given for the lack of full implementation of the communication and resolution program model were concerns that New York law does not shield apologies from being admissible in a lawsuit, that hospital negotiation with unrepresented patients and families might be seen as coercive, and lingering fear that any discussion of compensation might encourage rather than discourage the filing of claims.

The grantee reported the following findings about the first 125 adverse events resolved through the communication and resolution program at the five participating hospitals:³

• An initial disclosure of an adverse event to the patient or family member was documented in 79 percent of cases (the remaining cases were discussed with the patient or family but were typically considered known complications rather than adverse events). Explanation of the reason for harm was provided in 88 percent of cases, and expressions of sympathy (without an acknowledgement of responsibility) were made in 64 percent of cases. An apology acknowledging responsibility was given in only 13 percent of cases.
• A quarter (25%) of the 125 events were reported to the Risk Management or Quality Departments within 24 hours, half (50%) within 6 days, and three-quarters (75%) within 18 days, with only a short time elapsing between the first communication with patients and families about the event and the last communication (50% were concluded within 2 days). Overall, 75 percent of communications were concluded within 32 days.
• Of the 125 events, 30 were judged by communication and resolution program decisionmakers to involve all of the essential elements of a negligence claim: an injury, causation, and substandard care. For these 30, the responses deemed appropriate by communication and resolution program decisionmakers included compensation (9) and an offer to waive medical bills (12).
• A small proportion of communication and resolution program cases (20 of 125, or 16%) progressed to a claim or lawsuit. Twelve to 15 months after communication and resolution program completion, 14 of the 125 cases (11.2%) resulted in a filed lawsuit and six (4.8%) others resulted in a claim but no filed lawsuit. Three of these cases (2.4%) were settled with a release of liability. Eleven of the 20 cases with a claim or lawsuit involved substandard care and causation, and the hospital offered compensation in 3 of these 11 cases.

Evaluation Observations

The New York demonstration project was very ambitious, involving five hospitals undertaking a number of patient safety initiatives within the hospitals along with the implementation of the disclosure and resolution program. In fact, some participants worried that with so many related activities going on in the hospitals at the same time (both within and outside of the grant, including the Federal Partnership for Patients), it might prove difficult to attribute any change to the grant program. Participants were also concerned about the added burdens, such as project-related data collection, to already overworked risk management staff.

The project’s reported findings are both illustrative and suggestive regarding the kinds of cases that were selected for the communication and resolution program, the handling and resolution of those cases, and the experience of the participating hospitals. However, these findings are descriptive rather than evaluative, and preliminary, representing only the first 125 adverse events referred into the communication and resolution program. It is too small a number to evaluate the impact of the communication and resolution program for the individual hospitals and, the grantee has not used statistical methods to assess and test the significance of outcomes. In sum, we would describe the findings as interesting and promising, but further research would be needed to rigorously investigate the magnitude of the effects generated by these communication and resolution programs on malpractice outcome measures and to determine the statistical significance of the effects. This view matches the concerns of grant participants that the timeline was much too short to enable the project to adequately track the impact of the intervention, especially on malpractice outcomes.

As articulated by one site visit participant, “changing the way hospitals function and the way clinicians think is like turning an iceberg around. It is slow, persistent, takes continued engagement of leadership. How we keep that focus is a challenge.” At least as far as the implementation of disclosure and resolution in these hospitals is concerned, these data suggest some culture change.

University of Illinois Medical Center at Chicago (UIC)

This demonstration project had three principal goals: (1) to continue to refine the Seven Pillars intervention at the University of Illinois academic medical center (UIC), (2) to continue collecting and analyzing data on the impact of Seven Pillars at UIC, and (3) to replicate the Seven Pillars intervention components at 10 diverse community hospitals in the greater Chicago area. Replication of the model is important because disclosure and resolution models have heretofore been developed and tested at self-insured academic medical centers, where the liability interests of the institution and the physicians are aligned. The outstanding research question is whether such a model will also work at community hospitals, where the hospitals, physicians, and their respective insurers often have divergent interests with respect to malpractice claims.

Seven Pillars, developed in 2006 at the UIC Medical Center (now part of the University of Illinois Hospital and Health Sciences System), is a comprehensive process for responding to patient harm events. The objective of Seven Pillars is to improve patient safety and mitigate medical liability risk through improved communication with patients and families, disclosure and early offer when patients suffer preventable harm, and learning from medical errors. The intervention includes seven key components: (1) patient safety incident reporting; (2) investigation and root cause analysis; (3) communication and disclosure; (4) apology and remediation; (5) patient safety and systems improvement; (6) system process and performance improvement; and (7) education and training, including “care for the caregiver” training. The 10 community hospitals were eager to be trained in Seven Pillars, as they had been exposed to the principles of the model via publications and prior dissemination efforts.

Because the grantee began implementing Seven Pillars at UIC in 2006, years before the demonstration project, longitudinal data are available from this project to examine the impact of Seven Pillars across multiple outcomes. The following findings were reported by the grantee on the impact of Seven Pillars at UIC:⁴

• A comparison of means of each outcome measure from the pre- to post-intervention period (5 years before the intervention and 7 years after) revealed significant increases in the mean quarterly number of adverse event reports, patient communication consults, and root-cause analysis reviews, as well as decreases in the mean quarterly number of malpractice claims, legal fees and expenses, settlement amounts, and total liability costs.
• Similar positive effects for the Seven Pillars intervention were found in supplemental analyses undertaken on a “per malpractice claim” basis (e.g., number of lawsuits per claim, and liability costs per claim were reduced).
• Self-insurance costs for malpractice coverage, which were increasing prior to the intervention, were decreasing afterward; likewise, the self-insurance fund balance, which was negative and decreasing prior to the intervention, moved from a $30 million deficit to a $40 million surplus.

The grantee provided no data on the impact of Seven Pillars at the 10 replication hospitals. Training and technical assistance were provided to each of the community hospitals, and a secure data repository was established for the participating hospitals to submit patient safety data. Obtaining data on risk management and liability outcomes proved to be the most challenging. The grantee received only a few months of risk management and liability data from some of the replication sites. Although the community hospital staff valued the training and technical assistance, the risk management staff were already busy prior to the implementation of Seven Pillars and took on additional burdens related to the intervention without the support of grant funding.

Evaluation Observations

As mentioned above, the outstanding research question that this demonstration intended to answer is whether the existing disclosure and resolution model is a good “fit” for community hospitals, and if not, what alterations might need to be made to make it a good fit. While the demonstration was successful in accumulating more evidence for the model in self-insured academic settings, the inability to replicate the results of Seven Pillars in the community hospitals leaves that important question unanswered. We know from the experience of the UIC project that it is possible to package the training and tools and to implement them in community hospital settings; however, without outcome findings on the intervention, we do not yet know whether Seven Pillars will work outside of the settings in which it was developed.

The demonstration findings concerning the implementation of the Seven Pillars intervention at the UIC Medical Center reflect a well-designed, thoughtful assessment of liability and safety outcomes. Although the grantee’s evaluation of UIC outcomes did not use a control group for comparison, the use of an interrupted time series design helps to strengthen the inference that the observed differences are a result of the intervention rather than a larger trend affecting the experience of all hospitals in the community (e.g., adoption of new Joint Commission standards). The findings at UIC illustrate the impact that the Seven Pillars intervention is capable of having. We also know that the inability to replicate the results of Seven Pillars at the 10 community hospitals reflected a series of implementation and data collection challenges that the demonstration team struggled to overcome. This is not the only project that attempted to work with hospitals to implement disclosure and resolution programs and struggled to implement program components and data collection efforts without new staff (or dedicated staff time). The assumption that hospitals can establish these labor-intensive programs within existing resources may be flawed. Also flawed may be assumptions about their capacity to produce the kinds of data necessary for sophisticated evaluation efforts without specific funding to do so. Understanding the possible reasons for the partial implementation of Seven Pillars is critical to future replication work.

University of Texas (UT)

This project aimed to implement and evaluate a disclosure and resolution model in UT hospitals and to identify best practices using disclosure to improve patient safety, with a focus on incorporating patient and family input into efforts to understand why errors occur. Key activities were (1) developing and testing a tool for UT hospitals to use to assess “disclosure culture” and (2) using a structured interview guide for gathering information from patients and family members after an adverse event. Grant activities involved six health institutions (four medical schools, one cancer center, and one health science center) in the UT System.

Initially, the grantee intended to implement and evaluate disclosure and resolution programs in UT hospitals and provide training for hospital staff in disclosure and apology; however, the demonstration project was not designed to actually implement the programs. The implementation was left to the individual hospitals, and they did not succeed on their own. Participants in the site visits told us that the training provided through the grant was excellent, but it was merely the first step in raising consciousness of the issues. They found a one-time training to be insufficient for helping risk managers to develop disclosure programs on their own without resources already in place. Careful attention needs to be paid to the use of “train-the-trainer” models without followup support and ongoing technical assistance. The training did begin to stimulate culture change within the hospitals (as reported by site visit participants), and the grantee used project resources to develop tools to assess changes in attitudes over time. Although there are no findings to report on the impact of DRP, the grantee was able to share evaluative data on three key aspects of their efforts, described below.

Perceptions about error disclosure. One of the most important products from this project has been the development of a survey tool to collect data from hospital staff on their attitudes toward disclosure and resolution. A new 51-item survey was developed to examine the relationship between patient safety culture and error disclosure. Clinical faculty members from six UT hospitals were invited to complete the electronic survey before they participated in the disclosure and resolution training (2010) and at the end of the project (2013). Approximately 10 percent (497 of 5,000) of the clinical faculty completed the first survey, and 24 percent (1,217 of 5,000) of the clinical faculty completed the second survey.⁵ Findings included the following:⁵

• Compared with faculty who had not participated in the disclosure training, faculty who participated in the training had significantly more positive attitudes about error disclosure and perceived disclosure as less damaging to patient and peer trust in them.
• Comparing results from the first to the second survey, respondents overall had more positive perceptions toward disclosing minor errors (increasing significantly in their positive scores from 33% to 52%).^a Moreover, respondents overall had more positive perceptions toward disclosing serious errors (increasing significantly in their scores from 53% to 70%). This suggests that perceptions of error disclosure change in a positive direction after participation in error disclosure training.
• The survey asked respondents about the error disclosure “trust” culture (that is, their perceptions about whether disclosing a medical error in their clinical area might damage patient trust [or peer trust] in their competence). Comparing results from the first to the second survey, there were no changes overall in respondents’ perceptions about error disclosure “trust” culture in their organizations.
• Comparing results from the first to the second survey, there were significant improvements in respondents’ overall perceptions about their organization’s safety culture (scores increased from 50% to 63%) and teamwork culture (scores increased from 62% to 73%). This latter finding suggests that perceptions of an organization’s safety culture and teamwork culture change in a positive direction after participating in error disclosure training.

Patient and family member experience of medical harm after adverse event.⁶ Seventy-two patients and family members who had experienced harm after an adverse event were recruited to participate in structured interviews with the goal of creating a “debriefing tool” that could be used by UT hospitals and others. Most of these participants were not UT patients. Of those interviewed, most reported that they would like to participate in the hospital’s adverse event analysis process and be asked for their perspective, but they disagreed about the ideal timing of that involvement, which might vary by individual and the nature and severity of the event. Further, patient and family member preferences varied on the ideal person to conduct a debriefing. Although all had experienced harm, most respondents had no intention of pursuing legal action but felt (in some cases) that litigation was the only reliable way to access information about what happened to them. About half of participants indicated they would prefer to engage in adverse event analysis through a structured interview format rather than, for example, attend a meeting with the hospital board, complete a survey or questionnaire, or initiate a report to the hospital or a higher authority (i.e., a medical board or State regulatory agency).

Malpractice claims.⁷ The grantee aimed to establish a baseline level of legal claims and other malpractice outcomes and to develop an approach that could be used in the future to assess the effects of a uniformly implemented DRP on legal claims and outcomes over time within the UT system. Toward that end, the grantee examined five “snapshots” of malpractice claims over a period from 2001 to 2012. The five snapshots included all malpractice claims that closed within the system during the years 2001-2002; 2006-2007; 2009-2010; 2010-2011; and 2011-2012 for a total of 715 closed claims. Of the 715 claims reviewed, 148 (21%) resulted in a payment, with the remaining being closed without payment. All 148 payments were reached through settlement, none through trial.

These are baseline data. At this point, the grantee has no data on the effectiveness of the DRPs—the DRPs were not implemented as intended, and it would be too early to detect malpractice outcomes. What the grantee can do, however, is look at a contextual issue, which is the impact of malpractice reform in Texas on claiming behavior in the UT System. This contextual issue is important because future evaluations will need to be able to differentiate the impact of DRP from the impact of earlier, more traditional malpractice reforms (such as damage caps and changes to the statute of limitations) and from other environmental forces that may be affecting all hospitals in Texas (the “secular trend”).

Overall, the number of claims, lawsuits, and paid claims in the UT System decreased after Texas enacted malpractice reform in 2003). An analysis comparing data before malpractice reform (2001–2002) and after malpractice reform (2009–2012) revealed the following:

• The proportion of claims resulting in settlements did not change significantly after malpractice reform was enacted, but the proportion of claims in which lawsuits were filed decreased significantly (54% vs. 31%; p<.001).
• The proportion of claims that were dismissed decreased significantly (34% vs. 22%; p=.001), while the proportion of claims that were closed due to an expired statute of limitations increased significantly (32% vs. 51%; p<.001).
• The average (mean) payments decreased significantly from $290,992 to $90,429 (p<.001).
• Settlements reached after malpractice reform were significantly more likely than those before malpractice reform to prohibit error disclosure in each of a variety of ways (p<.001). Across all years, in a subset of relevant settlement agreements (124), the great majority (89%) included nondisclosure agreements.
• The length of time from an event to a claim decreased (no doubt due to changes in the statute of limitations), but there was no significant change in the amount of time from claim to closure.

Evaluation Observations

The UT demonstration stood out from the others for its primary focus on patients and families and its various efforts to determine how and under what circumstances patients who had been harmed by medical error could contribute to efforts by hospitals to improve patient safety. Patients and their family members who participated in the development of the debriefing tool told the UT investigators what others have said before them: Litigation is sometimes undertaken as much to get information about what happened, and to get assurance that someone is addressing the problem so that no other patient is harmed in the same way, as it is to get compensation for injury. This is an important “lesson learned,” and the efforts of this grantee to bring the perspective of the harmed patient to the center of the investigation of medical errors are a unique contribution that deserves follow-on work.

The UT demonstration findings focus primarily on specific tools and interventions that were developed by the UT team in the context of its broader effort to develop and implement a disclosure and resolution model among UT hospitals. The project provides results on a disclosure culture survey for physicians, before and after the administration of disclosure training. The survey results superficially indicate that disclosure training has a positive impact on disclosure culture. Furthermore, the results provide some additional support for describing and validating the survey instrument, which potentially could be a useful tool for other hospitals seeking to undertake DRP interventions in the future. The patient and family member interviews, by contrast, involve a qualitative approach to investigating patient and family perspectives on disclosure and event analysis processes. The results are interesting and potentially useful for the design of patient-centered debriefing and event-analysis processes in the future. Although worthwhile in their own right, neither of these project components speaks to the original intent of the UT team to field a DRP intervention in multiple hospitals and to investigate the impact of that intervention on malpractice claiming and other outcome measures. No data are available concerning the latter, although the grantee undertook a baseline analysis describing the impact of malpractice reform on UT closed claims. This analysis includes a series of comparisons of UT hospital claims before and after the introduction of malpractice reforms in Texas; however, the grantee did not explain what statistical methods were used to generate these findings. Based on what the researchers did describe, it is not possible to determine whether these reported changes are impacts of the reforms or are related to other changes that might have occurred in participating hospitals over the same time period.

University of Washington (UW)

This project formed a statewide initiative (The HealthPact) to serve as a vehicle for informing and sustaining the project’s goals. It comprised stakeholders from across the State interested in enhancing communication and accountability in health care delivery. An important focus of this project was to measure and track patient safety events that are especially sensitive to communication breakdowns. The grantee implemented three interventions aimed at improving communication to prevent and respond to medical errors: (1) team communication training, with the goal of establishing trainers at 10 sites to provide site-specific team communication training; (2) error disclosure training, with the goal of establishing coaches at the same 10 sites to provide guidance to providers and risk management staff on using error disclosure processes; and (3) disclosure and resolution programs (which the grantee called Communication and Resolution Programs), with the goal of developing a collaborative communication and resolution program in an “open” system (environments where multiple insurers are involved in addressing adverse events). The communication and resolution program was launched at five health care institutions: three hospitals within a single hospital system and two multispecialty physician clinics.

Unfortunately, the project was unable to implement the communication and resolution program across the five facilities within the 3-year grant period. Some sites did begin implementation, but risk and claims managers tended to use the communication and resolution program selectively and did not track cases as directed. As a result, the grantee was able to collect data on only 30 communication and resolution program cases, an insufficient number for drawing any conclusions about the effect of the communication and resolution program on liability. Although we do not have findings to report on the implementation of the communication and resolution program, the grantee was able to share evaluative data on two key aspects of its efforts, described below.

Perceptions about error disclosure training.⁸ In qualitative interviews with organizational leaders, respondents remarked on the ongoing need for organizational support, raised questions about the amount of training and available resources to sustain the budget for training, and voiced concern about management of errors. Training evaluations, which were completed by 159 of the 251 (63%) trainees, revealed that the training was well received, with ratings for each item and subscale being high. The results from an attitude survey,^b completed by 85 of the 251 (40%) trainees, showed that respondents were enthusiastic about the training but lacked confidence that their organizational leaders had a shared vision and perspectives around using new error disclosure processes. The Disclosure Culture Scale^c revealed respondents’ general agreement that there was organizational support for disclosure of error to patients, but their responses were less in agreement about whether adequate training on disclosure was provided or whether retaliation, loss of trust, or damaged reputations were concerns.

Communication-sensitive adverse events (CSAEs).⁹ the term “CSAE” was coined by the UW project and defined as “an adverse clinical event for which a lack of adequate communication may have been a contributing factor”⁹ (p. 6). To assess the impact of communication training on the identification of and response to CSAEs, the grantee plans to conduct a comparison of the rates of CSAEs at participating sites and control sites. At the time of the grantee’s final report, only baseline data on CSAEs were available. Data from two State databases were obtained to conduct retrospective analyses—administrative data from the Comprehensive Hospital Abstract Reporting System (CHARS) and clinical registry data from the Surgical Care and Outcomes Assessment Program (SCOAP). For the baseline period of 2009–2011, CHARS data included 4 of the intervention sites and 93 comparison sites, and SCOAP data included the same 4 intervention sites as well as 48 comparison sites. The CHARS dataset contained 1,684,916 unique patients, including 211,593 who experienced a CSAE. The SCOAP dataset contained 51,537 unique patients, including 5,004 who experienced a CSAE. The baseline rates for individual CSAEs were generally very low (<0.001% for CHARS and <1% for SCOAP), with most rate ratios favoring intervention hospitals over comparison sites at baseline. An assessment of the differences between CSAEs over time (before vs. after implementation of the communication training) is forthcoming.

Evaluation Observations

The UW demonstration was extremely ambitious in its project aims. It was an enormous undertaking to establish HealthPact and engage the right stakeholders. Once the project started encountering challenges with implementing the communication and resolution program, the group’s focus narrowed considerably to supporting communication and resolution program and promoting patient-centered accountability in systems that involve more than one insurer. Patient advocates associated with HealthPact led the way to expanding participant understanding of the immense distance between the health care system’s current response to medical injury and patient and family needs.

The project trained a large number of clinicians; for the team communication training, 71 participants fulfilled the Master TeamSTEPPS^® training, 69 participants completed the train-the-trainer module, and over 1,300 providers completed the front-line clinician training. Almost 400 disclosure and apology coaches were trained. The grantee learned a lot from conducting trainings, including recognizing the need to move to a “train-the-trainer” model. This approach required sites to identify a “champion” and a “change team” and to send them to UW for additional training. This modification was a significant expansion of the scope of the project that put enormous stress on grantee faculty and support staff as well as the sites.

As mentioned above, the UW demonstration project ran into difficulties in implementing the planned communication and resolution program intervention across the State. Perhaps the biggest challenge was that each participating site differed in the degree to which its organizational culture, policies, and practices were aligned with the key elements of the communication and resolution program. (This was true in each of the demonstration projects in the PSML initiative that attempted to implement DRP.) Each host organization also had a different level of commitment by organizational leaders, which was reflected in a different level of resource allocation (e.g., risk and claims staff dedicated to data collection). Additionally, when it was time to use the communication and resolution program, sites tended to be reticent about being the first to use the new process. Risk and claims managers (being risk adverse) preferred to learn from initial experimentation by others. Some risk and claims managers also preferred to wait for an “ideal” case (one without high stakes but one significant enough to warrant the effort). Physicians were also hesitant to participate in the communication and resolution program, their main concern being possible punitive action by the State board of medicine. The UW grantee concluded that “operating communication and resolution programs in which two or more insurers must collaborate to resolve cases is highly challenging and likely requires several preconditions not present for our sites, including a commitment from physicians to collaborate with facilities to resolve incidents, mechanisms for quickly transmitting information to remote insurers, tolerance for missteps in early attempts at collaboration, and clear protocols for joint investigations and resolutions”⁹ (p. 18).

The findings from the baseline assessment of CSAEs are descriptive, rather than evaluative, and are preliminary. We offer no comment on assessment methodology for the communication and resolution program because the grantee did not ultimately conduct such assessments.

Improving Communication—Lessons Learned

Building robust disclosure and resolution programs involves challenges for both implementation and evaluation. In principle, DRPs operate to ensure better disclosure and communication between providers and patients in the aftermath of an adverse event, with positive downstream effects on malpractice outcomes and patient safety culture. In practice, however, these programs can be challenging to establish, especially within the existing resources of host institutions. It is notable that grant resources tended to go toward the development of tools and training and not to operations. An assumption was made that DRP could be supported by hospitals within existing risk management resources, but the experience of these projects suggests that assumption may be faulty. One disadvantage that the four demonstration projects shared—unlike their predecessors (like University of Michigan)—was that they were “outsiders” (of a sort) trying to convince hospitals to implement and evaluate the impact of DRP, rather than “insiders” developing a home-grown approach to disclosure and resolution. From their combined experience, it seems critical to implementation that institutional cultural change take place. Fortunately, these four grantees have much to share about what that change looks like.

The experience of these projects can teach us a lot about what went right (e.g., support from engaged and motivated institutional leadership, building on existing patient safety initiatives, committed and purposeful project leadership), what can go wrong, and how future replications might avoid these problems. Review of future proposals to support DRP implementation might include specific attention to whether participating hospitals (and other host organizations) have an adequate budget provided by the grant or other sources and dedicated staffing to implement the DRPs described in the proposals. In addition, the review might assess whether timelines are sufficient for bringing the project to scale prior to the start of data collection.

While we learned a lot from the experience of these four demonstrations, one critical question posed in at least two of the applications remains unanswered: Is the DRP model suitable for non-academic, non-self-insured hospitals? Early demonstrations of DRP (including those at the University of Michigan, Stanford, and the UIC hospital within the PSML portfolio) have illustrated the promise of the DRP model but also many of the difficulties involved in exporting a DRP to community hospital settings. Within the PSML portfolio, the UW project also experienced these difficulties in its attempt to negotiate a plan for implementation that would bring together a large group of stakeholders across the State. This challenge exists because the non-academic hospital setting involves an elaborate web of risk relationships among physicians, hospitals, malpractice insurers, and patients, as well as “risk shifting” behavior among these parties. In this environment, the incentives to litigate or settle malpractice claims are not aligned across the parties involved. Because of the failure of any of the demonstration projects to replicate in non-academic settings despite plans to do so, this important policy question remains unanswered. Although there is accumulating evidence (some of it strong evidence) that DRPs can positively affect both patient safety and malpractice outcomes, there is no evidence to date as to the outcomes that could be reasonably expected from implementation across a broad range of hospital types (such as academic and non-academic, urban and rural, within integrated health systems and stand-alone hospitals, those that employ physicians and other health professionals and those that do not).

From a research standpoint, tracking the impact of a DRP requires good data on malpractice and safety outcomes, including measures related to mediating factors such as changes in institutional processes and patient safety culture. Capturing meaningful claims data, in particular, may require several years due to the malpractice “claims tail” (i.e., malpractice claims frequently take 5 years from the occurrence of an adverse event to surface and/or resolve). As a result, any effort to assess the outcomes associated with a DRP is likely to require ongoing support on the part of a host institution and 6 to 10 years of data collection.

Preventing Harm Through Best Practices

Three grantees—Ascension Health, Fairview Health Services, and the Massachusetts State Department of Public Health—aimed to prevent medical errors and poor health outcomes while reducing malpractice lawsuits by implementing “best practices.” Two projects (Ascension Health and Fairview Health Services) focused on spreading clinical best practices for safety interventions to hospital-based perinatal units and obstetrics departments. One project (Massachusetts State Department of Public Health) focused on increasing the efficiency and efficacy of high-risk clinical and communication processes in a group of outpatient primary care practices.

Ascension Health

This project focused on ways to improve the quality of perinatal care delivery and the management of adverse perinatal events in five geographically dispersed hospitals. The impetus for this project was twofold: Previous perinatal safety projects undertaken by Ascension Health had shown some success in reducing preventable injuries, but these projects had been disseminated somewhat unevenly and their impact was leveling off, and the organization had a desire to continue identifying new safety interventions in the perinatal area. For the current project, a uniform, evidence-based obstetrics practice protocol was established for dealing with shoulder dystocia.^d Use of a uniform practice protocol is based on the idea that eliminating variation in adherence to clinical guidelines during perinatal emergencies will translate to improved patient safety. Key project tasks included creating broad-based physician engagement and adoption of all elements of the previously tested perinatal safety initiative and adding a comprehensive team approach to labor and delivery management that incorporates a shoulder dystocia management program. This project also involved developing, implementing, and evaluating error disclosure through a coordinated communication intervention with patients and the care team. Other elements of the intervention included training labor and delivery clinicians on electronic fetal monitoring, teamwork and communication, and documentation of unintended events. The project team used the principles and practices of a High-Reliability Organization (HRO) to facilitate adoption and spread of the intervention. This package of interventions was intended to reduce the number of malpractice lawsuits, consequently decreasing the annual malpractice funding expense for obstetrics. This grantee’s findings are described below.

Shoulder dystocia management.¹⁰ Following the adoption of the new protocol to prevent shoulder dystocia, 99 percent of deliveries involving shoulder dystocia had a head-to-body delivery within 3 minutes, a finding sustained for 2 years after implementation of the new protocol.

Event reporting.¹¹ Six months following the end of the intervention, the five participating hospitals demonstrated improvement in event reporting, nearly doubling their rate of reporting for all unexpected medical events in labor and delivery per 1,000 births (43 vs. 84 per 1,000 births; p<.01).

Harm.¹¹ Six months following the end of the intervention, the project demonstrated reduction in harm at the five participating hospitals, decreasing the rate of high-risk malpractice events for shoulder dystocia per 1,000 births by half (14 vs. 7 per 1,000 births; p<.01). Further, incidents of shoulder dystocia decreased 50 percent.

Shoulder dystocia-related claims. In 2009, prior to the intervention, the rate of obstetric malpractice claims at the five participating hospitals ranged from 1.5 to 5.4 per 1,000 births. In 2013, 2 years after establishing the new protocol, none of the five hospitals had malpractice claims based on shoulder dystocia.¹⁰ As a result of these safety improvements, some of the money that had been saved for malpractice claims was diverted to support implementation of the intervention at 42 additional Ascension Health hospitals^.11

Evaluation Observations

This project accomplished what many other PSML demonstration projects did not—it created institutional cultural change that was necessary for the package of interventions to take root and be sustained past the grant period. Ascension Health’s High-Reliability Organization culture seemed to facilitate the uptake of both the new shoulder dystocia protocol and new adverse event reporting procedures. Moreover, acknowledging the specific culture of each demonstration site, the project team collaboratively developed the protocol, training, and maintenance plan with stakeholders at each hospital. Allowing the demonstration sites to work through implementation and data collection processes was critical given their diverse experiences with evidence-based guidelines and research. It also increased buy-in and follow-through. Similarly, initiating a “real-time” review of unexpected events through the DRP was very uncomfortable at first. Typically, risk managers review claims by themselves long after an event occurs; having a team review the claim together with the physician lead, nurse lead, and risk manager in real time was a cultural change that was closely monitored through regular phone calls with the project team and representatives from the demonstration sites. Open communication was encouraged when problems arose.

Compared with the other projects that offered training, Ascension trained a high percentage of clinicians. During the rollout of the training, the project team heard from participants that the training was valuable, but that it needed to be relevant and accessible. The project team adjusted the original 7-hour training requirement to create an individual learning path for each physician that fit with his or her level of decisionmaking. As a result, all 76 clinicians serving on an Obstetrics Event Response Team in each of the 5 participating hospitals received training on implementation of the high-reliability root cause analysis and quick-response model. In the same hospitals, 93 percent of 302 clinicians completed all the trainings offered on disclosure and resolution and documentation.¹¹

The project’s outcome findings related to reducing harm and high-risk malpractice claims are suggestive, but the research design the grantee used and the data analysis it reports make the findings difficult to interpret. In particular, the reported findings appear to be based on a simple pre/post comparison on several outcome measures in hospitals that implemented a new risk-reduction model without any control hospitals. The grantee did not address the possibility that factors other than the introduction of the new care model (e.g., adoption of different evidence-based guidelines in the labor and delivery units) might have contributed to the observed pre/post differences. Also, the outcome measures reported are not well described, and the measure of “high-risk malpractice events” in particular is not explained. Taken in total, and given the research design, analytic approach, and measures, we do not have strong confidence in the reported findings.

Fairview Health Services

The PSML grant supported Phase II of the Premier Patient Safety Initiative study (2011–2012), which built on the success of Phase I (2008–2010), conducted prior to the PSML initiative. Although improvements in bundle compliance^e were achieved in Phase I, the study period was presumed to be too short to achieve 90 percent or higher compliance. It was believed that additional time and effort were needed to achieve high-reliability performance. Phase II aimed to restart the Phase I program and create high-reliability teams and standardized care processes in the perinatal units of 14 of the 16 Phase I acute care hospitals. The grantee anticipated that increasing team communication skills around patient care bundle implementation would lead to improving compliance for these patient care bundles, which would lead to a reduction in harm and the associated malpractice claims and costs. Interdisciplinary teams were trained in teamwork concepts and management of critical labor and delivery events through a combination of in-situ simulation and TeamSTEPPS^® training. The project implemented the same three patient care bundles from Phase I: elective induction (a procedure used to stimulate labor before it begins on its own, for a nonmedical reason), augmentation (a medical intervention used to stimulate contractions when labor has started on its own but is not progressing), and vacuum extraction (the use of a suctioning device attached to the baby’s head to ease the baby down the birth canal). The grantee reported the results below.

Bundle compliance.¹² Based on chart review, the use of the three patient care bundles increased from the rates established during Phase I. In Phase II, between Quarter 3 of 2010 and Quarter 4 of 2012, average compliance increased from 54 percent to 70 percent for the vacuum extraction bundle, 74 percent to 86 percent for the augmentation bundle, and 87 percent to 88 percent for the elective induction bundle. Participating hospitals improved overall compliance with all the patient care bundles from 72 percent to 81 percent during this time.

Harm and clinical measures. The grantee reported findings on four composite scores that measure perinatal adverse events: the Adverse Outcome Index (AOI), the Weighted Adverse Outcome Score (WAOS), the Severity Index (SI), and the AHRQ Patient Safety Index (PSI) 17-birth trauma score. All of these scores, which are defined in the grantee profile in Appendix A, represent measures of events that are potentially modifiable through improved teamwork.

• When examining the differences between the intervention and control hospitals, from Quarter 2 of 2010 through Quarter 4 of 2012, the proportion of deliveries with an adverse event (as measured by the AOI) decreased for the 14 participating hospitals and increased for the 8 comparison hospitals (p=.025). Differences between the intervention and comparison hospitals were not detected for three other adverse event scores (WAOS, SI, and PSI 17).¹²
• When examining only the intervention hospitals, from Quarter 1 of 2006 through Quarter 4 of 2012, the proportion of deliveries with an adverse event (as measured by the AOI) decreased 14.4 percent (p=.032). There were no significant decreases in the other measures of harm (WAOS, SI, and the PSI 17).¹⁴

Malpractice claims. Two years into the intervention, a retrospective audit was conducted on 64 obstetric claims made against the 14 intervention hospitals. For the births occurring from 2001 (before Phase I) through 2012 (through Phase II), the project reported a 19% reduction in frequency of obstetric claims per 10,000 deliveries (2.1 to 1.7 per 10,000 deliveries). “Incurred” amounts (loss payments on closed claims and reserves set aside to pay pending claims) were significantly higher before the intervention at $17,908,000 compared with incurred amounts on claims for births during the intervention period at $4,651,325.¹²

Evaluation Observations

The Fairview Health Services project benefitted from the work completed prior to the PSML demonstration project, given that the 14 intervention hospitals participated in Phase I, during which many of the implementation startup issues had been addressed and the project protocols had been refined. Additionally, the participating hospital teams were familiar with the data collection and transmission processes. The demonstration did, however, face challenges with achieving the desired 100% compliance rate with the bundles due to competing demands of provider attention, a lack of active physician champions who were able to change peer behavior, and turnover of personnel. Team communication training was considered to be an important aspect of implementation; however, participation in training varied greatly by hospital, ranging from an estimated 10 percent to 93 percent of clinicians, with a median attendance of 50 percent of clinicians per participating hospital.¹²

Although the project findings reported by the grantee are suggestive of both patient safety and medical malpractice effects associated with the multisite perinatal care intervention, the strength of those findings is limited by concerns about the research design. Broadly, the design of the Fairview evaluation involved pre/post tracking of patient safety outcomes (adverse event measures) at 14 intervention hospitals and malpractice outcomes at 13 intervention hospitals over a period of 7 years (for the adverse event measures) and 4 years (for the malpractice measures). Although control hospitals were included in the study, many of the final analyses conducted by the Fairview team did not involve data collected from these control hospitals. As a result, it is difficult to draw conclusions from the evaluation, because it is unclear whether the observed findings are the result of the intervention or other factors that may have been influencing the intervention hospitals at the same time. This problem (pre/post comparisons without controls for all measures) broadly applies to many of the reported findings.

With regard to perinatal safety outcomes specifically, the Fairview researchers investigated the impact of their intervention on four different measures of adverse outcomes. The researchers detected a significant change in only one of the four measures (AOI) following the intervention. They did not address this finding or explain why three of the four outcome measures were apparently unaffected by the intervention, although they did note in their final report that the measures of harm selected by the project might not be amenable to improvement at the level aspired to by the project.

With regard to malpractice outcomes, the grantee had access to more years of claims data due to its participation in Phase I of this study, allowing for a longitudinal analysis of project findings. Given this advantage, it is not clear why the grantee did not include the control hospitals in the final claims analyses. Ultimately, the Fairview researchers found that the intervention was associated with statistically significant reductions in several measures of malpractice claims and costs at participating hospitals. However, the meaningfulness of these findings is qualified both by the uncontrolled pre/post research design and the small absolute number of claims observed during the study. Taken in sum, and given these methodological concerns, we believe that the outcome findings reported by the grantee should be interpreted with caution.

Massachusetts State Department of Public Health

The Proactive Reduction of Outpatient Malpractice: Improving Safety, Efficiency, and Satisfaction (PROMISES) project is an extension of the work done by researchers from the Harvard Malpractice Study team. This group published groundbreaking findings demonstrating that preventable harm in hospitalized patients was much more common than previously known and that malpractice suits often fail to identify seven of every eight cases when preventable harm occurred.¹⁵ PROMISES broadened the focus to outpatient primary care practices, where most health care is delivered and where the majority of malpractice claims now originate. The project aimed to determine whether implementation of one or more quality improvement (QI) activities in a group of 16 outpatient primary care practices would improve safety and reduce medical malpractice risk. Its focus was on specific areas of care management that contribute to a large proportion of medical errors in ambulatory settings: (1) the management of laboratory test results, (2) the management of referrals (for example, to specialists), (3) medication management, and (4) communication among care providers and with patients. Providers and staff from the intervention practices participated in learning Webinars and received data feedback and customized coaching by Improvement Advisors in the in the use of QI methods (PDSA cycles) for improving workflow and efficiency. Some grantee-reported outcomes from its evaluation are described below.

Safety practices.¹⁶ Chart reviews were conducted by the grantee at each of the intervention sites before and after the intervention to capture objective information about how abnormal lab results, referrals, and medications were handled. Up to 100 charts were reviewed at each of the 16 intervention sites. The project team recorded all occurrences of abnormal records for the intervention’s targeted labs or any high-risk referrals where there was no documented followup. From the beginning of the project to the end, chart reviews showed the following:

• The prevalence of four measures of “abnormal” laboratory test followups significantly decreased after the practices participated in the intervention. Specifically, there were decreases in instances when results were not in the chart (2.2% to 0.8%), when the action plan was not documented in the chart (20.4% to 14.4%), when a patient was not notified of a test result (20.8% to 15.0%), and when an action plan was not completed (19.3% to. 10.8%).
• The rates of potential safety risks^f associated with abnormal lab results significantly decreased after the practices participated in the intervention. Specifically, the rate of potential safety risks decreased from 155 per 1,000 patients to 54 per 1,000 patients, which is a 65 percent reduction in rates of abnormal lab results or high-risk referrals where there was no documented followup in the chart.
• The rates of serious potential safety risks^g associated with abnormal lab results significantly decreased after the practices participated in the intervention. Specifically, the rate of serious potential safety risk decreased from 28 per 1,000 patients to 13 per 1,000 patients, which is a 54 percent reduction in events where potential or actual harm was found and there was no documented followup in the chart.

Patient perceptions and attitudes.¹⁷ Based on patient surveys administered at the beginning of the project and again at the end, there were no significant differences in how patients perceived their experience at the 16 intervention sites versus the 9 control sites. Among only the intervention sites, patients had different attitudes about the sites identified by the Improvement Advisors as “more engaged” in the project’s improvement activities versus the sites identified as “less engaged.” Patients rated the more engaged intervention sites as significantly more positive when it came to practice communication, practice leadership, and test result management.

Staff perceptions and attitudes.¹⁶ Approximately 61 percent (292) of all practice staff members completed the first survey before the initiation of improvement activities, and 60 percent (287) of all practice staff members completed the second survey at the end of the project.

• Overall, from the first to the second survey, staff at intervention sites rated their experience with their practice as showing greater improvement compared with staff at control sites across all 11 domains (access to service and care, medication management, referral management, test result management, malpractice concerns, patient-focused care, quality and risk management, practice communication, work environment, teamwork, and practice leadership).
• From the first to the second survey, there were no significant differences between the average staff ratings of the intervention sites and the control sites in terms of the three high-risk domains targeted by the PROMISES intervention (malpractice concerns, patient-focused care, and teamwork).
• Among intervention sites, staff ratings from the “more engaged” sites showed improvement from the first to the second survey in their ratings on three domains—practice communication (p<.001), practice leadership (p<.05), and test result management (p<.05)—compared with ratings from staff associated with the “less engaged” intervention sites.

Evaluation Observations

The PROMISES demonstration was unique in that it implemented its intervention in outpatient practice settings. This involved conducting a lot of field work and spending time getting to know staff and operations at 16 individual primary care practices. PROMISES experienced delays in getting practices up and running, which affected implementation as well as evaluation measurement. The project team came to appreciate the different levels of innovation readiness of the practices. Moreover, the Improvement Advisors found it challenging to provide meaningful assistance for all levels of practices. In general, it was a challenge to engage and sustain the attention of busy practices; practices are overwhelmed with many competing demands (e.g., meaningful use, other QI projects), they are chronically understaffed (100% of the practices experienced turnover in leadership positions), and many practices could not automate data collection and had to put “workarounds” in place to improve work processes. The project has helped to identify the conditions under which such evidence-based programs can readily be adopted in primary care practices, as well as conditions under which their adoption becomes more difficult.

The work of the PROMISES demonstration is noteworthy for having proposed the strongest research design of the seven PSML demonstration sites, involving a prospective randomized controlled trial (RCT) to assess the impact of a statewide collaborative safety intervention in ambulatory care. An RCT is a strong research design; it involves intervention sites and similar control sites that do not participate directly in the intervention. This allows investigators to answer the “effectiveness” question because they can directly analyze the impact of the intervention over time while controlling for many unrelated effects that are observed at the control sites.

Unfortunately, the PROMISES demonstration ultimately did not follow through on the initial plan to conduct an RCT since the grantee spent much of its resources engaging practices to participate, leaving insufficient resources to conduct chart reviews at the control sites. Instead, the project’s reported findings on the outcome measures represent an uncontrolled, pre/post comparison of those measures within the intervention settings. This is a weaker research design than an RCT, diminishing our confidence that the observed findings (particularly on safety risks) were actually caused by the PROMISES interventions. In sum, we would say that this project produced an interesting and suggestive set of findings concerning a package of patient safety interventions in the ambulatory care setting. However, these findings do not directly address the impact of the demonstration on malpractice risk, and the main findings the grantee has reported regarding patient safety risks are correlational rather than causal.

Preventing Harm Through Best Practices—Lessons Learned

The projects that implemented evidence-based guidelines, with careful attention to training teams on communication processes and skills specific to those evidence-based guidelines, appear to be promising models for inpatient and outpatient settings that are seeking to prevent harm to patients. We learned a lot from the three projects (Ascension Health, Fairview Health Services, and Massachusetts State Department of Public Health), especially what it takes to seed, nurture, measure, and sustain improvements in patient safety processes. All three projects incorporated the input of stakeholders from the local settings (e.g., obstetrical units and risk management departments, primary care practices) from startup through project completion. They used site champions to inform the project about the local conditions and assist with identifying barriers and facilitators to implementation. The project leaders were inclusive of the patient’s perspective, as two of the three projects interviewed patients during the intervention to solicit their specific viewpoints. The three projects modeled approaches that emphasize flexibility and customization of the evidence-based intervention and data collection processes to meet site-specific needs and circumstances. They encouraged participants to raise issues and have open dialogue, working together to resolve problems. All three projects had resourceful, committed leaders at the helm. From our interactions during site visits, it appeared that these team members worked well together and were unified in operationalizing the project’s goals and approach. Their background in the research that supports the evidence-based guidelines and their working knowledge of quality improvement processes were also valued by the sites. Clearly, much has been learned from the project teams and their approach to seeding and sustaining standardized patient care processes under this PSML initiative.

Exploring Alternate Methods of Settling Claims

To improve dispute resolution after a malpractice claim has been filed, one demonstration project conducted by the New York State Unified Court System worked with five academic medical centers in the City of New York to send their malpractice claims through a novel program known as “judge-directed” negotiation (JDN).¹⁸

New York State Unified Court System

In 2004, the Honorable Douglas McKeon, Administrative Judge of the Civil Division of the New York State Supreme Court, Bronx County, pioneered a new approach for malpractice cases involving New York City’s 11 public hospitals (under the auspices of the Health and Hospitals Corporation [HHC]). Anecdotally, the JDN program has been credited with decreasing the number of claims against HHC and annual medical malpractice costs by as much as $50 million a year since the program’s inception, reducing HHC’s indemnity costs from $190 million in 2003 to $144 million in 2008. HHC currently settles about 95 percent of its cases in chambers. For the demonstration program, an expanded and enhanced JDN program was implemented to expedite the movement of cases through the claims process, increase the number of settlements, and, over time, lower malpractice costs and premiums for its five participating hospitals across New York City boroughs. The grant also provided funding for a formal evaluation of JDN to document the program’s effect on claims, time to settlement, and medical malpractice costs for the five participating hospitals.

Between September 2011 and May 2015, 716 cases entered JDN and 165 reached a final disposition. Data analysis is ongoing, but the grantee reported the following preliminary descriptive findings based on the 326 cases that entered into JDN by the end of the original grant period:¹⁹

• Of the 326 cases, most (more than 80%) involved serious, permanent harm or death.
• Of the 32 cases with a final disposition, 15 were settled and 17 were voluntarily dismissed. Across these cases, the median time between filing and disposition of the case was 189 days.
  • Based on anecdotal data provided by the grantee, JDN cases are being resolved in a shorter timeframe compared with the historical median for cases in the Manhattan courts (718 days), Brooklyn courts (952 days), and Bronx courts (1,266 days).
• For 12 cases settled during JDN conference, the median settlement was $237,000, and the median time from filing to disposition of the case was 8 days. For three cases settled outside of conference, the median settlement was $55,000, and the median time was 240 days.
  • Based on anecdotal data provided by the grantee, JDN cases are being settled for similar amounts as compared with the historical median for cases in Manhattan courts ($223,750), for slightly lower amounts compared with the historical median for cases in Brooklyn courts ($255,000), and for lower amounts than the historical median for cases in Bronx courts ($350,000).
• Preliminary findings from the attorney surveys are encouraging: 90 percent of respondents thought the JDN program had reduced litigation costs for their case, 80 percent thought the JDN was a positive contribution to case resolution, and 90 percent were satisfied or very satisfied overall.

Evaluation Observations

As mentioned previously, the demonstration project in New York was multifaceted. It was originally designed with the idea that patient safety interventions in hospitals would reduce adverse events and thus claims. The thinking was that the DRPs would be utilized by the hospitals to disclose adverse events, and the investigations would lead to certain decisions, including providing compensation without the need for litigation. But if the cases were filed, the JDN program would bring them to a quick resolution (either with payment if the hospital was at fault or closure without payment if not). In practice, the hospitals were much more interested in participating in the expansion of the JDN program than they were in building DRPs.

The New York demonstration provides a descriptive summary of 326 cases that were processed through the JDN during the demonstration period. The findings are both illustrative and suggestive regarding the kinds of cases that were selected to participate in the JDN program and the handling and resolution of those cases. Previously reported results by the New York demonstration team have included the observation (based on anecdotal data) that cases passing through the JDN were resolved, on average, in less than half the time of a comparison group of baseline New York medical liability cases, although we have no information on which to judge the comparability of those cases. In sum, we would describe the findings from the New York demonstration as promising, but further research would be needed to rigorously investigate the magnitude of the effects generated by this program on malpractice outcome measures and to determine the statistical significance of any effects.

Exploring Alternate Methods of Settling Claims—Lessons Learned

JDN appears to be a promising model. The New York project was given a 2-year no-cost extension to allow for the collection of data on additional JDN cases. These data should be available within a year. Two concerns were raised by the implementation of JDN in New York: (1) whether any judge could be “taught” the necessary negotiation skills, and (2) whether this model might be an efficient approach to dispute resolution outside of large urban areas with a high number of high-cost cases. The first question has been at least partially addressed by the experience of the New York demonstration—five additional judges were trained and became part of the JDN program over the course of the 3 years of implementation. The second question is unaddressed at this time; should the followup analyses currently underway establish a strong relationship with malpractice outcomes, the question is worth further exploration by AHRQ.

---