On September 9, 2009, President Obama directed the Secretary of the U.S. Department of Health and Human Services (HHS) to authorize demonstration projects to put “patient safety first,” with the intent of reducing the occurrence of preventable injuries and deaths and ultimately stemming liability costs. In response, the Secretary launched the HHS Patient Safety and Medical Liability (PSML) initiative in October 2009. Funding was intended to address four goals: (1) putting patient safety first by reducing preventable injuries, (2) fostering better communication between doctors and patients, (3) ensuring fair and timely compensation for medical injuries while reducing malpractice litigation, and (4) reducing liability premiums.
Under the PSML initiative, the Agency for Healthcare Research and Quality (AHRQ) funded 7 demonstration grants totaling $19.7 million and 13 planning grants totaling $3.5 million. AHRQ commissioned James Bell Associates, Inc. (JBA), in partnership with RAND Corporation (RAND), to conduct an overarching, independent evaluation for this initiative. The seven demonstration projects were originally scheduled to run for 3 years beginning in late summer 2010. Many of the grantees requested and received no-cost extensions of varying lengths. All but one of the demonstration projects were completed by June 2014; the New York project received an extension and was completed in June 2015.
The demonstration grantees implemented complex, broad-ranging innovations in real-world settings, including hospitals and court rooms. Some projects featured novel approaches, while others implemented continuations, replications, or adaptations of existing models. All seven demonstration projects encountered challenges—some expected and others unexpected—that stretched project resources and required adjustments to implementation and evaluation expectations and strategies. Nevertheless, the projects had many accomplishments, such as developing and refining trainings, tools, products, and data collection instruments and contributing valuable learnings about what it takes to develop and sustain an operational patient safety and medical liability program.
This final evaluation report highlights the most substantive findings and lessons learned by the seven demonstration grantees.
Findings and Observations
In general, the seven demonstration projects focused on three main approaches to improving patient safety and reducing medical liability: (1) improving communication, (2) preventing harm through the use of best practices, and (3) exploring alternative methods of settling claims.
Improving communication. These grantees (New York State Unified Court System [NY], University of Illinois Medical Center at Chicago [UIC], University of Texas Health Science Center [UT], and University of Washington [UW]) pilot tested, replicated, and disseminated disclosure and resolution programs (DRPs). They helped to identify the conditions under which such programs can readily be adopted, as well as conditions under which their adoption becomes more difficult. Their findings are mostly descriptive.
Two projects (UT and UW) expanded our understanding of the potential role for patients and their family members in adverse event investigation and remediation. All projects in this category encountered a series of implementation and data collection challenges that the demonstration teams struggled to overcome. Consequently, only the UIC project, which started 4 years prior to the grant, was able to examine impact across multiple outcomes.
Preventing harm through the use of best practices. These grantees (Ascension Health, Fairview Health Services, and Massachusetts State Department of Public Health) showed that the implementation of specific evidence-based interventions (e.g., clinical bundles, team communication) may be associated with improvements in patient safety performance. Ascension and Fairview also offered suggestive findings that malpractice risk also may be reduced.
Exploring alternative methods of settling claims. One demonstration project (NY) sought to expedite the movement of malpractice cases through the claims process, increase the number of settlements, and, over time, lower malpractice costs and premiums through implementation of a judge-directed negotiation (JDN) program. This project’s preliminary findings are both illustrative and suggestive regarding the kinds of cases selected to participate in the JDN, the handling and resolution of those cases, and the effect that JDN may have on time to settlement and settlement amount.
Table 1 below briefly summarizes the interventions studied and selected findings for each of the seven demonstration projects and presents a brief statement about our independent assessment of the findings. More detail about the individual projects (e.g., principal investigator, grant award amount, goals, methods, analyses, findings) and an expanded description of the findings, our independent assessment of the findings, and lessons learned follow in the full report and the individual grantee profiles in Appendix A.
Table 1. Selected Grantee-Reported Findings for Projects Focusing on Improving Communication by Grantee Organization
| Grantee Organization | Intervention Studied | Summary of Selected Findings | Strength of the Evidence | Accomplishments |
|---|---|---|---|---|
| New York State Unified Court System (NY) | Communication and Resolution Program (disclosure and resolution program) | Data were collected on 125 communication and resolution program cases at the 5 participating hospitals in New York City. A small number of cases (20) progressed to a claim (6) or lawsuit (14) within 12 to 15 months after communication and resolution program completion. Eleven of the 20 cases with a claim or lawsuit involved substandard care and causation, and the hospital offered compensation in 3 of these 11 cases. | Grantee reports descriptive findings only; no statistical analysis is reported. No conclusions can be drawn about the impact of disclosure and resolution programs on outcomes. | This was an ambitious project involving 5 of the city’s largest teaching hospitals. A second project intervention consisted of implementation of a disclosure and resolution program. Overall, the hospitals implemented the disclosure part of the model with some success, with an initial disclosure of an adverse event to the patient or family documented in 79% of cases, explanation of the reason for harm provided in 88% of cases, and an expression of sympathy (without acknowledging responsibility) made in 64% of cases. The hospitals experienced greater difficulty implementing the apology and compensation components. Although the project was challenged with the hospitals undertaking a number of patient safety initiatives at the same time and added burdens to already overworked risk management staff, these data suggest some culture change. |
| University of Illinois Medical Center at Chicago (UIC) | Seven Pillars (disclosure and resolution program) | In an extension of an existing program, UIC Medical Center showed improvements in malpractice outcomes and adverse event reporting, including reductions of approximately 42% in the number of claims, 51% in the costs per claim, and 47% in the number of lawsuits. A significant reduction was found in mean time to closure per claim (from 4 to 2.4 years). The project reported significant improvements in communication processes, including a 52% increase in incident reporting, 96% increase in peer reviews, and a 91% increase in patient consults. The self-insurance fund balance moved from a $30 million deficit to a $40 million surplus. | Grantees employed a relatively strong research design for assessing the impact of the Seven Pillars intervention at UIC medical center. Longitudinal data suggest significant impact of the intervention on both patient safety and malpractice outcomes. | The demonstration was successful in accumulating more evidence for the model in self-insured academic settings. |
| Replication of Seven Pillars | The impact of the Seven Pillars intervention at 10 replication sites is not known because of insufficient data. | No evidence. | It was possible to package the training and tools and implement them in the community hospital setting. However, the hospitals struggled to implement program components and provide data on risk management and liability outcomes without new staff or dedicated staff time. | |
| University of Texas Health Science Center (UT) | Disclosure and apology training | Clinical faculty from 6 UT hospitals who had participated in disclosure and apology training had significantly more positive attitudes about error disclosure, and they perceived disclosure of a medical error as less damaging to patient and peer trust in them than faculty who had not participated in disclosure and apology training. | It is difficult to assess the strength of the findings because it is not clear how some of the statistical comparisons were constructed. Low response rate to the survey limits the generalizability of findings. | Conducting a disclosure culture training and administering a survey across six health institutions (four medical schools, one cancer center, and one health science center) in the UT System required significant leadership buy-in and organizational coordination. A new 51-item survey was developed to examine the relationship between patient safety culture and error disclosure. The project suggests that perceptions of an organization’s safety culture and teamwork culture change in a positive direction after participating in error disclosure training. The training did begin to stimulate culture change within the hospitals (as reported by site visit participants), and the grantee used project resources to develop tools to assess changes in attitudes over time. |
| Development of an adverse event debriefing method | Based on 62 interviews conducted using a debriefing method developed through the grant, patients and their families can provide critical information about adverse events that is not otherwise known to those analyzing the event. | Qualitative, interview-based study. No quantitative statistical analysis is reported. | This work informed the development of a debriefing tool for potential use at the UT System. The project team constructed the patient-centered interview tool, IMproving Post-event Analysis and Communication Together (IMPACT) for eliciting patient and family perspectives on their harmful events. The grantees’ research in this area added to the existing knowledge base regarding what patients want following an adverse event. Patients and families reported that litigation is sometimes undertaken as much to get information about what happened, and to get assurance that someone is addressing the problem so that no other patient is harmed in the same way, as it is to get compensation for injury. |
|
| University of Washington (UW) | Disclosure and apology coaching training | In an attitude survey, 33% of staff who participated in disclosure and apology coach training were enthusiastic about the training but lacked confidence that their organizational leaders had a shared vision around using error disclosure processes. | Grantee reports descriptive findings only; no statistical analysis is reported. No conclusions can be drawn about impact. | Overall, the project trained a large number of clinicians. Almost 400 disclosure and apology coaches were trained. |
| Team communication training | The rates of communication-sensitive adverse events (CSAEs) were examined to assess the impact of communication training. The baseline rates were generally very low, with most CSAE improvements occurring in favoring intervention sites over comparison sites. An assessment of the differences between CSAEs before vs. after communication training is forthcoming. | Grantee reports baseline descriptive findings only; no statistical analysis is reported. No conclusions about impact can be drawn. | Team communication training was successfully completed by over 1,400 participants, i.e., 71 participants fulfilled the Master TeamSTEPPS® training, 69 participants completed the train-the-trainer module, and over 1,300 providers completed the front-line clinician training. | |
| Communication and resolution program (CRP) | No conclusions can be drawn on the impact of the intervention on medical liability due to variability of tracking methods, and insufficient number of cases that advanced through the communication and resolution program. | No evidence. | Overall, this project was extremely ambitious in its project aims. A fourth project aim was to form HealthPact, a statewide multi-stakeholder collaborative to provide input on all aspects of the project. The UW HealthPact Web site (http://www.healthpact.org) offers HealthPact materials used for team communication training, disclosure coach training, and the Communication and Resolution Program (e.g., implementation, training, and evaluation material. Hospitals and other health care entities were willing to support the continuance of HealthPact activities after the project ended. |
Table 2. Selected Grantee-Reported Findings for Projects Focusing on Preventing Harm through Best Practices by Grantee Organization
| Grantee Organization | Intervention Studied | Summary of Selected Findings | Strength of the Evidence | Accomplishments |
|---|---|---|---|---|
| Ascension Health | Evidence-based obstetrics practice model | The project reported a 50% drop in injuries caused by difficulties in delivery due to shoulder dystocia (when the baby’s shoulder becomes lodged behind the mother’s pubic bone). | Confidence in findings is limited because of grantee’s use of pre/post comparison design with no control sites. Findings are descriptive, but grantees report no statistical testing to determine significance of findings. | For this project, a uniform, evidence-based obstetrics practice protocol was established for dealing with shoulder dystocia. Use of a uniform protocol is intended to eliminate variation in adherence to clinical guidelines during perinatal emergencies and translate to improved patient safety. Key project tasks included creating broad-based physician engagement and adoption of all elements of the perinatal safety initiative, adding a comprehensive labor and delivery team approach. Other elements of the intervention included training labor and delivery clinicians on electronic fetal monitoring, teamwork and communication, and documentation of unintended events. By using the principles and practices of a High-Reliability Organization (HRO), the project created the institutional culture change that was necessary for the package of interventions to take root and be sustained past the grant period. |
| Evidence-based obstetrics practice model | Six months following the end of the intervention, the 5 hospitals almost doubled their rate of reporting all adverse events in labor and delivery (increasing from 43 to 84 reports per 1,000 births). | Confidence in findings is limited because of grantee’s use of pre/post comparison design with no control sites. Findings are descriptive, but grantees report no statistical testing to determine the significance of findings. | This project also involved developing, implementing, and evaluating error disclosure through a coordinated communication intervention with patients and the care team. The project created a video on HRO principles and the effects of disclosure. It relays the story of a family whose child was injured during labor and delivery at an Ascension Health hospital and how the organization responded to the family and involved staff members. The project also trained a large number of clinicians, by tailoring training for physicians based their level of decisionmaking. As a result, all 76 clinicians serving on an Obstetrics Event Response Team in the 5 participating hospitals received training on the high-reliability root cause analysis and quick-response model. Also, 93 percent of 302 clinicians completed all the trainings offered on disclosure and resolution and documentation. |
|
| Evidence-based obstetrics practice model | Three years after establishing the new guidelines, none of the 5 hospitals had any malpractice claims based on shoulder dystocia. | Conclusions are not supported by reported data. | No additional findings. | |
| Fairview Health Services | Standardized processes, teamwork training, and performance feedback in perinatal units | Following the adoption of 3 standardized care processes by perinatal units, from 2010-2012, the proportion of deliveries with an adverse event decreased significantly for 14 participating hospitals and increased significantly for 8 comparison hospitals. | The grantee does not discuss why three of the four reported patient safety outcome measures studied failed to show significant effects from the intervention. | The PSML grant supported Phase II of the Premier Patient Safety Initiative study (2011–2012), which built on the success of Phase I (2008–2010). The demonstration did, however, face challenges with achieving the desired 100% compliance rate with the bundles due to competing demands of provider attention, a lack of active physician champions who were able to change peer behavior, and turnover of personnel. In doing so, some implementation challenges were curbed as 14 of the participating hospitals were previously involved with Phase I, and some staff members were previously exposed to the clinical interventions, teamwork training and simulation protocols. |
| Standardized processes, teamwork training, and performance feedback in perinatal units | A retrospective audit was conducted on 64 obstetric claims made against the participating hospitals. For births occurring from 2001-2012, the project reported a 19% reduction in frequency of obstetric claims per 10,000 deliveries. "Incurred" amounts (loss payments on closed claims and reserves set aside to pay pending claims) were significantly higher before the intervention ($17,908,000) than during the intervention period ($4,651,325). | Confidence in the findings is undermined by the use of a pre/post research design with no control groups. It is unclear whether the findings might have resulted from factors other than the intervention. | The grantee had access to more years of claims data due to its participation in Phase I of this study, allowing for a longitudinal analysis of project findings. Given this advantage, it is not clear why the grantee did not include the control hospitals in the final claims analyses. | |
| Massachusetts State Department of Public Health | Clinical improvement processes within ambulatory care | Sixteen primary care practices that had participated in learning Webinars and received coaching and data feedback on improving clinical processes showed significant improvement in followup of abnormal test results. This was indicated by a 65% reduction in rates of abnormal lab results or high-risk referrals where there was no documented followup, and a 54% reduction in the rate of serious potential safety risk events where potential or actual harm could occur to the patient. | Confidence in the findings is undermined because the reported findings on the outcome measures represent an uncontrolled, pre/post comparison of those measures within the intervention settings. | Up to 100 charts were reviewed at each of the 16 intervention sites, despite the multiple implementation challenges each site encountered. The project team engaged multiple stakeholders to learn about the staff and operations at each of the sites, which demonstrated the complexity of implementing such an intervention in primary care facilities. Engaging and sustaining the attention of busy practices was challenging, and technical assistance needed to be customized for each practice. The project has helped to identify the conditions under which such evidence-based programs can readily be adopted in primary care practices, as well as conditions under which their adoption becomes more difficult. |
| Clinical improvement processes within ambulatory care | Based on patient surveys, the primary care practices identified as "more engaged" in quality improvement activities showed significant improvement in 4 domains of patient experience (communication, coordination, patient-centered care, and office flow) than "less engaged" intervention practices. | The findings appear to be entirely descriptive in nature. No conclusions can be drawn about the impact of the intervention. | The grantee created a 4-page document and companion video that provides guidelines for outpatient primary care practice staff on how to communicate with patients after an error has occurred and has caused the patient harm. The tool, "When Things Go Wrong in the Ambulatory Setting," contains "tips and suggested language for communicating with the patient, and responses to frequently asked questions about how to communicate, provide an apology, and offer needed emotional support" (http://www.macrmi.info/blog/valuable-tool-when-things-go-wrong-ambulatory-setting-guideline-communication-and-resolution-outpatient-practices/#sthash.jLyop6cm.dpuf). |
Table 3. Selected Grantee-Reported Findings for Projects Focusing on Exploring Alternative Methods of Settling Claims by Grantee Organization
| Grantee Organization | Intervention Studied | Summary of Selected Findings | Strength of the Evidence | Accomplishments |
|---|---|---|---|---|
| New York State Unified Court System (NY) | Judge-directed negotiation | Thirty-two of the 326 malpractice cases against the 5 participating hospitals completed the judge-directed negotiation (JDN). Of the 32 cases, 15 were settled and 17 were voluntarily dismissed. Across these cases, the median time between filing and disposition of the case was 189 days. For 12 cases settled during JDN conference, the median settlement was $237,000, and the median time from filing to disposition of the case was 8 days. For three cases settled outside of negotiations, the median settlement was $55,000, and the median time was 240 days. | At the end of the study, the grantee reports descriptive findings only; no statistical analysis is reported. No conclusions can be drawn about the impact of the intervention. | When describing JDN programs, the research evaluators found that it "appears to be a promising model." While 326 cases had descriptive data available, 716 cases entered JDN and 165 reached a final disposition, between September 2011 and May 2015 from the 11 New York City public hospitals that participated in this project. Over the course of the 3-year implementation period, five additional judges were trained and integrated into the JDN program indicating that the negotiation curriculum was teachable. |
