In this chapter, we highlight the strengths and limitations of the current study and discuss some recommendations and next steps for AHRQ to consider.
Study Strengths and Limitations
ST-PRA is a valuable and important tool for assessing the risks associated with the occurrence of many different patient safety events in a variety of different health care contexts. In this study, we were able to use this novel approach to identify the contributors to the occurrence of SSIs in the ASC environment. The value of ST-PRA lies in the capacity to consider both individual contributors of risk, as well as unique combinations of risks that contribute to the adverse outcome. By including both quantitative and qualitative data into the models, a real-world experience can be created and tested using the sensitivity analysis methodology, allowing the user to maintain the scientific integrity of the tool. Finally, the ST-PRA model also serves as a living document that can continue to be modified over time as new risk information is acquired, either through direct observation or improved methods for studying the ASC environment.
Despite these important strengths, notable limitations of this study should be acknowledged. First, the quantitative estimates from the datasets were limited, because these data failed to include the more granular estimates of risk that are important for creating the risk models. As improvements in the SASD, NIS, and the SEDD databases occur, additional information regarding the context of care come into focus and can be used to further refine this research. Second, the lack of integrated data systems, linking patients between the ASC, emergency department, and inpatient settings, significantly limited the ability to inform the model with real risk estimates. Until these linkages can be established, investigators studying care in the ASC setting will continually face the challenge of "loss to followup." Patients tend to seek care for surgical complications in emergency departments, inpatient settings, or their primary care physicians' offices, and the attribution of the complication will not return to the ASC where it originated.
Finally, although a typical limitation of modeling is that the resulting models do not create a real-world picture of what it is intended to portray, this was not a limitation of the current study, as discussed previously. That is, the use of quantitative estimates from the literature and the modeling of the in vivo process flows contributed to a real life understanding of the system under study when combined with the sensitivity analyses, which ensured that the risk estimates and conclusions were supported across a range of values.
Next Steps
The use of ST-PRA as a modeling tool to identify risks in the ASC environment is an important outcome of this work. This model can be refined as new information becomes available in the literature and as improvements in care in the ASC environment are realized through interventions like those proposed in this report. AHRQ should consider the following suggestions for next steps to continue this work, including developing the intervention, conducting a followup study to determine the impact of the intervention, developing an integrated database to track patients across care settings, and examining ways to make the ST-PRA methodology more accessible.
Developing the Intervention
One of the most important next steps of this work is to develop the intervention, as proposed in this report. We recommend developing the intervention as components and offering them in modular format, so that ASCs can select and implement the components in the order that best suits their needs. Depending upon the intervention selected, efforts aimed at understanding the critical process steps necessary for its implementation will be next. For example, if the intervention related to antibacterial administration is selected, providers will need to understand the expected performance steps that they need to include in their process and then field test these steps prior to implementation, to ensure that performance and measures of performance can be established. Intervention implementation can occur with or without regulatory oversight; however, the ability to hold providers accountable for performance in these settings over time is most likely to occur with the establishment of new regulations.
Once the performance steps are identified, education needs to occur, with feedback that addresses the barriers to implementation. The use of simulation exercises can be beneficial in this phase of the process. For example, rather than testing the intervention in vivo, the team could use microsimulation techniques and test the intervention in vitro on a desktop simulator, to see if the appropriate outcomes from the process change are achieved.
Once developed, we recommend conducting a pilot test and usability study on the intervention, prior to its introduction to the ASCs. In particular, we suggest that any checklists, training programs, and infection control practices be examined to ensure that they can be used properly, effectively, and efficiently by staff and to identify ways to improve the functioning of these elements of the intervention. In addition, this study may also examine the effectiveness of each module of the intervention as a stand-alone, or in combination with the other modules that comprise this intervention.
Conducting a Followup Impact Study
An essential feature of ST-PRA modeling is that it is a living document. As new information becomes available, either in the literature or the public domain, risk estimates can be updated and new cut sets can be generated. This provides the ability to assess the impact based upon already completed work. Once the intervention and its components have been developed and implemented, followup studies should be conducted to determine the impact that these efforts have had on improving infection control practices and communications, and ultimately on the reduction of SSIs. For example, the model can be modified to include new probabilities associated with compliance with new processes and procedures, and the corresponding impact on the top-level event could be examined.
Developing an Integrated Database
As highlighted in our discussion of data sources, a major challenge to understanding the care provided in the ASC environment is that the data sources are not integrated. A benefit of integrating the data sources is that the potential costs associated with SSIs in the ASC environment may be more appropriately analyzed with a single data source, because the care received by a CMS dependent across practice settings is included in one place. Specifically, we recommend that the data sources include a method for tracking patients across practice settings, for example through a common patient identifier field.
Examining How to Make ST-PRA More Accessible
Over the last 10 years, there have been a variety of methods and tools designed to improve our understanding of the many risks in health care, including root cause analysis (RCA) and failure modes effects analysis. Despite the widespread use of these tools, they tend to be applied inconsistently, often with little to no expertise. An important limitation to these tools is that they are dependent upon the people who come together to analyze the event. For example, health care workers learn these strategies on the job, in the setting where a serious or sentinel event has occurred. Often these settings are emotionally charged, because a patient has experienced a severe injury at the hands of the providers, and the providers often cope by protecting and defending their own practice rather than taking the opportunity to learn through using the RCA tool.
As discussed, ST-PRA adds additional value over the other existing risk assessment tools. However, one of its major limitations at this time is that it is still restricted to the research domain, because the current fault tree software available for these analyses, Relex, is difficult to use and not understood by health care quality improvement teams. Until probability and fault tree analyses can be performed using readily available software tools, ST-PRA will remain out of reach of health care providers. We recommend that AHRQ investigate ways in which ST-PRA can be made more accessible to and easier to use by health care providers as they attempt to improve system design and reduce the risks associated with the delivery of health care. Examples of different ways this may be accomplished include modifying the ST-PRA tool as:
- An add-in to a readily available software program such as Microsoft Excel®, to simplify the modeling process.
- A series of questions and answers to frame the logic in a base case fault tree and allow it to be adapted to the local context.
- A toolkit of core fault trees for important safety problems such as medication errors, serious and sentinel events, and central line-associated blood stream infections, which includes instructions on how to modify the core content in the risk assessment to account for differences in the local context.
In summary, the results of this study offer several directions for future AHRQ work, including the development of:
- The proposed intervention that focuses on building skills and networks among health care providers.
- Additional studies such as usability, pilot testing, and followup impact.
- Better methods to track patients across provider sites.
- The development of more accessible tools that enable health care provider organizations better ways to examine other high-risk, low-base-rate events, and ultimately to improve the quality of care being delivered.
