Improving the Measurement of Surgical Site Infection Risk Stratification/Outcome Detection

Challenges Encountered, and Strategies for Overcoming Them

Below we note specific challenges encountered in executing our work plan as planned and those strategies used to overcome these.

November 2009

Task 2: The VA system relies on CPT codes for administrative coding whereas other systems use ICD-9 codes.

A mapping of ICD-9/CPT codes for the specific procedures being evaluated in the project was done to allow for a fully representative sample of surgeries at the VA system.

February 2010

Task 2 & 3: Craig Gale left Intermountain

Jef Huntington, from Intermountain, joined the project team for the analytic work of tasks 2 and 3.

March 2010

Task 3: Russ Staheli left Intermountain

Jef Huntington was included to coordinate the data pulls for task 3.

April 2010

Task 3: After completing the master risk factor list, many identified variables were dependent upon the definition of the variable, e.g. chronic diseases had to be clearly defined.

An SSI risk comorbidity table was created using the ICD-9/CPT map along with other identified factors ICD-9 codes to standardize the definitions.

June 2010

Task 3: Certain identified SSI risk factors were reevaluated for the data collection process, e.g. Anemia was identified as a risk factor and was to be recorded as a yes/no variable.

Instead of initially defining conditions and providing a yes/no value, it was decided that measured values would be more useful. In the case of anemia, hemoglobin levels were to be recorded—from which a set definition of Anemia could be derived.

July 2010

Task 2: While algorithm development and testing was planned to use SLC VAMC and Intermountain NSPQIP data, the data was only available at SLC VAMC.

It was decided that National VA NSQIP data could be used to develop and test the algorithm, then validated at the other systems.

Task 3: The risk factor data collection at each site was delayed due to continued refining of the master risk factor list.

As data collection started, Intermountain did a provisional analysis of about 20,000 patients to determine collection reliability. All sites recorded the data collection process and noted any elements that were unreliable or difficult to obtain. The master risk factor list was updated as necessary to include reliable and obtainable elements.

October 2010

Task 1: A no-cost extension was considered to use funds for travel and conference attendance to present the work of the project after the project end date.

The no cost extension was denied. However, the possibility of paying for expenses associated with the dissemination plan prior to the project end date was considered as a possible option and required further review from AHRQ Contract Officers.

Task 3: As the data pulls progressed across the systems it was realized that the data sources varied between each system.

In order to aid the implementation of the tool at other systems, it was decided to record, with detailed specificity, where the data were found at each organization.

Task 4: Given the delay of conducting the nursing focus group as originally planned, a repurposed focus group strategy was proposed.

Repurposed focus groups were submitted for approval and approved to conduct 2 separate groups to solicit input from key stakeholders on adoption and implementation as well as developing use cases for the e-detection surveillance tool.

January 2011

Task 2: With IRB/Privacy Board requirements, data could not be sent to SLC VAMC for validation of the algorithm.

The algorithm was sent to each organization to test and validate with their data where chart reviews will be performed on all positives produced by the algorithm.

Return to Contents

Lessons Learned and Recommendations for Next Steps

There are several lessons to be learned from this work, summarized in the following bullet points:

Task 1:

• In-person meetings among project team members, held in addition to regularly scheduled teleconferences, confirmed our expectations of their value for promoting teambuilding and collaboration among geographically distributed team members.

Task 2:

• The most appropriate use of automated systems, whether alone or in combination with manual surveillance, will take careful consideration of the purpose and requirements of the events being surveilled. The performance of automated systems may vary, particularly when attempting to detect events that occur in the outpatient setting where differences in data availability may be pronounced. More work is necessary to improve the discriminability index of electronic algorithms, but allowing IP to select rules that suit their own needs may be a reasonable measure in the interim.
• Current estimates for national rates of SSI are unknown. Our smaller group of hospitals is only informative in a very limited way. More accurate estimates would require sophisticated patient case mix adjustment and a much larger sampling of hospitals.
• There may be value in exploring natural language programming (NLP) and what could be added from text notes. A new study shows that NLP will be more beneficial for the electronic identification of hospital-acquired wounds than bacteremias, UTIs, respiratory infections, etc. (publication in progress).

Task 3:

• Postdischarge surveillance remains a challenge, requiring data from the full continuum of care (inpatient and outpatient). Postop admission within 30 days is a common “trigger” used by many facilities that rely on manual SSI surveillance for postdischarge infection identification. Integrated inpatient/outpatient medical records are expected to have more utility for electronic algorithms.
• No single set of risk factors that can be used to predict SSI across all types of surgical procedures or facilities. This study found that SSI risk factors are dependent on the type of surgical procedure. Thus, SSI rate comparison needs to be at the surgical procedure level and not the surgical service level, i.e., orthopedics, general surgery, thoracic, etc. In addition, SSI rates should also be compared at the facility level against its own baseline rates.

Task 4:

• An extreme burden of unsupported practitioner SSI surveillance exists.
• There is documented need for enhanced risk factor assessment (surgeon focus group) and receptivity toward cognitive support from an electronic surveillance tool (nurse focus groups).

Task 5:

• An analysis of DH publicly reported data revealed SIR was superior than the NHSN basic risk index in predicting SSI risk. The SIR uses logistic regression modeling and takes into account more variables and procedure-specific risk factors. We hypothesize that improved risk adjustment is due to consideration of these extra risk factors. Although superior, the SIR still is not broadly applicable to all procedures and settings. Based on this, we are now looking at surgery specific risk factor assessment, accounting for differential impact of, e.g., smoking on infection risk for hernia vs. CABG, etc. We need to investigate risk factors as surgery specific. Now with more publicly reported data we can use that to get the needed large numbers.

Several recommendations for next steps have emerged from our work. These are:

1. Algorithms should be validated at each system to which they will be introduced, in a manner akin to the quality assurance policies regarding new laboratory equipment.
2. Algorithms will need to be trained and validated on even broader scales that demonstrate more variation in clinical practice and electronic systems.
3. Validate risk factors more broadly on a national scale.
4. We should investigate ways to develop electronic algorithms that might search multiple inpatient and/or outpatient networks to help ascertain postdischarge SSIs.
5. We need further validation of SIR on publicly reported data; perhaps validate on subgroups of settings (public health safety net, academic, community, etc.).
6. Explore how risk factors may be more or less relevant to specific procedures (e.g., risk of smoking on herniorraphies vs. CABG.
7. National estimates of SSI should not be pursued without larger datasets that are representative of the variation among the nation's hospitals.
8. We should explore natural language processing (NLP) methodology to extract more information from text notes. For instance, a recent study showed that NLP could identify a number of postoperative surgical complications in the Veterans Health Administration⁷⁷.
9. We should consider exploration of more sophisticated decision support methods that deliver the probability of SSI and/or important nuance information instead of binary yes/no information (which loses much of the original information content).