The primary purpose of this task was to design and test methods to risk-stratify surgical patients for surgical site infection. We began by clarifying conditions and data availability for analyses. Exhibit 22 below summarizes data availability by facility/system. It can be seen that data were available for all target procedures across partner facilities except CABG data that were not available for Denver Health or Vail Valley Medical Center.
Exhibit 22. Availability of data by procedure and facility/system
| Procedure | Denver Health | Intermountain | SLC VAMC | Vail |
|---|---|---|---|---|
| CABG | NA | √ | √ | NA |
| Herniorrhaphies | √ | √ | √ | √ |
| Hip prothesis | √ | √ | √ | √ |
| Knee prothesis | √ | √ | √ | √ |
CABG = cardiac artery bypass grafting, NA = not applicable, SLC VAMC = Salt Lake City Veterans Affairs Medical Center
Subtask 3.1. Identify Strong Predictors of SSI, Particularly Important Variables Not Currently Used in Mainstream Risk-stratification Methods
The list of potential risk factors for surgical site infections was developed with a two-tier process:
An active surgeon on the study team used his extensive experience with SSIs and a previous list of risk factors used by his institution to identify potential risk factors. That initial list consisted of 88 risk factors for SSI (go to Appendix I). This list was then used in a focus group to solicit input with surgeons (go to Chapter 4).
An extensive literature review was also performed using Internet search engines (including PubMed and Google Scholar) to identify any published risk factors for SSIs at any site. All English-language publications for the previous 10 years were included. Keywords used for the search included: SSI, surgical site infection, surgical risk factor, risk factor, surgical wound, surgical infection. Risk factors identified from any surgical site were included in the list. From that search, 24 additional potential risk factors were included in the Master Risk Factor table (go to Appendix J). Each of the potential risk factors in the list was then clinically reviewed and categorized as modifiable or nonmodifiable.
Subtask 3.2. Develop a Risk-adjustment Method that Utilizes the Identified Risk Variables to Validly Compare Rates of SSI Across Facilities
The Master List of all identified potential risk factors (go to Appendix K) was sent to each of the four study sites—Intermountain Healthcare, Denver Health, Salt Lake VA, and Vail Valley Medical Center. Each site examined the Master List and determined if it had electronic access to each of the individual risk factors. Each site then returned its marked list to Intermountain Healthcare where their site-specific information was added to the Master List. After that information was collected from all four sites, a union set of 34 potential risk factors was identified. From that union set, a new list of risk factors common to all four sites was created (go to Appendix L; also, Appendix M compares the initial and final lists of risk factors). Each of those risk factors was then further defined to remove any ambiguity between the study sites, ensuring identical collection and reporting. The data values for each risk factor, their description, and type were agreed upon via conference calls and email. Based on that process, a final data collection spreadsheet was created in Microsoft Excel® and sent to each of the four study sites. Each site then met with their data access colleagues to plan how the data would be collected, and made sure both groups had the same definitions of the needed risk factors. Each site developed programs based on their specific data-retrieval needs to access and collect the data elements needed for each risk factor.
Each site then collected their data for patients older than 18 years of age, using unique patient identifiers for patients who had CABG, herniorrhaphies, hip arthroplasty, and knee arthroplasty. Intermountain also collected data for appendectomies and added that data to their final dataset.
Each patient in the study group from each of the four sites was then identified as having or not having an SSI, based on the specific surgical procedure, and marked accordingly in the spreadsheet. SSI data was collected based on the reporting site's specific collection method, with SSIs defined using NNIS criteria from CDC. A random sample of patients and identifiers were manually selected to verify that data access was correct at each site. Each site then deidentified the data and submitted their final spreadsheet to Intermountain Healthcare. Mappings to actual patient identifiers were kept behind the firewalls at each of the study sites.
All four sites then examined each of the risk factors they had electronic access to and documented the original data source for each. Exhibit 23 denotes data sources for the 33 risk factors at each of the sites. As is often the case during the actual data collection process, some of the risk factors contained in the common list were found not to be stored (or only stored occasionally) in the databases at the different sites. For example, at Intermountain, the American Society of Anesthesiologists ASA score is a data element in the surgery database, but we found it was only populated 17 percent of the time. This was often the case at the other sites also. Thus, ASA score was not included in the final list of 33 common risk factors used in the analysis (go to Appendix L). While each of the four sites had access to the common list of risk factors, each site often collected that data from different clinical departments or databases. This information should help other facilities to determine where they may find these risk factors at their institutions. At Intermountain, eight additional risk factors electronically found in the database and not included in the common list were also included in a second spreadsheet (Exhibit 24). Other sites also identified some unique risk factors not found at the other participating sites, but that were not included in this study.
Description of Intermountain data. All of the data from Intermountain Healthcare was collected from data contained in the Enterprise Data Warehouse (EDW), which resides on an Oracle® relational database. All clinical data from the EMR, surgery database, hospital-acquired infection database and other databases contained in the Intermountain EMR are loaded into the EDW each night. The EDW contains 35,000,000,000 records and 8 Terabytes of data. Each of the common risk factors was collected using SQL queries on specific tables or the union of multiple tables. All Intermountain patients have an enterprise[-wide identification] number that is consistent for all encounters at any of the 22 hospitals and over 100 InstaCare facilities, clinics, and physician offices. That number was used to link all patient specific data. Surgery data was queried first to identify all patients undergoing the study procedures during the study period. Each of those patients was then checked for SSIs linked by the date of the surgery. All the other data elements listed in the common data list (and the eight other data elements available to Intermountain only) were then collected, based on the definitions included in the common list. After the data was then checked and verified, the final study database was loaded and sent for statistical analysis.
Description of Denver Health data. Data were available in the warehouse. The many ancillary services housed in this single database include lab, radiology, pharmacy, scheduling, and surgery. Along with the Web Portal, information in the data warehouse can be accessed through Crystal Reports, Executive View, and Microsoft Analysis Services, as well as by using other tools. These data were collected using Statistical Analysis Software® (SAS®) Enterprise guide, version 4.2. DH data were limited to total knee (TK) and total hip (TH) replacements, and herniorrhaphies (HE), as CABGs are not performed at this location. These data were found in utilization, lab, surgery, and pharmacy electronic repositories. Other variables were found in NHSN reports. Denver Health has a data warehouse with a unique patient identifier and unique episode identifiers used to link data across systems. There are some limitations to DH data. Surgeon experience is limited to the number of years practicing at DH. There are missing datapoints, including 616 for ASA and 427 for surgeon's experience. We were unable to locate data on antibiotics discontinued within 24 hours. There were three out-of-range values found for preop stay length (less than 0 days) and six out-of-range values for surgery duration (less than 0 minutes). Up to 15 diagnosis codes were available per surgery, and up to 10 procedure codes. The limitation on procedure codes is not a large problem, though, as only seven surgeries (0.5 percent) had 10 procedure codes.
Description of Vail data. Data are available via an EMR system (CERNER). This data is accessed via either chart review or through prebuilt reports that require a coded program. The system allows unlimited diagnosis and procedure codes for each surgery. Data was collected using Excel®. VVMC data were limited to total knee (TK) and total hip (TH) replacements and herniorrhaphies (HE), as CABGs are not performed at this institution. These data were found in demographics, ADT, Surgery, Nursing, ICD-9, Microbiology, Laboratory, and Pharmacy modules of our EMR.
There were two limitations to the VVMC data: first, surgeon experience was unable to be collected; and second, the algorithm was run without including postdischarge antibiotic prescription data for a large majority of patients who return home for postdischarge care.
Description of SLC VAMC data. The VA Salt Lake City Health Care System Data Mart is a compilation of operational data designed to extend the utilization of the clinical and administrative systems. The data mart is comprised of a collection of databases storing data from The Veterans Health Information Systems and Technology Architecture (VistA) and other data sources. VistA is an enterprise-wide information system built around an EHR. Data are stored in a relational database. Targeted patient population was selected using both ICD-9 codes and CPT codes (total knee [TK], and total hip [TH] replacements, herniorrhaphies HE], and CABG). Multiple VA data sources are merged and cohorts are definable by attributes such as ICD-9 codes and CPT codes from both inpatient and outpatient encounters within a target time period. These data are kept current by frequent updates with new data from the source databases, so that timely data are available for research. Additionally, surgical outcome data were obtained from VASQIP. VASQIP data represents an extensive surgical quality improvement program and data collection tool. Comprehensive selections of approximately 69 clinical variables are collected for each case in this option. The dataset contains a broad range of variables that can be used for research purposes, as well as for identifying opportunities for surgical process improvement and other quality improvement efforts.
Data collection summary:
A total of 3,612 herniorrhaphies, 3,410 total hip and 9,728 total knee procedures were included in the study using Intermountain, Vail Valley Medical Center, VAMC, and Denver Health. An additional 1,802 CABG and 5,873 appendectomy procedures were submitted from Intermountain and the VAMC (Exhibit 25). A total of 222 SSIs were associated with the various surgical procedures and participating facilities (Exhibit 26). The SSI rates varied by reporting site and procedure each year, and ranged from 0.0 to 7.1 percent.
Subtask 3.3. Evaluate the Quality of the Risk-Adjustment Relative to Method Complexity and Data Collection Costs
We met with a statistician and selected appropriate statistical tests to identify the risk factors for SSI. The dataset was checked by the statistician for any obvious coding problems or other issues that would complicate or confound the analysis. The statistician also verified that the dataset was formatted correctly so that it could be loaded into the statistical software for analyses. After the data was loaded into the statistical program, it was then further cleaned and any missing or incomplete data was resolved.
After combining the data from each health care system, we found numerous missing lab values (Exhibit 27), especially for the outpatient procedures. We found that the variables preop_hematocrit, preop_hemoglob, preop_albumin, postop_hemoglob, and postop_heatocrit were most often missing because those tests were not ordered prior to the patient's procedure (Exhibit 28). An analysis of lab values 30 days prior to surgery showed that 99.7 percent of lab values were captured within 30 days preop, and verified that the missing values were indeed the result of the tests not being ordered prior to the procedure. However, dropping an entire record due to missing values can result in undesired outcomes and misleading results,73, 74 therefore we decided to impute the missing values. We considered a bootstrapping approach, but ultimately chose to use a multiple imputation (MI) method to approximate missing data.
In multiple imputation, missing values for any variable are predicted using existing values from other variables. The predicted values, called “imputes”, are substituted for the missing values, resulting in a full dataset called an “imputed dataset.” This process is performed multiple times, producing multiple imputed datasets (hence the term “multiple imputation”). Standard statistical analysis is carried out on each imputed dataset, producing multiple analysis results. These analysis results are then combined to produce one overall analysis.
Multiple imputation accounts for missing data by restoring not only the natural variability in the missing data, but also by incorporating the uncertainty caused by estimating missing data. Maintaining the original variability of the missing data is done by creating imputed values, which are based on variables correlated with the missing data and causes of data being missing. Uncertainty is accounted for by creating different versions of the missing data and observing the variability between imputed datasets. It is important to note that imputed values produced from an imputation model are not intended to be “guesses” as to what a particular missing value might be; rather, this modeling is intended to create an imputed dataset that maintains the overall variability in the population while preserving relationships with other variables. Thus, in performing multiple imputation, a researcher is interested in preserving important characteristics of the dataset as a whole (e.g., means, variances, regression parameters). Creating imputes is merely a mechanism to deliver an analysis that makes use of all possible information.
New to SAS® version 9† is the Multiple Imputation (MI) procedure,75 which uses a random sample of missing values to account for any uncertainty from the missing data. For our project, we chose a Markov Chain Monte Carlo method to create five complete datasets, each with a slightly different value in the missing slot. We then used standard statistical analyses on the complete datasets. Also new to SAS 9 is the MIANALYZE procedure that we used to combine the five dataset-analysis results into a single inferential result.
Our next step was to randomly select 60 percent of the data to be placed in a derivation environment where we could develop the statistical models. The remaining 40 percent of the data was placed in the validation dataset, which was used for comparison and confirmation of the models built with the derivation dataset. After validation, the final statistical models used the full dataset that included both the derivation and validation datasets.
Univariate regression was used to determine the independent association of potential risk factors and SSI. The final model included risk factors with probability P < 0.05, or that contributed to the predictive value of the model. We first used a binary logistic regression model to evaluate the relationship of each variable with the occurrence of an SSI. For the nonimputed variables, we used the original dataset with logistic regression. For the imputed variables, we used the five imputed datasets with a combination of logistic regression and the MIANALYSE procedure to generate results.
The risk factors for SSI were tested by including the type of procedure as a binary variable (yes/no) and the risk factors were also independently tested for each of the five different procedures (CABG, herniorrhaphies, hip arthroplasty, knee arthroplasty and appendectomies). The predictive models were created with stepwise logistic regression, using the five imputed datasets. The entry level probability for each variable was set at .2 and the probability used to keep a variable in the model was set at .25. Running logistic regression on five separate datasets resulted in five different candidate models that were mostly the same, but had a few differences. We then took any variable identified in the five multivariate models to create the final logistic regression model. We then ran the final logistic regression model on the five imputed datasets and used the MIANALYZE procedure to produce one set of results from the five iterations.
Subtask 3.4. Identify SSI Risk Factors using combined datasets from all four facilities/systems.
3.4.1. Multivariate analysis of the datasets, including each procedure as a binary variable.
During the univariate analyses of the derivation dataset, 13 different risk factors were included in the model. That analysis also included each of the five different procedures as a binary variable, yes/no. Each of those 13 risk factors was then included in three different logistic regression analyses using the derivation, validation, and combined datasets (Exhibit 29, Exhibit 30, and Exhibit 31). The statistical significance of each of the 13 potential risk factors changed during each test using the three different datasets. For the derivation dataset, 6 of the 13 univariate risk factors remained significant in the model compared to only 3 in the validation set, and 7 when both the derivation and validation sets were combined. Only a history of MRSA infection and a postoperative admission within 30 days were significant in all three tests. In many cases, postoperative admission was indicative for admission due to a postoperative wound. Chronic kidney disease was significant in the derivation and combined datasets, along with an increase in the number of procedures and a low postoperative hematocrit. Male gender was only significant in the derivation analysis. CABG surgery was only found to be significant in the validation and combined datasets. A longer preoperative stay was only significant in the combined dataset.
3.4.2. Multivariate analysis of the datasets including only CABG surgery
During the univariate analyses of the derivation dataset for CABG surgeries, seven different risk factors were included in the model. Each of those seven risk factors was then included in three different logistic regression analyses using the 60-percent derivation, 40-percent validation, and 100-percent combined datasets (Exhibits 32–34). The probability of each of the seven potential risk factors changed for each of the three different datasets. For the derivation dataset, five (transfer admission, increased BMI, history of MRSA, postoperative admission within 30 days, and longer surgery duration) of the seven univariate risk factors remained significant in the model, while only one (increased BMI) remained in the validation set. For the combined dataset, except for transfer admission, the same four factors identified using the derivation set remained significant. Of interest, postoperative admission within 30 days had a nonsignificant probability of 0.2134 in the validation set.
3.4.3. Multivariate analysis of the datasets, including only herniorrhaphy
During the univariate analyses of the derivation dataset for CABG surgeries, seven different risk factors were included in the model. Each of those seven risk factors was then included in three different logistic regression analyses using a 60-percent derivation, 40-percent validation and 100-percent combined datasets (Exhibits 35–37). The probability of each of the seven potential risk factors changed for each of the three different datasets. For the derivation dataset, three factors (postoperative admission within 30 days, postoperative hematocrit, and postoperative hemoglobin) remained significant in the model, and only postoperative admission remained in the validation and the combined sets. Of interest, postoperative admission within 30 days had a nonsignificant probability of 0.2134 in the validation set.
3.4.4. Multivariate analysis of the datasets including only total hip surgery
During the univariate analyses of the derivation dataset for total hip surgeries, eight different risk factors were included in the model. Each of those eight risk factors was then included in three different logistic regression analyses using 60-percent derivation, 40-percent validation and 100-percent combined datasets (Exhibits 38–40). The probability of each of the eight potential risk factors changed for each of the three different datasets. For the derivation dataset, four factors (emergency surgery, number of procedures, postoperative admission within 30 days, surgery duration) remained significant in the model. Chronic lung disease, emergency surgery, and postoperative admission were significant in the validation set, and the probability related to surgery duration was 0.05866. For the combined dataset, the same three factors as for the validation set were significant, plus surgery duration. Because the stepwise logistic regression entry and stay probabilities were set at .20 and .25, respectively, it was possible for some variables to be nonsignificant in the derivation dataset and significant in the validation and combined datasets (chronic lung disease, for example).
3.4.5. Multivariate analysis of the datasets including only total knee surgery.
During the univariate analyses of the derivation dataset for total knee surgeries, only five different risk factors were included in the model. Each of those five risk factors was then included in three different logistic regression analyses using a 60 percent derivation, 40 percent validation and 100 percent combined datasets (Exhibit 41, Exhibit 42, and Exhibit 43). The probability of each of the five potential risk factors changed for each of the three different datasets. For the derivation dataset, three univariate risk factors (history of MRSA, number of procedures, and postoperative admission within 30 days) remained significant in the model. Three of the five (history of MRSA, postoperative admission within 30 days, and preop hematocrit) were significant in the validation set. When the five univariate risk factors were tested with the combined dataset, all five (including male gender) were significant.
3.4.6. Multivariate analysis of the dataset including only appendectomy surgery at Intermountain Healthcare.
During the univariate analyses of the derivation dataset for appendectomy surgeries, seven different risk factors were included in the model. Each of those seven risk factors was then included in three different logistic regression analyses using the derivation, validation, and combined datasets (Exhibits 44–46). The probability of each of the seven potential risk factors changed during each test, using the three different datasets. For the derivation dataset, only two (postoperative admission within 30 days and postoperative hematocrit) of the seven univariate risk factors remained significant in the model, and only one (postoperative admission) remained in the validation set. For the combined datasets, the same two as the derivation set remained significant. Again, only postoperative admission within 30 days was significant in all three tests.
Subtask 3.6. Summary
3.6.1. Identified SSI risk factors using datasets from all four facilities/systems
This study examined a large number of potential risk factors contained in electronic medical records from four different facilities/systems in an effort to identify predictors for SSI. The potential risk factors were identified through surgeons' experience and an extensive literature review for any potential risk factors for any surgical procedure during the past 10 years. Four different surgical procedures (herniorrhaphy, CABG, total hip, total knee) were included in this study from all four facilities/systems, and appendectomy was also included from Intermountain Healthcare. Analyses of six different data partitions were conducted which included all data and each of the different surgical procedures as a potential risk factor, and then data from each surgical procedure separately. For each of the different data partitions, three iterations were conducted which included a 60-percent random derivation set, the remaining 40-percent validation set, and then the combined dataset.
During the analyses of the three iterations that included the different surgical procedures as binary risk factors and the three iterations for the individual tests of each of the five different surgical procedures, 21 (64 percent) of the 33 potential risk factors tested were identified during the univariate analyses during at least one iteration. However, only 13 of the 21 were found to be significant in at least one of the derivation, validation, or combined multivariate logistic regression models. While herniorrhaphy was included as a risk factor during the univariate analysis that included surgery type as a binary variable, only CABG surgery was statistically significant in the validation and combined regression models. The most common risk factor (identified during 16 of the 18 different iterations) was postoperative admission within 30 days. Rather than a preoperative risk factor, this finding was indicative of the need for hospitalization for postoperative wound treatment. One might ask why was postop admission within 30 days included in the analysis for this study. First, it is a common risk factor or “trigger” used by many facilities that rely on manual SSI surveillance, including postdischarge phone calls to patients, for postdischarge SSI identification. This study shows that it is still a reliable method to identify SSIs. Second, since it was the most common risk factor identified in this study, that led us to feel the statistical analysis was working appropriately. If postoperative admission was not identified as a risk factor, it would have immediately led us to question our results and methods. The next most common risk factor was history of MRSA (identified seven times) followed by postop hematocrit (six times), number of procedures (five times), surgery duration (four times), and increased BMI and postop hemoglobin at three times each. It was interesting to note that the significant risk factors almost always varied between the derivation, validation, and combined datasets.
This study indicates that risk factors for surgical site infections vary by the type of procedure. The risk models generated for each of the five different types of surgical procedure and the inclusion of procedure type varied. The number of risk factors included in the different models also varied from 5 to 13 in this study. We also found that the risk factors for total hip surgery differed from those for total knee surgery. Thus, total hip surgeries need to be compared with total hip surgeries, and not just the group of other orthopedic surgeries.
3.6.2. Identified SSI risk factors, using data only from Intermountain Healthcare, and including eight additional potential risk factors
The Intermountain data contributed 87 percent of the total dataset—including 57 percent of the data for herniorrhaphy, 85 percent of the total hip replacement data, 90 percent or the total knee replacement data, 95 percent of the CABG data, and all of the appendectomy data. The analysis with the Intermountain data only demonstrated the impact the other data had when included in the total dataset. It also reinforced the conclusion that risk factors can change, depending on the facilities data. As before, postop admission was the most common factor identified and was significant in all 18 iterations, followed again by history of MRSA (significant nine times). This was possibly influenced by Intermountain's ability to monitor postop admissions across all 22 hospitals and its enterprise-wide MRSA surveillance network that monitors all MRSA patient movement throughout the system. When each of the five surgical types was included in the analyses, only herniorrhaphy was significantly associated with SSI. While herniorrhaphy was identified in the univariate analysis of the total dataset, it was not significant in either the derivation or validation analyses, whereas CABG was significant in both the validation and combined analyses. Thus, the 13 percent of the data from the other facilities did have an impact on the risk factors identified by the univariate analyses and the significance of the risk factors in the multiple regression analyses.
The inclusion of the eight new, potential risk factors in the analysis of only Intermountain data did not result in any major differences in the identification of risk factors. Charlson score, number of surgeons, and preop glucose were the only new risk factors that were identified in any of the univariate analyses, and only Charlson score and number of surgeons were ever found to be significant in the multiple regression. The number of procedures during the same surgery (concurrent procedures) was identified as a significant risk factor from the total dataset. It was significant during the derivation and combined analyses, which included the five surgical procedures as risk factors. The number of procedures was not identified in the univariate analysis using the Intermountain data only, and seems to have been replaced by the number of surgeons. The number of different surgeons would seem to be associated with the number of different procedures performed, and probably kept the number of procedures from remaining in the model.
3.6.3. Conclusion.
There is not a single set of risk factors that can be used to predict SSI across all types of surgical procedures or facilities. This study found that SSI risk factors are dependent on the type of surgical procedure. Thus, SSI rate comparison needs to be at the surgical procedure level and not the surgical service level (i.e., orthopedics, general surgery, thoracic, etc.). In addition, SSI rates should also be compared at the facility level against its own baseline rates. When compared at that level, the sets of risk factors identified for the five different surgical procedures in this study could be used to identify changes in SSI rates over time, and differences between surgeons.
† The data analysis for this paper was generated using SAS/STAT software, Version 9.2 of the SAS System for Windows. Copyright © 2002-2008 SAS Institute Inc. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc., Cary, NC, USA
