A Practical Approach to Predicting Surgical Site Infection Risk Among Patients Before Leaving the Operating Room

A surgical site infection (SSI) prediction model that identifies at-risk patients before leaving the operating room can support efforts to improve patient safety. In this study, eight pre-operative and five perioperative patient- and procedure-specific characteristics were tested with two scoring algorithms: 1) count of positive factors (manual), and 2) logistic regression model (automated). Models were developed and validated using data from 3,440 general and oncologic surgical patients. In the automated algorithm, two pre-operative (procedure urgency, odds ratio [OR]: 1.7; and antibiotic administration >2 hours before incision, OR: 1.6) and three intraoperative risk factors (open surgery [OR: 3.7], high-risk procedure [OR: 3.5], and operative time OR: [2.6]) were associated with SSI risk. The manual score achieved an area under the curve (AUC) of 0.831 and the automated algorithm achieved AUC of 0.868. Open surgery had the greatest impact on prediction, followed by procedure risk, operative time, and procedure urgency. At 80% sensitivity, the manual and automated scores achieved a positive predictive value of 16.3% and 22.0%, respectively. Both the manual and automated SSI risk prediction algorithms accurately identified at-risk populations. Use of either model before the patient leaves the operating room can provide the clinical team with evidence-based guidance to consider proactive intervention to prevent SSIs.


Introduction
Surgical site infections (SSIs) are the most common post-surgical complication, and in the United States, 2-4% of inpatient surgical procedures are associated with SSIs and are the leading cause of unplanned hospital readmissions among surgical patients [1]. SSIs represent an estimated $3.3 billion in US healthcare costs [2]. While substantial reductions in SSIs have been achieved through infection controls, sterilization methods, maintaining normothermia, perioperative glycemic control, and antimicrobial prophylaxis [2,3], SSIs continue to be a major source of morbidity and cost.
Identifying patients at high-risk for SSIs is an important component of SSI reduction programs. Identifying high-risk SSI patients can support surgical consent, the application of intensive prevention measures, and increased post-operative monitoring. Accurate SSI risk prediction can also enable risk-adjusted SSI comparative performance reporting for the purpose of transparency and improvement [4].
While there are many published procedure-specific SSI risk calculators, those intended to be used across surgical procedures include the Surgical Site Infection Risk Score (SSIRS) [4], JSS-SSI Risk Scoring Tool [5], and the American College of Surgeons -National Surgical Quality Improvement Program (ACS-NSQIP) risk score [6]. The SSIRS is composed of 13 items and has a reported area under the curve (AUC) statistic of 0.8, the JSS-SSI tool is composed of five items and has a reported AUC of 0.66, and the ACS-NSQIP is composed of 21 items with a reported AUC for SSI ranging from 0.73-0.82 [7,8]. The most widely used of these tools, the ACS-NSQIP, with 21 items to enter is relatively onerous to use, especially in the standard workflow of the operating room (OR), one of the most complex, high-stress environments in healthcare. All three of the aforementioned models require the use of an actual calculator embedded with specific variable weights that may not be easy to access in all surgical environments. With the goal of incorporating SSI risk prediction into routine surgical operations, we sought to develop and evaluate a straightforward predictive instrument based on commonly available pre-operative and intraoperative data to identify high-risk SSI patients before leaving the OR.

Materials And Methods
In this study, we developed an SSI predictive model (SSI-PM) based on review of SSI literature and performed a retrospective, observational, split sample cohort study to assess the SSI-PM's predictive accuracy in a real-world population of surgical patients using data extracted from an electronic health record (EHR).

Risk factor identification
To identify candidate SSI risk factors, a PubMed narrative literature review was performed using search terms related to SSI events and risk. The review identified 14 risk factors that were a) independently associated with statistically significant increased SSI risk and b) based on data that is readily available in EHR data or easily reported from data collected from the surgical theater. Eight pre-operative risk factors were identified: low albumin (present if <3.5) [8], American Society of Anesthesiologists Physical Status Classification (ASA) score (present if ≥ 3) [8,9], abnormal body mass index (BMI) (present if <20 or >35 kg/m 2 ) [9], comorbid diabetes mellitus [8,9], procedure urgency (present if trauma or emergency) [10], prophylactic antibiotic administered <120 minutes before incision [9], current tobacco use [8,9], and current steroid use [8]. Six intraoperative risk factors were identified: open surgery [11], blood loss (present if ≥600 mL) [10], perioperative blood transfusion [10], prolonged operative time (present if >2 hours) [9], high-risk procedure (present if colorectal or small bowel procedure) [8,10], and wound classification [8] ( Table 1). Though wound classification is readily reportable in the OR, it was not available in the EHR and was excluded from the analysis, leaving 13 in the final model.

Statistical analysis and modeling
Descriptive statistics were calculated on the patient cohort using prevalence and percentage values for categorical variables and means and standard deviations for continuous variables. The prevalence of each risk factor and its association with SSIs was assessed based on the training dataset containing 2,048 surgical patients. Variable distribution and skew are presented by median and interquartile range. Group comparisons were performed using Student's T-test for continuous variables. Comparisons for categorical variables were calculated using the χ2 test or Fisher's exact test as appropriate. Unadjusted odds ratios (ORs) for individual risk factors' association with SSI events were also calculated.
Predictive accuracy was assessed using two models: a) simple count of risk factors (score: 0-13) and b) multivariate logistic regression model predicting SSI outcomes in a population of surgical events. To develop and validate each predictive method, the dataset was randomly divided (70%/30%) into training (n= 2,408 procedures) and test (n=1,032 procedures) datasets. Training of the logistic regression model was performed on the dataset using generalized linear models with a logit link function. Risk factors served as inputs for the model, and SSI occurrence served as the output. Measures of model performance for both predictive models were derived from application of the models in the test dataset. Overall, model performance was based on AUC of the receiver operating characteristic (ROC) curve, which plots model sensitivity versus model specificity.
To calculate different levels of model sensitivity, specificity, and positive and negative predictive value, a series of progressively higher score threshold (or cut-off) levels were selected. In this approach, individuals above that score threshold are assumed to be at a high risk of SSI and then compared against the actual presence of SSI as measured in the EHR data. The high-risk cut-off scores for the simple count of positive items were the count of positive risk factors, ranging from the lowest threshold requiring one or more risk factors to be considered high risk up to the highest threshold measured requiring at least eight positive risk factors to be considered at high-risk SSI. Cut-off risk probabilities for the logistic regression model were matched to the ones of the simple model based on equal sensitivity values.
To summarize the association between the observed and predicted binary classifications in a single metric, F1 score and Matthews's correlation coefficient (MCC) were calculated for each high-risk cut-off. Though the F1 score, the harmonic mean of the positive predicted value and sensitivity, is the more commonly used measure, it ignores correctly classified negative examples. MMC, on the other hand, is regarded as a superior measure as it describes all four confusion matrix categories (true positives, false positives, true negatives, false negatives). The MCC returns values between -1 and +1, whereas a coefficient of +1 represents a perfect prediction, 0 is no better than random chance, and -1 indicates complete disagreement. Statistical testing and regression modeling using the glm library was performed in R software [12].

Ethics approval
The Tel Hashomer Helsinki Committee approved the research and waived the requirement to obtain informed consent (SMC-6411-19). All methods were performed in accordance with privacy and confidentiality guidelines and regulations.
The 70% sample randomly assigned to the training dataset were comparable to the 30% assigned to the validation sample, with no statistically significant differences in demographic characteristics, comorbidities, surgery type, or surgical outcomes, including SSI rate, length of stay, ICU admission rate, 30day readmission rate, or 30-day mortality rate ( Table 2).

Predictive accuracy
Model performance statistics including ROC curves were calculated on the test dataset containing 1,032 surgical patients ( Table 5, Figure 1).

FIGURE 1: Receiver operator characteristic (ROC) curves for the two versions of the SSI prediction models
The AUCs for the predictive models based on count of risk factors and logistic regression were 0.83 and 0.87, respectively. For both models, the MCC -a measure of association between the predicted group (high versus low risk) and the actual group (SSI versus no SSI) -was largest at a sensitivity of 80% (= the models identify 80% of all patients with an SSI event). However, the MCC score was higher for the logistic regression model with an MCC of 0.35 than for the count of positive items model, with MCC = 0.30, at 80% sensitivity. In line with this, other accuracy metrics also varied between the two models. At a sensitivity of 80%, the manual score requires the presence of four or more risk factors; it has a specificity of 77.5%, a positive predictive value of 16.3%, and a negative predictive value of 98.7%; it categorizes 25.5% of patients as high-risk for SSI who might benefit from aggressive management including prophylaxis; and 74.5% of patients as low risk who can be managed with greater efficiency and fewer interventions. Similarly, setting the automated (regression-based) model output at a sensitivity of 80% and the model specificity is 82.9%, has a positive predictive value of 20.5% and a negative predictive value of 98.8%, and categorizes 20.3% of patients as high-risk for SSI and 79.7% as low-risk.

Discussion
Despite international efforts to reduce infections, SSIs continue to be a substantial source of clinical morbidity and economic burden. Focusing surveillance and prevention efforts on high-risk patients has the potential to support cost-effective SSI reduction. Existing general SSI prediction models have limited predictive accuracy, can require substantial data entry, and require data entry into a pre-programmed calculator commonly external to the day-to-day workflow of clinicians. Anticipating the potential for surgical teams to have limited access to real-time analytics, we developed and validated two different predictive algorithms derived from 13 evidence-based risk factors readily available to the surgical team, a manual count, and a regression-based algorithm requiring real-time computation. The manual scoring algorithm has the advantage of being easy to integrate into non-electronic clinical workflows, whereas the automated algorithm requires real-time analytical capabilities but provides a more accurate risk assessment. Both SSI risk models had comparable or slightly better AUC curves compared to other reported SSI tools including the SSIRS, JSS-SSI, and ACS-NSQIP.
In our analysis, after adjusting for all present risk factors, eight of the proposed 14 risk factors including BMI, diabetes mellitus, tobacco use, albumin level, ASA score, intraoperative blood loss, intraoperative blood transfusion, and concomitant steroid use were not associated with SSI risk. In systematic reviews of SSI risk factors, inconsistent associations between potential risk factors and SSI event risk have been noted [9,13].
Reasons for inconsistent results regarding risk factor predictive capability can include variable definition, specific surgeries included in a study, and variations in study characteristics. Although these eight factors were not associated with increased risk for SSI in this study, the published literature provides compelling evidence that they can be associated with increased risk and thus should be included in further research and development of the SSI-PM.
The incorporation of SSI-PM into routine surgical care has the potential to support and augment existing efforts to reduce SSIs. The impact of traditional surgical safety checklists has been inconsistent [14]. While overall SSIs for inpatient procedures in the United States decreased by 16% from 2010-2015, initial reports of one of the more prominent U.S. efforts to reduce SSIs using checklists, the Surgical Care Improvement Program (SCIP) (2010-11), was ineffective at reducing SSIs [15,16]. Barriers to successful surgical checklist implementations and achieving reductions in SSIs include confusion regarding how to properly use a checklist, challenges in workflow efficiency, general surgeon resistance to change, concerns about legal implications of signing a checklist form, failure to perform safe surgical practices despite documentation of completion, variations in surgical team competency, and the importance of a culture of safety [15][16][17][18]. Using SSI prediction tools such as the SSI-PM has the potential to increase motivation and focus surgical staff to adhere to checklist procedures for those at greatest risk.
Considering the need for fiscal responsibility in healthcare, SSI risk prediction has the potential to optimize the impact of limited resources, enabling more targeted use of SSI prevention measures such as antibiotics and negative pressure wound therapy. Additionally, almost all SSIs resulting in readmission occur after patient discharge [19], and delays in detection and treatment can result in serious consequences. Moreover, patients at high-risk for SSI are excellent candidates to receive more intensive prevention such as the use of negative pressure wound therapy [20] as well as enhanced post-operative monitoring to enable early SSI detection and treatment [21,22].
The success of SSI risk identification and reduction tools will continue to be limited until they seamlessly integrate into clinical workflow and provide simple, actionable analytics tied to clear clinical decisionmaking guidance at the point of care. Future research will seek to assess the SSI model in different surgical populations and evaluate impact in real-world surgical settings. To reduce barriers to adoption, advanced versions of the SSI-PM regression model should minimize data entry, auto-populate with risk factors recorded in the EHR, and enable real-time guidance to reduce SSI.

Limitations
In keeping with a real-world evidence generation approach, the identification and selection of potential risk factors was based on clinical experience and narrative literature review rather than an exhaustive systematic literature review, and we cannot rule out the possibility that important risk factors were omitted. Moreover, this initial study population was restricted to individuals undergoing major, open, complex abdominal procedures within a single healthcare provider organization, factors that limit the generalizability of these findings to other types of procedures (e.g., orthopedic, neurosurgery) and populations from other geographic regions. With regard to model training and testing using a single 70%-30% split, there are always concerns for model overfitting. In future studies, other model validation approaches will include crossvalidation or the creation of a second split to create a holdout dataset.
Despite being a well-documented risk factor for SSI, wound class (per the U.S. Centers for Disease Control and Prevention: I -clean; II -clean/contaminated; III -contaminated; and IV -dirty) [23] was not available in the EHR in this data sample and was excluded from the analysis [8,9]. Future efforts will explore the use of wound classification in SSI-PM.

Conclusions
Two SSI predictive models -one that can be calculated manually and one that requires analytical resourceshad high predictive accuracy. Use of either prediction model before the patient leaves the OR can enable proactive real-time intervention to mitigate risk and reduce SSIs. SSI risk prediction tools are an important component in a comprehensive approach to SSI reduction but will be limited in their effectiveness until they are seamlessly integrated into clinical workflow and enable real-time action at the point of care.

Additional Information
Disclosures