Predictive Modeling of Ambulatory Outcomes After Spinal Cord Injury

And 20 more flashcards and 40 more questions in the document.

Document text

ARTICLE Predictive modeling of ambulatory outcomes after spinal cord injury using machine learning Christina Draganich 1 , Dustin Anderson2 , Grant J. Dornan 3 , Mitch Sevigny4 , Jeffrey Berliner1,4 , Susan Charlifue4 , Abigail Welch4 and Andrew Smith1 © The Author(s), under exclusive licence to International Spinal Cord Society 2024 STUDY DESIGN: Retrospective multi-site cohort study. OBJECTIVES: To develop an accurate machine learning predictive model using predictor variables from the acute rehabilitation period to determine ambulatory status in spinal cord injury (SCI) one year post injury. SETTING: Model SCI System (SCIMS) database between January 2000 and May 2019.

METHODS: Retrospective cohort study using data that were previously collected as part of the SCI Model System (SCIMS) database. A total of 4523 patients were analyzed comparing traditional models (van Middendorp and Hicks) compared to machine learning algorithms including Elastic Net Penalized Logistic Regression (ENPLR), Gradient Boosted Machine (GBM), and Arti cial Neural Networks (ANN). RESULTS: Compared with GBM and ANN, ENPLR was determined to be the preferred model based on predictive accuracy metrics, calibration, and variable selection. The primary metric to judge discrimination was the area under the receiver operating characteristic curve (AUC).

When compared to the van Middendorp all patients (0.916), ASIA A and D (0.951) and ASIA B and C (0.775) and Hicks all patients (0.89), ASIA A and D (0.934) and ASIA B and C (0.775), ENPLR demonstrated improved AUC for all patients (0.931), ASIA A and D (0.965) ASIA B and C (0.803). CONCLUSIONS: Utilizing arti cial intelligence and machine learning methods are feasible for accurately classifying outcomes in SCI and may provide improved sensitivity in identifying which individuals are less likely to ambulate and may bene t from augmentative strategies, such as neuromodulation. Future directions should include the use of additional variables to further re ne these models.

Spinal Cord (2024) 62:446–453; https://doi.org/10.1038/s41393-024-01008-2 INTRODUCTION Spinal cord injury (SCI) remains one of the most devastating injuries an individual can sustain and substantially impacts all aspects of life, including social, medical, and economic well-being [1]. The degree of neurologic recovery has a large in uence on the expected medical treatment and quality of life for a person with SCI [2]. For individuals with motor incomplete SCI, rehabilitation often focuses on recovery of ambulation, however establishing a prognosis for recovering walking ability within the community is dif cult [3].

This has led to the development of statistical models and machine learning algorithms that use neurologic and demographic information to predict recovery [2–12]. The van Middendorp et al. logistic regression model is a 5-variable model that uses age, quadratus femoris motor score, gastrocnemius motor score, light touch L3 score and light touch S1 score to predict walking recovery following SCI and has been externally validated in several countries [1, 13]. Hicks et al. also developed a 3-variable reference model that uses age, L3 motor score and S1 light touch score to predict ambulatory outcomes [14]. These are the two most widely validated prediction models. However, Phan et al.

recently showed that these models poorly predicted walking in individuals with AIS B and C SCI whose outcomes would be most clinically useful [3, 14]. The individual classi cation accuracies were 91.49% and 97.38% for AIS A and D respectively as opposed to 71.52% and 78.52% for AIS B and C respectively. Limitations in these models also include the dichotomization of age and failure to include clinical factors that may impact rehabilitation outcomes, such as comorbidities and social support systems. Literature has shown that presence of co- existing injuries can impact survival, but has not examined the impact on ambulatory outcomes [6].

The accurate prediction of neurological outcomes in individuals with SCI is dif cult due to the heterogeneity of patient characteristics and treatment strategies. Machine learning has become a promising tool to increase the accuracy of outcome predictions in various elds and has recently been used increasingly in the study of SCI outcomes [15, 16]. Compared with conventional statistical models, such as a logistic regression analysis, machine learning prediction models can readily incorpo- rate non-linear interactions among prognostic factors [16]. Furthermore, they can allow for dynamic and real time modeling of interactions among a variety of predictors [17].

Machine Received: 3 May 2023 Revised: 12 May 2024 Accepted: 6 June 2024 Published online: 18 June 2024 1 University of Colorado Department of Physical Medicine and Rehabilitation, Aurora, CO, USA. 2 The Steadman Clinic, Aspen, CO, USA. 3 Steadman Philippon Research Institute, Vail, CO, USA. 4 Craig Hospital, Englewood, CO, USA. email: cdraganich@gmail.com www.nature.com/sc 1234567890();,: learning typically requires large data sets designed for prediction research, including a broad variety of predictors and appropriately re ecting the population under study [17].

A more accurate predictive model would allow for the optimiza- tion of early targeted rehabilitation for individuals with SCI, ultimately leading to improvements in recovery and rehabilitation outcomes. Sophisticated research is needed to develop such a model, especially with regard to individuals with ASIA Impairment Scale (AIS) B and C SCI [16]. We hypothesized that machine learning algorithms could be used to create a more accurate predictive model that could be compared to the most widely validated existing models.

Thus, the objective of this study was to develop a more accurate predictive model of ambulatory outcomes in SCI to aid in the optimization of early targeted rehabilitation and patient education, and to compare this model to the van Middendorp and Hicks models. Speci cally, we sought to examine which co-existing conditions as well as demographic and neurologic variables are the strongest predictors of future ambulatory function when included in a machine learning algorithm. METHODS Subjects This was a retrospective cohort study using data that were previously collected as part of the SCI Model System (SCIMS) database between January 2000 and May 2019.

The SCIMS database is a repository of data collected from 18 SCI Model Systems Centers which represents data from more than 51,000 persons with SCI making it the world's largest and longest active SCI research database. SCIMS centers provide a multi- disciplinary system of rehabilitation care ranging from emergency services through rehabilitation specialized in SCI care. Nine-thousand seven hundred eight (9708) patients met the inclusion criteria of primary diagnosis of SCI admitted to an SCI model system center.

Among those, 5085 patients were then excluded due to unavailability of light touch and pin prick testing, and 100 patients were excluded due to unknown ambulatory status at 1-year, leaving 4523 patients included in the nal model development and validation process. This retrospective observa- tional chart review was approved by the local Institutional Review Board. Measures Data were obtained from the SCIMS database. Clinical prognostic variables included demographics, injury characteristics, rehabilitation characteristics, psychosocial characteristics and past medical history data collected during rehabilitation admission at 12 sites as part of the SCIMS program.

Speci c predictor variables were chosen based on the researchers' consensus of predictive importance of a given variable and literature support for the predictive importance of the variable [6–8, 11, 12, 14]. These variables are described in Table 1. As part of the SCIMS program, neurological examinations were performed according to the International Standards for Neurological Classi cation of Spinal Cord Injuries (ISNCSCI). The primary clinical outcome measure was of one-year ambulatory status collected as part of the SCIMS.

One year outcome data was chosen since standard follow up measures to assess long term outcomes are typically conducted at one year post injury at which point neurologic and functional outcomes are known to plateau [18]. The outcome measure of ability to walk 150 feet in the home (with or without ambulatory aid) was chosen based on the authors' consensus of the clinical importance of the variable and literature support for the variable [6–8, 13, 14, 19]. See Table 1 for a descriptive summary of the variables available for algorithm building process. Statistical analysis General approach and data preparation.

Additional details about data preparation, algorithm development, software and computational hard- ware, model diagnostics, visualizations, and thorough descriptive and comparative reporting are provided as Supplemental Material. Transparent Reporting of a multivariable prediction model for Individual Prognosis OR Diagnosis (TRIPOD) guidelines were followed [20]. The statistical program- ming language R version 4.2.1 was used for all analyses [21]. After removal of cases for which full baseline ASIA classi cation data was unavailable, missing data patterns were analyzed and a missing-at-random mechanism was judged to be reasonable.

Length of stay in acute care was missing in 53% of cases, while hypertension, arthritis and hyperlipidemia were missing in 28%. All other variables were missing for less than 7% of cases. Five multiply imputed data sets were performed with fully conditional speci cation using the R package mice to assess for convergence and distribution properties [22]. Due to the complexity and diversity of the three machine learning models trained, and the lack of consensus for whether and how to pool multiple imputation results for such models, a single imputation set was ultimately selected for algorithm development and testing.

One-hot encoding was performed to transform categorical covariates into multiple dummy variables. An 80/20 percent random split was performed to separate the data into training and independent testing sets. To avoid information leakage between sets, separate imputation processes were applied to each. Three modern machine learning model algorithms were developed for comparison: 1) elastic net penalized logistic regression (ENPLR), 2) gradient boosted machine (GBM), and 3) arti cial neural network (ANN). Machine learning algorithm development and tuning. ENPLR was imple- mented using the package glmnet [23].

The elastic net model is a hybrid between the LASSO model and ridge regression, thus combining two attractive features: 1) variable selection and 2) shrinkage of coef cients of correlated predictors [24]. The ENPLR hyperparameter alpha was tuned via grid search and the largest lambda value within 1 standard error from the minimum was selected. The GBM model was developed via the extreme gradient boosting framework using the xgboost package [25]. GBM models can natively handle missing data, so the original non-imputed dataset was used for model training.

Tuning of the eta (range 0.001–1), maximum tree depth (range 2–10), minimum child weight (range 1–25), and subsample (range 0.25–1) hyperparameters was conducted via Bayesian optimization to more ef ciently traverse the parameter space than a naïve grid search [26]. The ANN model was constructed as a 1-hidden-layer feedforward neural network. Training was executed by TensorFlow 2.0 as implemented through the keras and tfruns R packages [27, 28].

Up to 50 training epochs were allowed and hyperparameter optimization was performed by a 5% grid search across: hidden layer nodes (8, 16, 32, 64, 128, 256), dropout rate (0.1, 0.2, 0.3, 0.4), optimizer (rmsprop or adam), and learning rate (0.1, 0.05). Assessments of model performance. Model performance was tested on the 20% of data reserved for internal validation. Comparison among the three models was performed on the basis of both discrimination and calibration. The primary metric to judge discrimination was the area under the receiver operating characteristic curve (AUC).

An AUC of 1.00 would indicate perfect discrimination while an AUC of 0.50 indicates a model that predicts 1-year ambulatory status no better than random chance. In general, 0.70 to 0.80 is considered acceptable, 0.80 to 0.90 is considered excellent, and greater than 0.90 is considered outstanding [29]. We used these standard cut points to determine diagnostic accuracy. To further aid in clinical interpretation, optimal cut point analysis was performed to maximize the Youden index independently within each model [30]. Using the associated optimal cut point, positive predictive value (PPV), negative predictive value (NPV), sensitivity, and speci city were reported.

The entire pro le of predictive performance metrics was considered when selecting a preferred model. Bootstrapping was performed to calculate the 95% con dence interval for each discrimination metric. Calibration curves were produced for each model, and visual inspection was used to judge calibration success. While it is possible to attain good discrimination with a poorly calibrated model, a well-calibrated model provides an additional concrete bene t whereby model predictions may be interpreted as personalized probability of 1-year ambulatory status for future individual SCI patients.

For comparison, the Hicks and van Middendorp models were applied to the entire dataset and compared on the basis of discrimination performance. Discrimination metrics were calculated distinctly for 1) patients with ASIA classi cation A or D at baseline, and 2) patients with ASIA classi cation B or C at baseline. Lastly, variable importance for each model was assessed using the package vip [31]. RESULTS A total of 9708 patients met the inclusion criteria of primary diagnosis of SCI admitted to a SCIM center. Among those, 5085 patients were then excluded due to unavailability of light touch C. Draganich et al.

447 Spinal Cord (2024) 62:446 – 453 and pin prick testing, and 100 patients were excluded due to unknown ambulatory status at 1-year, leaving 4523 patients included in the nal model development and validation process. The majority of individuals were male (79.7%) and Caucasian (70.7%). The median age and time from injury to rehabilitation were 41 years (range 16–95) and 14 days (range 1–336) respectively. The most common etiology of injury was vehicular accident (39.6%), the most common neurologic level of injury was cervical (60.8%), and the most common type of injury was incomplete (58%). Table 2 includes descriptive statistics of the sample.

Using this sample, we aimed to create a predictive model of ambulation for SCI utilizing arti cial intelligence and machine learning methods. We speci cally studied the use of ENPLR, ANN, and GBM. For the entire sample, the ANN model had the highest AUC (0.932), the ENPLR model had the second highest AUC (0.931), and the GBM model had the lowest AUC (0.924). Table 3 includes a comparison of the predictive performances for the Table 1. Descriptive summary of variables available for algorithm building process.

Categorical variables Number of factor levels Factor levels Site ID 12 Coded as: A, B, C, D, H, L, M, N, O, P, R, S Sex 3 Male; Female; Other, Transgender Race 5 American Indian, Alaska Native; Asian, Paci c Islander; Black, African American; Other Race, Multiracial; White, Caucasian Hispanic 2 Yes; No Marital Status 7 Divorced; Married; Other, Unclassi ed; Separated; Signi cant Other, Partner, Unmarried Couple; Single, Never Married; Widowed Education 8 8th Grade or less; 9th–11th Grade; Associate Degree; Bachelor's degree; Doctorate; High School Diploma/GED; Master's Degree; Other, Unclassi ed Employment Status 6 Homemaker; Other; Retired; Student; Unemployed; Working Injury Etiology Category 7 Falls; Medical; Other; Pedestrian; Sports; Vehicular; Violence Vertebral Injury 2 Yes; No Associated Injury 2 Yes; No Spinal Surgery 2 Yes; No Mechanical Ventilation 2 Yes; No Neuro Level 4 Cervical; Lumbar; Sacral; Thoracic Voluntary Anal Contraction 2 Yes; No Deep Anal Pressure 2 Yes; No ASIA at Rehab 4 A; B; C; D Neuro Impairment 6 Paraplegia, Complete; Paraplegia, Incomplete; Paraplegia, Minimal De cit; Tetraplegia, Complete; Tetraplegia, Incomplete; Tetraplegia, Minimal De cit Anxiety 2 Yes; No Depression 2 Yes; No Diabetes 2 Yes; No Hypertension 2 Yes; No Hyperlipidemia 2 Yes; No Arthritis 2 Yes; No Alcohol: # drinks per day 6 1 or 2; 10 or more; 3 or 4; 5 or 6; 7 to 9; None Continuous Variables Units Age at Injury Years BMI kg/m2 LOS Acute Care Days Injury to Rehab Admission Days ASIA Sub-Tests Laterality Vertebral Levels Values Motor Score Left, Right L2, L3, L4, L5, S1 0, 1, 2, 3, 4, 5 Sensory: Light Touch Left, Right L1, L2, L3, L4, L5, S1, S2, S3, S4-5 0, 1, 2 Sensory: Pin Prick Left, Right L1, L2, L3, L4, L5, S1, S2, S3, S4-5 0, 1, 2 Endpoint Variables Number of Factor Levels Factor Levels Walk 150 ft at 1-year 2 Yes; No Walk One Block at 1-year 2 Yes; No Walk Flight of Stairs at 1-year 2 Yes; No ASIA Score at 1-year 5 A; B; C; D; E C.

Draganich et al. 448 Spinal Cord (2024) 62:446 – 453 three models. All models met the criteria for outstanding discrimination (AUC > 0.90). Calibration plots were created to further validate each respective model. Compared with GBM and ANN, ENPLR demon- strated the best calibration in that the predicted probabilities from the model most closely aligned with the observed probabilities. Figures 1, 2 and 3 illustrate the calibration plot for ENPLR, GBM, and ANN respectively.

Overall, the elastic net algorithm was determined to be the most clinically useful predictive model given that it had an AUC that met the criteria for excellent discrimination, the superior calibration, and the fewest number of variables. Thirty-one out of 74 candidate variables were included in the nal ENPLR model. These variables along with their relative predictive in uence can be seen in a variable importance plot in the supplementary materials. We then compared the ENPLR model with the existing van Middendorp and Hicks models. For all individuals, the AUC for van Middendorp was 0.916 and Hicks was 0.890, whereas the AUC was 0.931 for ENPLR.

For all individuals, the majority of these AUCs met the criteria for outstanding discrimination (greater than 0.90), except for Hicks which was excellent (0.80–0.90). See Table 4 for reporting of predictive accuracy metrics for the entire sample. For individuals with AIS A and D SCI, the AUC for van Middendorp was 0.951 and Hicks was 0.934, whereas the AUC was 0.965 for ENPLR. For individuals with AIS A and D SCI, all of these AUCs met the criteria for outstanding discrimination (greater than 0.90). For individuals with AIS B and C SCI, the AUC for van Middendorp was 0.775 and Hicks was 0.745, whereas the AUC was 0.803 for ENPLR.

For individuals with AIS B and C SCI, the Hicks and van Middendorp models met the criteria for acceptable discrimination Table 2. Demographic characteristics.

Covariate N = 4523a Percent missing Injury to Rehab Admission (days) 14 [1, 336] – 23 ± 28 0 LOS Acute Care (days) 9 [1, 242] – 14 ± 19 54 Age at Injury 41 [16,95] – 42 ± 18 0.6 Sex 0 Female 915/4523 (20%) Male 3604/4523 (80%) Other, Transgender 4/4523 ( < 0.1%) Race 1.3 American Indian, Alaska Native 36/4463 (0.8%) Asian, Paci c Islander 93/4463 (2.1%) Black, African American 1009/4463 (23%) Other Race, Multiracial 171/4463 (3.8%) White, Caucasian 3154/4463 (71%) Ethnicity 1.3 Hispanic 574/4464 (13%) Not Hispanic 3890/4464 (87%) Marital Status 0.6 Divorced 443/4496 (9.9%) Married 1703/4496 (38%) Separated 93/4496 (2.1%) Signi cant Other, Partner, Unmarried Couple 148/4496 (3.3%) Single, Never Married 2004/4496 (45%) Widowed 104/4496 (2.3%) Other, Unclassi ed 1/4496 (<0.1%) Education 1.2 8th Grade or less 115/4469 (2.6%) 9th–11th Grade 550/4469 (12%) High School Diploma/GED 2295/4469 (51%) Associate Degree 362/4469 (8.1%) Bachelor Degree 690/4469 (15%) Masters Degree 271/4469 (6.1%) Doctorate 120/4469 (2.7%) Other, Unclassi ed 66/4469 (1.5%) Employment Status 0.9 Homemaker 41/4484 (0.9%) Retired 515/4484 (11%) Student 368/4484 (8.2%) Unemployed 566/4484 (13%) Working 2913/4484 (65%) Other 81/4484 (1.8%) Injury Etiology Group 0.2 Falls 1435/4514 (32%) Medical 184/4514 (4.1%) Pedestrian 41/4514 (0.9%) Sports 413/4514 (9.1%) Vehicular 1787/4514 (40%) Violence 630/4514 (14%) Table 2.

continued Covariate N = 4523a Percent missing Other 24/4514 (0.5%) Vertebral Injury 3734/4512 (83%) Associated Injury 1613/4500 (36%) Spinal Surgery 3759/4512 (83%) Mechanical Ventilation Rehab 441/4514 (9.8%) Neuro Level Rehab 1.5 Cervical 2709/4453 (61%) Lumbar 255/4453 (5.7%) Sacral 2/4453 ( < 0.1%) Thoracic 1487/4453 (33%) Anxiety 454/4407 (10%) Depression 611/4400 (14%) Hypertension 658/3258 (20%) Hyperlipidemia 506/3238 (16%) Arthritis 524/3241 (16%) Number of Alcoholic Drinks per Day 6.2 None 1154/4242 (27%) 1 or 2 1680/4242 (40%) 3 or 4 870/4242 (21%) 5 or 6 343/4242 (8.1%) 7 to 9 108/4242 (2.5%) 10 or more 87/4242 (2.1%) a Median [Range] – Mean ± SD; n/N (%). C. Draganich et al.

449 Spinal Cord (2024) 62:446 – 453 (0.70–0.80), while the ENPLR model met the criteria for excellent discrimination (0.80–0.90). See Table 4 for reporting of predictive accuracy metrics. DISCUSSION In the current study, we aimed to create a predictive model of ambulation for SCI utilizing arti cial intelligence and machine learning methods. We speci cally studied the use of ENPLR, ANN, and GBM. Overall, the elastic net algorithm was determined to be the most clinically useful predictive model given that it had an AUC that met the criteria for excellent discrimination, the superior calibration, and the fewest number of variables.

Although our model was only slightly more accurate than the existing Hicks and van Middendorp models, it met the criteria for outstanding discrimination both for the entire sample and for individuals with AIS A and D SCI. It also met the criteria for excellent discrimination for individuals with AIS B and AIS C SCI. The van Middendorp and Hicks models met the criteria for either excellent or outstanding discrimination for the entire sample and Table 3. Comparison of predictive performance among 3 machine learning models (ENPLR, ANN, XGB) and 2 previously published models (vM, Hicks) for the entire sample.

AUROC Accuracy Sensitivity Speci city PPV NPV ENPLR 0.931 (0.9142, 0.9536) 0.8707 (0.8368, 0.9018) 0.8817 (0.8585, 0.9458) 0.863 (0.7688, 0.888) 0.818 (0.7327, 0.8531) 0.9127 (0.901, 0.9566) ANN 0.9322 (0.9098, 0.9545) 0.8729 (0.8283, 0.8947) 0.8656 (0.7907, 0.9296) 0.878 (0.7783, 0.9246) 0.832 (0.7417, 0.8865) 0.9035 (0.8564, 0.9435) XGB 0.9235 (0.8995, 0.9481) 0.8586 (0.8092, 0.8845) 0.8495 (0.7661, 0.9353) 0.8649 (0.7301, 0.9171) 0.8144 (0.7074, 0.8741) 0.8917 (0.8385, 0.9445) Each performance metric is presented with its 95% bootstrap con dence interval in parentheses.

Sensitivity, Speci city, PPV and NPV were derived after de ning the optimal cutpoint for each model based on Youden's index. ENPLR elastic net penalized logistic regression, ANN arti cial neural network, XGB extreme gradient boosted machine, AUROC area under the receiver operating characteristic curve, PPV positive predictive value, NPV negative predictive value. Fig. 1 ENPLR calibration plot. Fig. 2 GBM calibration plot. C. Draganich et al. 450 Spinal Cord (2024) 62:446 – 453 for individuals with AIS A and D SCI. For individuals with AIS B and C SCI, these models met the criteria for acceptable discrimination.

These ndings evidence the feasibility of creating an accurate model of ambulatory prediction using machine learning and also con rm the accuracy of the prior van Middendorp and Hicks models using a large independent dataset. While the differences in overall accuracy of ambulatory prediction for individuals with AIS B and C SCI were small, we did see signi cant differences between the groups in sensitivity and negative predictive value (NPV). Compared with the van Middendorp and Hicks models, ENPLR had a higher sensitivity (0.86 compared to 0.76 and 0.68 respectively) and a higher NPV (0.88 compared to 0.79 and 0.74 respectively).

Therefore, the ENPLR model would be expected to improve the ability to rule out future ambulators compared to the van Middendorp and Hicks models. This is similar to prior studies that demonstrated similar overall accuracy with machine learning, but increased sensitivity [32]. This improvement in NPV and sensitivity would allow for improved ability to determine which individuals with AIS B and C SCIs would be less likely to ambulate, potentially allowing these individuals to be assigned to augmentative interventions, such as neuromodulatory interventions, earlier in their rehabilitation process [33–35].

Thus, if a clinician's goal is to determine which individuals may need to focus on compensatory or augmentative strategies, they may consider electing to use the ENPLR model. The majority of the variables used in the ENPLR prediction model are collected as a part of the ISNCSCI examination which is routinely performed on all individuals with traumatic SCI. The remaining variables include length of stay (LOS), injury to rehab admission days, age, mechanical ventilation and alcohol use which are variables easily obtained on interview with the patient or in chart review.

Although this model includes more variables than the Hicks and van Middendorp models, they are readily available and by improving the sensitivity and NPV, we are able to rule out more potential ambulators in this group. A strength of this study is the large dataset (n = 4523) utilized to create this model. Additionally, the dataset was derived from 12 independent SCIMS centers throughout the US.

Given the feasibility of machine learning demonstrated in this study, future directions should include the incorporation of other variables, such as imaging, which could help to create more accurate ambulatory prediction models in the future with a goal of outstanding discrimination for individuals with AIS B and C SCIs [36, 37]. Future directions should also include the creation of a Fig. 3 ANN calibration plot. Table 4. Comparison of predictive performance among ENPLR and 2 previously published models (vM, Hicks).

Subgroup AUROC Accuracy Sensitivity Speci city PPV NPV ENPLR All 0.931 (0.9142, 0.9536) 0.8707 (0.8368, 0.9018) 0.8817 (0.8585, 0.9458) 0.863 (0.7688, 0.888) 0.818 (0.7327, 0.8531) 0.9127 (0.901, 0.9566) ASIA A and D 0.9605 (0.9488, 0.9817) 0.9368 (0.9212, 0.9599) 0.9398 (0.9114, 0.9597) 0.9347 (0.9171, 0.9661) 0.9105 (0.8847, 0.9522) 0.9564 (0.9396, 0.9672) ASIA B and C 0.8032 (0.7391, 0.8499) 0.7196 (0.6554, 0.7772) 0.8632 (0.7328, 0.8925) 0.6257 (0.503, 0.7172) 0.6012 (0.5198, 0.7044) 0.875 (0.7952, 0.8901) vM All 0.9159 (0.9057, 0.9299) 0.8572 (0.8308, 0.871) 0.863 (0.8421, 0.9032) 0.8531 (0.7865, 0.8654) 0.8073 (0.7489, 0.8204) 0.8972 (0.8784, 0.9218) ASIA A and D 0.9512 (0.9459, 0.9648) 0.9226 (0.9114, 0.9335) 0.9409 (0.9298, 0.9611) 0.9103 (0.8942, 0.9253) 0.8762 (0.8625, 0.8894) 0.958 (0.9502, 0.973) ASIA B and C 0.7746 (0.7496, 0.7943) 0.7247 (0.7027, 0.7415) 0.7631 (0.7347, 0.7889) 0.6941 (0.673, 0.7224) 0.6658 (0.6226, 0.6957) 0.7858 (0.7453, 0.8132) Hicks All 0.8895 (0.881, 0.9075) 0.8276 (0.8138, 0.8449) 0.9106 (0.7863, 0.9285) 0.7683 (0.7496, 0.8823) 0.7371 (0.7173, 0.8219) 0.9234 (0.8575, 0.9401) ASIA A and D 0.9344 (0.9296, 0.948) 0.915 (0.9009, 0.9303) 0.9073 (0.8974, 0.9362) 0.9203 (0.8878, 0.937) 0.8848 (0.853, 0.902) 0.9363 (0.9276, 0.9557) ASIA B and C 0.7446 (0.7232, 0.7628) 0.707 (0.6732, 0.7181) 0.6831 (0.6391, 0.7128) 0.726 (0.6987, 0.7446) 0.6657 (0.6211, 0.6903) 0.7415 (0.6943, 0.7766) Each performance metric is presented with its 95% bootstrap con dence interval in parentheses.

Each model was tested on all test set patients, the subset of test patients that were classi ed as ASIA A or D at rehab admission, and the subset of patients that were classi ed as ASIA B or C at rehab admission. Sensitivity, Speci city, PPV and NPV were derived after de ning the optimal cutpoint for each model based on Youden's index. ENPLR elastic net penalized logistic regression, ANN arti cial neural network, XGB extreme gradient boosted machine, vM model published in van Middendorp et al., Hicks model published in Hicks et al., AUROC area under the receiver operating characteristic curve, PPV positive predictive value, NPV negative predictive value. C. Draganich et al.

451 Spinal Cord (2024) 62:446 – 453 clinical prediction tool utilizing machine learning models that can be easily utilized by the rehabilitation team. Given that the majority of the variables may be derived from the ISNCSCI examination, technology could be leveraged to input data directly from the ISCNCSI scoring sheet into the prediction tool for increased use. Future research in this area is important given that a more accurate predictive model can aid in the optimization of early targeted rehabilitation and patient education for individuals with SCI, potentially leading to improvements in recovery and rehabilitation outcomes. Limitations This study includes several limitations.

First, these models included variables from ISNCSCI examinations performed on admission to rehabilitation and not ISNCSCIs in the acute phase of injury. Additionally, time to rehabilitation admission varied, so the ISNCSCI examinations utilized for this model were not conducted at the same time post injury for the individuals in the sample. Second, we are unable to verify the accuracy of the ISNCSCI examinations performed given that the data were collected retrospectively, however all data were obtained from SCIMS centers, so the authors are con dent in the quality of the ISNCSCI examinations utilized.

Third, the complex and multifactorial nature of machine learning models can make intuitive interpretation of the model's predictive results dif cult or impossible to obtain. Fourth, the nal machine learning model included 31 variables, which is a greater number of variables than prior models. Although these variables are typically available via chart review and review of the ISNCSCI, the larger number of variables required for prediction may limit the user friendliness for healthcare practitioners. In this case, simpler prediction models may be considered [38].

Opportunity also exists to connect health data systems to standardize and automate the process of conducting personalized prediction for future patients Finally, machine learning models rely on large data sets for accuracy, and over half of the eligible individuals were excluded due to lack of light touch and pinprick, or unknown ambulatory status. CONCLUSION In conclusion, utilizing arti cial intelligence and machine learning methods are feasible for accurately classifying outcomes in spinal cord injury and may provide improved sensitivity in identifying which individuals with incomplete SCI are less likely to ambulate and may bene t from augmentative strategies, such as neuromo- dulation.

This could have implications on counseling patients appropriately for expectations in regard to function, identifying appropriate adaptive equipment, and directing rehabilitation programs to enhance recovery. Future directions should include investigating the use of additional variables, such as imaging variables, to further re ne these models and improve predictive accuracy. DATA AVAILABILITY Data available upon request from the corresponding author. REFERENCES 1. van Middendorp JJ, Goss B, Urquhart S, Atresh S, Williams RP, Schuetz M. Diag- nosis and Prognosis of Traumatic Spinal Cord Injury. Glob Spine J. 2011;1:001–7. 2. Qiu Z, Wang F, Hong Y, Zhang J, Tang H, Li X, et al.

Clinical Predictors of Neu- rological Outcome within 72 h after Traumatic Cervical Spinal Cord Injury. Sci Rep. 2016;6:38909. 3. Phan P, Budhram B, Zhang Q, Rivers CS, Noonan VK, Plashkes T, et al. Highlighting discrepancies in walking prediction accuracy for patients with traumatic spinal cord injury: an evaluation of validated prediction models using a Canadian Multicenter Spinal Cord Injury Registry. Spine J. 2019;19:703–10. 4. Jean S, Mac-Thiong JM, Jean MC, Dionne A, Bégin J, Richard-Denis A. Early Clinical Prediction of Independent Outdoor Functional Walking Capacity in a Prospective Cohort of Traumatic Spinal Cord Injury Patients. Am J Phys Med Rehabil. 2021;100:1034–41. 5.

Draganich C, Weber KA, Thornton WA, Berliner JC, Sevigny M, Charlifue S, et al. Predicting Outdoor Walking 1 Year After Spinal Cord Injury: A Retrospective, Multisite External Validation Study. J Neurol Phys Ther. 2023. Available from: https://journals.lww.com/10.1097/NPT.0000000000000428. 6. Wilson JR, Grossman RG, Frankowski RF, Kiss A, Davis AM, Kulkarni AV, et al. A Clinical Prediction Model for Long-Term Functional Outcome after Traumatic Spinal Cord Injury Based on Acute Clinical and Imaging Factors. J Neurotrauma. 2012;29:2263–71. 7. Belliveau T, Jette AM, Seetharama S, Axt J, Rosenblum D, Larose D, et al.

Devel- oping Arti cial Neural Network Models to Predict Functioning One Year After Traumatic Spinal Cord Injury. Arch Phys Med Rehabil. 2016;97:1663–.e3. 8. Denis AR, Feldman D, Thompson C, Mac-Thiong JM. Prediction of functional recovery six months following traumatic spinal cord injury during acute care hospitalization. J Spinal Cord Med. 2018;41:309–17. 9. Waters RL, Adkins R, Yakura J, Vigil D. Prediction of ambulatory performance based on motor scores derived from standards of the American Spinal Injury Association. Arch Phys Med Rehabil. 1994;75:756–60. 10. Oleson CV, Burns AS, Ditunno JF, Geisler FH, Coleman WP.

Prognostic Value of Pinprick Preservation in Motor Complete, Sensory Incomplete Spinal Cord Injury. Arch Phys Med Rehabil. 2005;86:988–92. 11. Kaminski L, Cordemans V, Cernat E, M'Bra KI, Mac-Thiong JM. Functional Outcome Prediction after Traumatic Spinal Cord Injury Based on Acute Clinical Factors. J Neurotrauma. 2017;34:2027–33. 12. Kay ED, Deutsch A, Wuermser LA. Predicting Walking at Discharge From Inpatient Rehabilitation After a Traumatic Spinal Cord Injury. Arch Phys Med Rehabil. 2007;88:745–50. 13. van Middendorp JJ, Hosman AJ, Donders ART, Pouw MH, Ditunno JF, Curt A, et al.

A clinical prediction rule for ambulation outcomes after traumatic spinal cord injury: a longitudinal cohort study. Lancet. 2011;377:1004–10. 14. Hicks KE, Zhao Y, Fallah N, Rivers CS, Noonan VK, Plashkes T, et al. A simpli ed clinical prediction rule for prognosticating independent walking after spinal cord injury: a prospective study from a Canadian multicenter spinal cord injury reg- istry. Spine J. 2017;17:1383–92. 15. Inoue T, Ichikawa D, Ueno T, Cheong M, Inoue T, Whetstone WD, et al. XGBoost, a Machine Learning Method, Predicts Neurological Recovery in Patients with Cer- vical Spinal Cord Injury. Neurotrauma Rep. 2020;1:8–16. 16.

Khan O, Badhiwala JH, Wilson JRF, Jiang F, Martin AR, Fehlings MG. Predictive Modeling of Outcomes After Traumatic and Nontraumatic Spinal Cord Injury Using Machine Learning: Review of Current Progress and Future Directions. Neurospine. 2019;16:678–85. 17. Hodel J, Stucki G, Prodinger B. The potential of prediction models of functioning remains to be fully exploited: A scoping review in the eld of spinal cord injury rehabilitation. J Clin Epidemiol. 2021;139:177–90. 18. Kirshblum S, Millis S, McKinley W, Tulsky D.

Late neurologic recovery after trau- matic spinal cord injury11No commercial party having a direct nancial interest in the results of the research supporting this article has or will confer a bene t upon the author(s) or upon any organization with which the author(s) is/are associated. Arch Phys Med Rehabil. 2004;85:1811–7. 19. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594–g7594. 20. Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al.

Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration. Ann Intern Med. 2015;162:W1–73. 21. R Core Team. R: A language and environment for statistical computing. R Found Stat Comput Viennea Austria. 2020; Available from: https://www.R-project.org/. 22. Van Buuren S, Groothuis-Oudshoorn K. Mice: Multivariate Imputation by Chained Equations. R J Stat Softw. 2011;45:1–67. 23. Friedman J, Hastie T, Tibshirani R. Regularization Paths for Generalized Linear Models via Coordinate Descent. J Stat Softw. 2010;33. Available from: http:// www.jstatsoft.org/v33/i01/. 24. Fiedman J, Ribshirani R, Hastie T.

The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. Netherlands: Springer New York; 2008. 25. Chen T, Guestrin C XGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY United States: SIGMOD; 2016. 26. Wilson S. ParBayesianOptimization: Parallel Bayesian Optimization of Hyperparameters. Available from: https://CRAN.R-project.org/package=ParBayesianOptimization. 27. Allaire J, Chollet F. keras: R Interface to 'Keras'. 2022. Available from: https:// CRAN.R-project.org/package=keras. C. Draganich et al. 452 Spinal Cord (2024) 62:446 – 453 28. Allaire J.

tfruns: Training Run Tools for 'TensorFlow'. 2022. Available from: https:// CRAN.R-project.org/package=tfruns. 29. Mandrekar JN. Receiver Operating Characteristic Curve in Diagnostic Test Assessment. J Thorac Oncol. 2010;5:1315–6. 30. Thiele C, Hirschfeld G. cutpointr: Improved Estimation and Validation of Optimal Cutpoints in R. J Stat Softw. 2021;98. Available from: http://www.jstatsoft.org/v98/ i11/. 31. Greenwell BM, Boehmke BC. Variable Importance Plots—An Introduction to the vip Package. R J. 2020;12:343. 32. DeVries Z, Hoda M, Rivers CS, Maher A, Wai E, Moravek D, et al.

Development of an unsupervised machine learning algorithm for the prognostication of walking ability in spinal cord injury patients. Spine J. 2020;20:213–24. 33. Rowald A, Komi S, Demesmaeker R, Baaklini E, Hernandez-Charpak SD, Paoles E, et al. Activity-dependent spinal cord neuromodulation rapidly restores trunk and leg motor functions after complete paralysis. Nat Med. 2022;28:260–71. 34. Boakye M, Ball T, Dietz N, Sharma M, Angeli C, Rejc E, et al. Spinal cord epidural stimulation for motor and autonomic function recovery after chronic spinal cord injury: A case series and technical note. Surg Neurol Int. 2023;14:87. 35.

Wagner FB, Mignardot JB, Le Goff-Mignardot CG, Demesmaeker R, Komi S, Capogrosso M, et al. Targeted neurotechnology restores walking in humans with spinal cord injury. Nature. 2018;563:65–71. 36. Smith AC, Weber KA, Parrish TB, Hornby TG, Tysseling VM, McPherson JG, et al. Ambulatory function in motor incomplete spinal cord injury: a magnetic reso- nance imaging study of spinal cord edema and lower extremity muscle mor- phometry. Spinal Cord. 2017;55:672–8. 37. Berliner JC, O'Dell DR, Albin SR, Dungan D, Sevigny M, Elliott JM, et al. The in uence of conventional T 2 MRI indices in predicting who will walk outside one year after spinal cord injury. J Spinal Cord Med. 2023;46:501–7. 38.

Smith AC, Draganich C, Thornton WA, Berliner JC, Lennarson PJ, Rejc E, et al. A Single Dermatome Clinical Prediction Rule for Independent Walking 1 Year After Spinal Cord Injury. Arch Phys Med Rehabil. 2023 Jul;S000399932300374X. ACKNOWLEDGEMENTS We would like to acknowledge the X Hospital Foundation, and the patients, families, and staff of the hospitals included in this study. AUTHOR CONTRIBUTIONS CD was responsible for study design, interpreting study results, and writing the report. DA was responsible for study design, interpreting study results, and writing the report. GD was responsible for machine learning analysis, interpreting results, and writing the report.

MS was responsible for analyzing data, interpreting results, and writing the report. JB was responsible for study design and writing the report. SC was responsible for writing the report. AW was responsible for data extraction and writing the report. AS was responsible for study design, interpreting study results, and writing the report. FUNDING This work was supported by the X Hospital Foundation under grant number XXX. COMPETING INTERESTS The authors declare no competing interests. ETHICS APPROVAL AND CONSENT TO PARTICIPATE This study was approved by the hospital institutional review board. Since this was a retrospective study, informed consent was not obtained.

ADDITIONAL INFORMATION Supplementary information The online version contains supplementary material available at https://doi.org/10.1038/s41393-024-01008-2. Correspondence and requests for materials should be addressed to Christina Draganich. Reprints and permission information is available at http://www.nature.com/ reprints Publisher's note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional af liations. Springer Nature or its licensor (e.g.

a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law. C. Draganich et al. 453 Spinal Cord (2024) 62:446 – 453