Original Research Article|Articles in Press

High-performance pediatric surgical risk calculator: A novel algorithm based on machine learning and pediatric NSQIP data



      New methods such as machine learning could provide accurate predictions with little statistical assumptions. We seek to develop prediction model of pediatric surgical complications based on pediatric National Surgical Quality Improvement Program(NSQIP).


      All 2012–2018 pediatric-NSQIP procedures were reviewed. Primary outcome was defined as 30-day post-operative morbidity/mortality. Morbidity was further classified as any, major and minor. Models were developed using 2012–2017 data. 2018 data was used as independent performance evaluation.


      431,148 patients were included in the 2012–2017 training and 108,604 were included in the 2018 testing set. Our prediction models had high performance in mortality prediction at 0.94 AUC in testing set. Our models outperformed ACS-NSQIP Calculator in all categories for morbidity (0.90 AUC for major, 0.86 AUC for any, 0.69 AUC in minor complications).


      We developed a high-performing pediatric surgical risk prediction model. This powerful tool could potentially be used to improve the surgical care quality.



      ACS-NSQIP (American College of Surgeons- National Surgical Quality Improvement Program), CPT (Current Procedural Terminology), ICD-CM (International Classification of Disease diagnosis), CCS (Clinical Classification Software), HCUP (Healthcare Cost and Utilization Project)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to The American Journal of Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Pollack M.M.
        • Patel K.M.
        • Ruttimann U.E.
        Prism III: an updated pediatric risk of mortality score.
        Crit Care Med. 1996; 24: 743-752
        • Pollack M.M.
        • Ruttimann U.E.
        • Getson P.R.
        Pediatric risk of mortality (PRISM) score.
        Crit Care Med. 1988; 16: 1110-1116
        • Kraemer K.
        • Cohen M.E.
        • Liu Y.
        • et al.
        Development and evaluation of the American College of Surgeons NSQIP pediatric surgical risk calculator.
        J Am Coll Surg. 2016; 223: 685-693
        • Schneider A.L.
        • Deig C.R.
        • Prasad K.G.
        • et al.
        Ability of the national surgical quality improvement Program risk calculator to predict complications following total laryngectomy.
        JAMA Otolaryngol Head Neck Surg. 2016; 142: 972-979
        • McCarthy M.H.
        • Singh P.
        • Nayak R.
        • et al.
        Can the American College of Surgeons risk calculator predict 30-day complications after spine surgery?.
        Spine (Phila Pa. 2020; 45 (1976): 621-628
        • Gadgil N.
        • Pan I.W.
        • Babalola S.
        • Lam S.
        Evaluating the national surgical quality improvement program-pediatric surgical risk calculator for pediatric craniosynostosis surgery.
        J Craniofac Surg. 2018; 29: 1546-1550
        • Nasr V.G.
        • DiNardo J.A.
        • Faraoni D.
        Development of a pediatric risk assessment score to predict perioperative mortality in children undergoing noncardiac surgery.
        Anesth Analg. 2017; 124: 1514-1519
        • Stey A.M.
        • Kenney B.D.
        • Moss R.L.
        • et al.
        A risk calculator predicting postoperative adverse events in neonates undergoing major abdominal or thoracic surgery.
        J Pediatr Surg. 2015; 50: 987-991
        • Tabbutt S.
        • Schuette J.
        • Zhang W.
        • et al.
        A novel model demonstrates variation in risk adjusted mortality across pediatric cardiac intensive care units after surgery.
        Pediatr Crit Care Med : J Soc Critical Cre Med World Federate Pediatric Intensive Critical Care Soc. 2019; 20: 136-142
        • Advanced Analytics Group of Pediatric U, Group ORCPM
        Targeted workup after initial febrile urinary tract infection: using a novel machine learning model to identify children most likely to benefit from voiding cystourethrogram.
        J Urol. 2019; 202: 144-152
        • Bertsimas D.
        • Li M.
        • Estrada C.
        • Nelson C.
        • Scott Wang H.H.
        Selecting children with vesicoureteral reflux who are most likely to benefit from antibiotic prophylaxis: application of machine learning to RIVUR.
        J Urol. 2021; 205: 1170-1179
        • Bertsimas D.
        • Dunn J.
        • Velmahos G.C.
        • Kaafarani H.M.A.
        Surgical risk is not linear: derivation and validation of a novel, user-friendly, and machine-learning-based predictive OpTimal trees in emergency surgery risk (POTTER) calculator.
        Ann Surg. 2018; 268: 574-583
        • Bertsimas D.
        • Kung J.
        • Trichakis N.
        • Wang Y.
        • Hirose R.
        • Vagefi P.A.
        Development and validation of an optimized prediction of mortality for candidates awaiting liver transplantation.
        Am J Transplant. 2019; 19: 1109-1118
        • Vagefi P.A.
        • Bertsimas D.
        • Hirose R.
        • Trichakis N.
        The rise and fall of the model for end-stage liver disease score and the need for an optimized machine learning approach for liver allocation.
        Curr Opin Organ Transplant. 2020; 25: 122-125
        • Cohen M.E.
        • Ko C.Y.
        • Bilimoria K.Y.
        • et al.
        Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus.
        J Am Coll Surg. 2013; 217 (336-46.e1)
        • Saito J.M.
        • Chen L.E.
        • Hall B.L.
        • et al.
        Risk-adjusted hospital outcomes for children's surgery.
        Pediatrics. 2013; 132: e677-e688
        • Quality AfHRa
        HCUP Overview.
        • Quality AfHRa
        HCUP CCS-Services and Procedures.
        • Quality AfHRa
        Tools Archive for Clinical Classifications Software Refined.
        • Dimick J.B.
        • Chen S.L.
        • Taheri P.A.
        • Henderson W.G.
        • Khuri S.F.
        • Campbell D.A.
        Hospital costs associated with surgical complications: a report from the private-sector National Surgical Quality Improvement Program.
        J Am Coll Surg. 2004; 199: 531-537
        • Bertsimas D.
        • Pawlowski C.
        • Zhuo Y.D.
        From predictive methods to missing data imputation: an optimization approach.
        J Mach Learn Res. 2018; 18: 1-39
        • Dimitris Bertsimas C.P.
        • Zhuo Ying Daisy
        From predictive methods to missing data imputation: an optimization approach.
        J Mach Learn Res. 2018; 18: 1-39
        • Bertsimas D.
        • Dunn J.
        Optimal classification trees.
        Mach Learn. 2017; 106: 1039-1082
        • Malik A.T.
        • Yu E.
        • Kim J.
        • Khan S.N.
        Intensive care unit admission following surgery for pediatric spinal deformity: an analysis of the ACS-NSQIP pediatric spinal fusion procedure targeted dataset.
        Global Spine J. 2020; 10: 177-182
        • Nasr V.G.
        • DiNardo J.A.
        • Faraoni D.
        Development of a pediatric risk assessment score to predict perioperative mortality in children undergoing noncardiac surgery.
        Anesth Analg. 2017; 124: 1514-1519
        • Tollinche L.E.
        • Yang G.
        • Tan K.S.
        • Borchardt R.
        Interrater variability in ASA physical status assignment: an analysis in the pediatric cancer setting.
        J Anesth. 2018; 32: 211-218