Crowdsourced assessment of surgical skills: A systematic review


      • Skill assessment requires the presence of a senior surgeon which is time-consuming.
      • Crowdsourced assessment is a process that utilizes a group of untrained individuals.
      • Crowd workers can assess surgical procedures with high correlation to experts.
      • In general, crowd workers were faster to give feedback than experts.



      Crowdsourced assessment utilizes a large group of untrained individuals from the general population to solve tasks in the medical field. The aim was to examine the correlation between crowd workers and expert surgeons for the use of crowdsourced assessments of surgical skills.

      Material and methods

      A systematic literature review was performed on April 14th, 2021 from inception to the present. Two reviewers screened all articles with eligibility criteria of inclusion and assessed for quality using The Medical Education Research Study Quality Instrument (MERSQI) and Newcastle-Ottawa Scale-Education (NOS-E)(Holst et al., 2015).7General information was extracted for each article.


      250 potential studies were identified, and 32 articles were included. There appeared to be a generally moderate to very strong correlation between crowd workers and experts (Cronbach's alpha 0.72–0.95, Pearson's r 0.7–0.95, Spearman Rho 0.7–0.89, linear regression 0.45–0.89). Six studies had either questionable or no significant correlation between crowd workers and experts.


      Crowdsourced assessment can provide accurate, rapid, cost-effective, and objective feedback across different specialties and types of surgeries in dry lab, simulation, and live surgeries.



      CW (Crowd workers), AMT ( Mechanical Turk), PROSPERO (Prospective Register of Systematic Reviews), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analysis), MERSQI (The Medical Education Research Study Quality Instrument), NOS-E (Newcastle-Ottawa Scale-Education), GOALS (Global Operative Assessment of Laparoscopic Skills), CVS (Critical View of Safety), R-OSATS (Robotic-Objective Structured Assessment of Technical Skills), GEARS (Global Evaluative Assessment of Robotic Skills), mGEARS (modified GEARS), RACE (Robotic Anastomosis and Competency Evaluation), OSATS (Objective Structured Assessment of Technical Skills), GRS (Global Rating Scale), mOSATS (modified OSATS), APM (Automated Performance Metrics), PULS (Post-Ureteroscopic Lesion Scale)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to The American Journal of Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Lendvay T.S.
        • White L.
        • Kowalewski T.
        Crowdsourcing to assess surgical skill.
        JAMA Surg. 2015 Nov 1; 150: 1086
        • Birkmeyer J.D.
        • Finks J.F.
        • O'Reilly A.
        • et al.
        Surgical skill and complication rates after bariatric surgery.
        N Engl J Med. 2013 Oct 10; 369: 1434-1442
        • Goldenberg M.G.
        • Goldenberg L.
        • Grantcharov T.P.
        Surgeon performance predicts early continence after robot-assisted radical prostatectomy.
        J Endourol. 2017 Sep 1; 31: 858-863
        • Govaerts M.J.B.
        • Schuwirth L.W.T.
        • van der Vleuten C.P.M.
        • Muijtjens A.M.M.
        Workplace-based assessment: effects of rater expertise.
        Adv Health Sci Educ. 2011 May; 16: 151-165
        • Aghdasi N.
        • Bly R.
        • White L.W.
        • Hannaford B.
        • Moe K.
        • Lendvay T.S.
        Crowd-sourced assessment of surgical skills in cricothyrotomy procedure.
        J Surg Res. 2015; 196: 302-306
        • Dai J.C.
        • Lendvay T.S.
        • Sorensen M.D.
        Crowdsourcing in surgical skills acquisition: a developing technology in surgical education.
        J Grad Med Educ. 2017; 9: 697-705
        • Holst D.
        • Kowalewski T.M.
        • White L.W.
        • et al.
        Crowd-sourced assessment of technical skills: an adjunct to urology resident surgical simulation training.
        J Endourol. 2015; 29 (5 PG-604–9):604–9. Available from: NS
        • White L.W.
        • Kowalewski T.M.
        • Dockter R.L.
        • Comstock B.
        • Hannaford B.
        • Lendvay T.S.
        Crowd-sourced assessment of technical skill: a valid method for discriminating basic robotic surgery skills.
        J Endourol. 2015; 29 (11 PG-1295–301):1295–301. Available from: NS -
        • Vernez S.L.
        • Huynh V.
        • Osann K.
        • Okhunov Z.
        • Landman J.
        • Clayman R.V.
        C-SATS: assessing surgical skills among urology residency applicants.
        J Endourol. 2017; 31 (S1 PG-S95-S100):S95–100. Available from: NS -
        • Holst D.
        • Kowalewski T.
        • Comstock B.
        • et al.
        Crowd-Sourced Assessment of Technical Skills: a novel method to evaluate surgical performance.
        J Surg Res. 2013; 187: 65-71
        • Holst D.
        • Kowalewski T.M.
        • White L.W.
        • et al.
        Crowd-sourced assessment of technical skills: differentiating animate surgical skill through the wisdom of crowds.
        J Endourol. 2015; 29 (10 PG-1183–8):1183–8. Available from: NS -
        • Oh P.J.
        • Chen J.
        • Hatcher D.
        • Djaladat H.
        • Hung A.J.
        Crowdsourced versus expert evaluations of the vesico-urethral anastomosis in the robotic radical prostatectomy: is one superior at discriminating differences in automated performance metrics?.
        J Robot Surg. 2018; 12 (4 PG-705–711):705–11. Available from: NS
        • Créquit P.
        • Mansouri G.
        • Benchoufi M.
        • Vivot A.
        • Ravaud P.
        Mapping of crowdsourcing in health: systematic review.
        J Med Internet Res. 2018; 20 (Journal of Medical Internet Research)
        • Chen S.P.
        • Kirsch S.
        • Zlatev D.V.
        • et al.
        Optical Biopsy of Bladder Cancer Using Crowd-Sourced Assessment. vol. 151. JAMA Surgery. American Medical Association, 2016: 90-93
        • Nguyen T.B.
        • Wang S.
        • Anugu V.
        • et al.
        Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography.
        Radiology. 2012 Mar; 262: 824-833
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        J Clin Epidemiol. 2009 Oct 1; 62: 1006-1012
        • Cook D.A.
        • Reed D.A.
        Appraising the quality of medical education research methods: the medical education research study quality instrument and the newcastle-ottawa scale-education.
        Acad Med. 2015 Aug 31; 90: 1067-1076
        • Landis J.R.
        • Koch G.G.
        The measurement of observer agreement for categorical data.
        Biometrics. 1977 Mar 1; 33: 159
        • George D.
        • Paul Mallery W.
        IBM SPSS statistics 25 step by step: a simple guide and reference.
        in: Fifteenth. Routledge Taylor & Francis Group, 2019: 244
        • Hu Y.
        • Jiang B.
        • Kim H.
        • Schroen A.T.
        • Smith P.W.
        • Rasmussen S.K.
        Vessel ligation fundamentals: a comparison of technical evaluations by crowdsourced nonclinical personnel and surgical faculty.
        J Surg Educ. 2018; 75 (3 PG-664–670):664–70. Available from: NS
        • Goldenberg M.
        • Ordon M.
        • D’A Honey J.R.
        • Andonian S.
        • Lee J.Y.
        Objective assessment and standard setting for basic flexible ureterorenoscopy skills among urology trainees using simulation-based methods.
        J Endourol. 2020; 34 (4 PG-495–501):495–501. Available from: NS
        • Kowalewski T.M.
        • Comstock B.
        • Sweet R.
        • et al.
        Crowd-sourced assessment of technical skills for validation of basic laparoscopic urological skills (BLUS) tasks.
        Can Urol Assoc J. 2016; 10 (5-6 Supplement 1 PG-S60):S60. Available from:
        • Mahmood O.
        • Dagnæs J.
        • Bube S.
        • Rohrsted M.
        • Konge L.
        Nonspecialist raters can provide reliable assessments of procedural skills.
        J Surg Educ. 2018 Mar 1; 75: 370-376
        • Karani R.
        • Tapiero S.
        • Jefferson F.A.
        • et al.
        Crowd-sourced assessment of surgical skills of urology resident applicants: four-year experience.
        J Surg Educ. 2021; 78: 2030-2037
        • Chan Y.H.
        Biostatistics 104: correlational analysis.
        Singap Med J. 2003; 44: 614-619
        • Paley G.L.
        • Grove R.
        • Sekhar T.C.
        • et al.
        Crowdsourced assessment of surgical skill proficiency in cataract surgery.
        J Surg Educ. 2021; 78 (PG-). Available from: NS: 1077-1088
        • Lee J.Y.
        • Andonian S.
        • Pace K.T.
        • Grober E.
        Basic laparoscopic skills assessment study: validation and standard setting among Canadian urology trainees.
        J Urol. 2017; 197 (6 PG-1539–1544):1539–44. Available from: NS
        • Deal S.B.
        • Lendvay T.S.
        • Haque M.I.
        • et al.
        Crowd-sourced assessment of technical skills: an opportunity for improvement in the assessment of laparoscopic surgical skills.
        Am J Surg. 2016; 211: 398-404
        • Malpani A.
        • Vedula S.S.
        • Chen C.C.G.
        • Hager G.D.
        A study of crowdsourced segment-level surgical skill assessment using pairwise rankings.
        Int J Comput Assist Radiol Surg. 2015; 10 (9 PG-1435–47):1435–47. Available from: NS
        • Ghani K.R.
        • Miller D.C.
        • Linsell S.
        • et al.
        Measuring to improve: peer and crowd-sourced assessments of technical skill with robot-assisted radical prostatectomy.
        Eur Urol. 2015; 69: 547-550
        • Powers M.
        • Boonjindasup A.
        • Pinsky M.
        • et al.
        Crowdsourcing assessment of surgeon dissection of renal artery and vein during robotic partial nephrectomy: a novel approach for quantitative assessment of surgical performance.
        J Urol. 2016; (4 SUPPL. 1 PG-e114):e114. Available from: NS: 195
        • Kaler K.S.
        • Valley Z.A.
        • Bettir K.C.
        • et al.
        Crowdsourcing evaluation of ureteroscopic videos using the post-ureteroscopic lesion scale to assess ureteral injury.
        J Endourol. 2018; 32 (4 PG-275–281):275–81. Available from: NS
        • Deal S.B.
        • Stefanidis D.
        • Telem D.
        • et al.
        Evaluation of crowd-sourced assessment of the critical view of safety in laparoscopic cholecystectomy.
        Surg Endosc. 2017; 31 (12 PG-5094–5100):5094–100. Available from: NS
        • Conti S.L.
        • Brubaker W.
        • Chung B.I.
        • et al.
        Crowdsourced assessment of ureteroscopy with laser lithotripsy video feed does not correlate with trainee experience.
        J Endourol. 2019; 33 (1 PG-42–49):42–9. Available from: NS
        • Ratner B.
        The correlation coefficient: its values range between 1/1, or do they. J Targeting.
        Meas Anal Mark. 2009 May 18; 17: 139-142
        • Fukuoka K.
        • Teishima J.
        • Inoue S.
        • Hayashi T.
        • Matsubara A.
        The influence of reviewer’s occupation on the skill assessment of urethrovesical anastomosis in robot-assisted radical prostatectomy.
        Asian J Endosc Surg. 2020; 14 (PG-). Available from: NS: 451-457
        • Bendre H.H.
        • Rajender A.
        • Barbosa P.V.
        • Wason S.E.L.
        Robotic dismembered pyeloplasty surgical simulation using a 3D-printed silicone-based model: development, face validation and crowdsourced learning outcomes assessment.
        J Robot Surg. 2020; 14 (6 PG-897–902):897–902. Available from: NS
        • Addison P.
        • Yoo A.
        • Duarte-Ramos J.
        • et al.
        Correlation between Operative Time and Crowd-Sourced Skills Assessment for Robotic Bariatric Surgery.
        Surg Endosc [Internet, 2020 (PG-). Available from: NS
        • Deal S.B.
        • Scully R.E.
        • Wnuk G.
        • George B.C.
        • Alseidi A.A.
        Crowd-sourced and attending assessment of general surgery resident operative performance using global ratings scales.
        J Surg Educ. 2020; 77 (6 PG-214–219):e214–9. Available from: NS
        • Chen C.
        • White L.
        • Kowalewski T.M.
        • et al.
        Crowd-Sourced Assessment of Technical Skills: a novel method to evaluate surgical performance.
        J Surg Res. 2013; 187 (1 PG-65–71):65–71. Available from: NS
        • Ershad M.
        • Rege R.
        • Fey A.M.
        Meaningful assessment of robotic surgical style using the wisdom of crowds.
        Int J Comput Assist Radiol Surg. 2018 Jul 1; 13: 1037-1048
        • Rice M.K.
        • Zenati M.S.
        • Novak S.M.
        • et al.
        Crowdsourced assessment of inanimate biotissue drills: a valid and cost-effective way to evaluate surgical trainees.
        J Surg Educ. 2019; 76 (3 PG-814–823):814–23. Available from: NS
        • Deal S.B.
        • Alseidi A.A.
        Concerns of quality and safety in public domain surgical education videos: an assessment of the critical view of safety in frequently used laparoscopic cholecystectomy videos.
        J Am Coll Surg. 2017; 225 (6 PG-725–730):725–30. Available from: NS
        • Kelly J.D.
        • Petersen A.
        • Lendvay T.S.
        • Kowalewski T.M.
        The effect of video playback speed on surgeon technical skill perception.
        Int J Comput Assist Radiol Surg. 2020; 15 (5 PG-739–747):739–47. Available from: NS
        • Goldenberg M.G.
        • Nabhani J.
        • Wallis C.J.D.
        • et al.
        Feasibility of expert and crowd-sourced review of intraoperative video for quality improvement of intracorporeal urinary diversion during robotic radical cystectomy.
        Can Urol Assoc J. 2017; 11 (10 PG-331–336):331–6. Available from: NS
        • Kelly J.D.
        • Petersen A.
        • Lendvay T.S.
        • Kowalewski T.M.
        Bidirectional long short-term memory for surgical skill classification of temporally segmented tasks.
        Int J Comput Assist Radiol Surg. 2020 Dec 1; 15: 2079-2088
        • Polin M.R.
        • Siddiqui N.Y.
        • Comstock B.A.
        • et al.
        Crowd sourcing: a valid alternative to expert evaluation of robotic surgery skills.
        Female Pelvic Med Reconstr Surg. 2016; 21 (5 SUPPL. 1 PG-19–20):S19–20. Available from: NS
        • Martino M.A.
        • Siddiqui N.Y.
        • Polin M.R.
        • et al.
        Crowdsourcing: a valid alternative to expert evaluation of robotic surgery skills.
        Am J Obstet Gynecol. 2016; 215 (644.e1-644.e7)