Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy

Published:November 25, 2020DOI:


      • Faculty use distinct terminology when describing high vs low entrustment level behaviors.
      • Topic modeling can discriminate between surgical EPA entrustment levels.
      • Topics generated by LDA map coherently to EPA entrustment levels.



      Entrustable Professional Activities (EPAs) contain narrative ‘entrustment roadmaps’ designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice.


      All text comments associated with EPA microassessments at a single institution were combined. EPA—entrustment level pairs (e.g. Gallbladder Disease—Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters.


      Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics).


      LDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to The American Journal of Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Greenberg J.
        • Minter R.
        Entrustable professional activities: the future of competency-based education in surgery may already Be here.
        Ann Surg. 2019; 269: 407-408
        • Brasel K.J.
        • Klingensmith M.E.
        • Englander R.
        • et al.
        Entrustable professional activities in general surgery: development and implementation.
        J Surg Educ. 2019; 76: 1174-1186
        • Stahl C.C.
        • Collins E.
        • Jung S.A.
        • et al.
        Implementation of entrustable professional activities into a general surgery residency.
        J Surg Educ. February 8, 2020; (Published online)
        • Grün B.
        • Hornik K.
        Topicmodels: an R package for fitting topic models.
        J Stat Software. 2011; 40: 1-30
        • Gross A.
        • Murthy D.
        Modeling virtual organizations with Latent Dirichlet Allocation: a case for natural language processing.
        Neural Network. 2014; 58: 38-49
      1. Cambria E, White B. Jumping NLP Curves: A Review of Natural Language Processing Research. :10.

        • Blei D.M.
        • Ng A.Y.
        • Jordan M.I.
        Latent dirichlet allocation.
        J Mach Learn Res. 2003; 3: 993-1022
        • Robinson JS. and D.
        Text mining with R.
        Date accessed: April 7, 2020
        • Feinerer I.
        • Hornik K.
        • Meyer D.
        Text mining infrastructure in R.
        J Stat Software. 2008; 25: 1-54
        • Erkan G.
        • Radev D.R.
        LexRank: graph-based lexical centrality as salience in text summarization.
        J Artif Intell Res. 2004; 22: 457-479
        • Mihalcea R.
        • Tarau P.
        TextRank: bringing order into texts.
        Proc 2004 Conf Empir Methods Nat Lang Process. July 2004: 404-411 (Published online)