Highlights
- •Faculty use distinct terminology when describing high vs low entrustment level behaviors.
- •Topic modeling can discriminate between surgical EPA entrustment levels.
- •Topics generated by LDA map coherently to EPA entrustment levels.
Abstract
Background
Entrustable Professional Activities (EPAs) contain narrative ‘entrustment roadmaps’
designed to describe specific behaviors associated with different entrustment levels.
However, these roadmaps were created using expert committee consensus, with little
data available for guidance. Analysis of actual EPA assessment narrative comments
using natural language processing may enhance our understanding of resident entrustment
in actual practice.
Methods
All text comments associated with EPA microassessments at a single institution were
combined. EPA—entrustment level pairs (e.g. Gallbladder Disease—Level 1) were identified
as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm,
was used to identify latent topics in the documents associated with a single EPA.
These topics were then reviewed for interpretability by human raters.
Results
Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for
80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels
(Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels
(words demonstrating high entrustment were consistently found in high entrustment
topics, word demonstrating low entrustment were found in low entrustment topics).
Conclusions
LDA is capable of identifying topics relevant to progressive surgical entrustment
and autonomy in EPA comments. These topics provide insight into key behaviors that
drive different level of resident autonomy and may allow for data-driven revision
of EPA entrustment maps.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to The American Journal of SurgeryAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Entrustable professional activities: the future of competency-based education in surgery may already Be here.Ann Surg. 2019; 269: 407-408https://doi.org/10.1097/SLA.0000000000003153
- Entrustable professional activities in general surgery: development and implementation.J Surg Educ. 2019; 76: 1174-1186https://doi.org/10.1016/j.jsurg.2019.04.003
- Implementation of entrustable professional activities into a general surgery residency.J Surg Educ. February 8, 2020; (Published online)https://doi.org/10.1016/j.jsurg.2020.01.012
- Topicmodels: an R package for fitting topic models.J Stat Software. 2011; 40: 1-30https://doi.org/10.18637/jss.v040.i13
- Modeling virtual organizations with Latent Dirichlet Allocation: a case for natural language processing.Neural Network. 2014; 58: 38-49https://doi.org/10.1016/j.neunet.2014.05.008
Cambria E, White B. Jumping NLP Curves: A Review of Natural Language Processing Research. :10.
- Latent dirichlet allocation.J Mach Learn Res. 2003; 3: 993-1022
- Text mining with R.https://www.tidytextmining.com/Date accessed: April 7, 2020
- Text mining infrastructure in R.J Stat Software. 2008; 25: 1-54https://doi.org/10.18637/jss.v025.i05
- LexRank: graph-based lexical centrality as salience in text summarization.J Artif Intell Res. 2004; 22: 457-479https://doi.org/10.1613/jair.1523
- TextRank: bringing order into texts.Proc 2004 Conf Empir Methods Nat Lang Process. July 2004: 404-411 (Published online)
Article Info
Publication History
Published online: November 25, 2020
Accepted:
November 21,
2020
Received in revised form:
November 19,
2020
Received:
May 18,
2020
Identification
Copyright
© 2020 Elsevier Inc. All rights reserved.
