Research Article| Volume 216, ISSUE 5, P835-840, November 2018

The hands and head of a surgeon: Modeling operative competency with multimodal epistemic network analysis

Published:November 30, 2017DOI:


      • This study examined how intraoperative errors affect surgical outcomes.
      • Surgery residents performed the final steps of a simulated hernia repair.
      • Neither the number nor types of errors committed predicted outcomes.
      • Differences in error management did correlate significantly with outcomes.



      This paper explores a method for assessing intraoperative performance by modeling how surgeons integrate psychomotor, procedural, and cognitive skills to manage errors.


      Audio-video data were collected from general surgery residents (N = 45) performing a simulated laparoscopic ventral hernia repair. Errors were identified using a standard checklist, and speech was coded for elements related to error recognition and management. Epistemic network analysis (ENA) was used to model the integration of error management skills.


      There was no correlation between number or type of errors committed and operative outcome. However, ENA models showed significant differences in the integration of error management skills between high-performing and low-performing residents.


      These results suggest that error checklists and surgeons' speech can be used to model the integration of psychomotor, procedural, and cognitive aspects of intraoperative performance. Moreover, ENA can identify and quantify this integration, providing insight on performance gaps in both individuals and populations.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to The American Journal of Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Makary M.A.
        • Daniel M.
        Medical error―The third leading cause of death in the US.
        Br Med J. 2016; 353
      1. Wiegmann DA. Analysis of error management strategies during cardiac surgery: theoretical and practical implications. Presented at the Thousand Oaks, CA.

        • Madani A.
        • Vassiliou M.C.
        • Watanabe Y.
        • et al.
        What are the principles that guide behaviors in the operating room? Creating a framework to define and measure performance.
        Ann Surg. 2017; 265: 255-267
        • DiSessa A.A.
        Knowledge in pieces.
        in: Forman G. Pufall P. Constructivism in the Computer Age. Erlbaum, Hillsdale, NJ1988: 47-70
        • Linn M.C.
        • Eylon B.-S.
        • Davis E.A.
        The knowledge integration perspective on learning.
        in: Linn M.C. Davis E.A. Bell P. Internet Environments for Science Education. Lawrence Erlbaum Associates, Mahwah, NJ2004: 29-46
        • Shaffer D.W.
        Models of situated action: computer games and the problem of transfer.
        in: Steinkuehler C. Squire K.D. Barab S.A. Games, Learning, and Society: Learning and Meaning in the Digital Age. Cambridge University Press, Cambridge, UK2012: 403-431
        • Greenberg C.C.
        • Ghousseini H.N.
        • Quamme S.R.P.
        • et al.
        Surgical coaching for individual performance improvement.
        Ann Surg. 2015; 261: 32-34
        • Glarner C.E.
        • McDonald R.J.
        • Smith A.B.
        • et al.
        Utilizing a novel tool for the comprehensive assessment of resident operative performance.
        J Surg Educ. 2013; 70: 813-820
        • Martin J.A.
        • Regehr G.
        • Reznick R.
        • et al.
        Objective structured assessment of technical skill (OSATS) for surgical residents.
        Br J Surg. 1997; 84: 273-278
        • Yule S.
        • Paterson-Brown S.
        Surgeons' non-technical skills.
        Surg Clin. 2012; 92: 37-50
        • Anderson D.D.
        • Long S.
        • Thomas G.W.
        • et al.
        Objective Structured Assessments of Technical Skills (OSATS) does not assess the quality of the surgical result effectively.
        Clin Orthop. 2016; 474: 874-881
        • D'Angelo A.-L.D.
        • Cohen E.R.
        • Kwan C.
        • et al.
        Use of decision-based simulations to assess resident readiness for operative independence.
        Am J Surg. 2015; 209: 132-139
        • Hatala R.
        • Cook D.A.
        • Brydges R.
        • Hawkins R.
        Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence.
        Adv Health Sci Educ. 2015; 20: 1149-1175
        • Schmitz C.C.
        • DaRosa D.
        • Sullivan M.E.
        • et al.
        Development and verification of a taxonomy of assessment metrics for surgical technical skills.
        Acad Med. 2014; 89: 153-161
        • Law Forsyth K.
        • DiMarco S.M.
        • Jenewein C.G.
        • et al.
        Do errors and critical events relate to hernia repair outcomes?.
        Am J Surg. 2017; 213: 652-655
        • Shaffer D.W.
        • Hatfield D.L.
        • Svarovsky G.N.
        • et al.
        Epistemic network analysis: a prototype for 21st century assessment of learning.
        Int J Leg Med. 2009; 1: 1-21
        • Shaffer D.W.
        • Collier W.
        • Ruis A.R.
        A tutorial on epistemic network analysis: analyzing the structure of connections in cognitive, social, and interaction data.
        J Learn Anal. 2016; 3: 9-45
        • Shaffer D.W.
        • Ruis A.R.
        Epistemic network analysis: a worked example of theory-based learning analytics.
        in: Lang C. Siemens G. Wise A.F. Gasevic D. Handbook of Learning Analytics. Society for Learning Analytics Research, 2017: 175-187
        • Shaffer D.W.
        Quantitative Ethnography.
        Cathcart Press, Madison, WI2017
        • Arastoopour G.
        • Shaffer D.W.
        • Swiecki Z.
        • et al.
        Teaching and assessing engineering design thinking with virtual internships and epistemic network analysis.
        Int J Eng Educ. 2016; 32: 1492-1501
      2. Quardokus Fisher K, Hirshfield L, Siebert-Evenstone AL, et al. Network Analysis of Interactions between Students and an Instructor during Design Meetings.

        • Chesler N.C.
        • Ruis A.R.
        • Collier W.
        • et al.
        A novel paradigm for engineering education: virtual internships with individualized mentoring and assessment of engineering thinking.
        J Biomech Eng. 2015; 137: 1-8
        • Bagley E.A.
        • Shaffer D.W.
        Stop talking and type: comparing virtual and face-to-face mentoring in an epistemic game.
        J Comput Assist Learn. 2015; 26: 369-393
        • Nash P.
        • Shaffer D.W.
        Mentor modeling: the internalization of modeled professional thinking in an epistemic game.
        J Comput Assist Learn. 2011; 27: 173-189
        • Hatfield D.L.
        The right kind of telling: an analysis of feedback and learning in a journalism epistemic game.
        Int J Gaming Comput-Mediat Simul. 2015; 7: 1-23
        • Wooldridge A.R.
        • Carayon P.
        • Eagan B.R.
        • Shaffer D.W.
        Quantifying the qualitative with epistemic network analysis: a human factors case study of task-allocation communication in a primary care team.
        IIE Trans Healthc Syst Eng. 2018; (in press)
        • Sullivan S.A.
        • Warner-Hillard C.
        • Eagan B.R.
        • et al.
        Using epistemic network analysis to identify targets for educational interventions in trauma team communication.
        Surgery. 2017; (in press)
        • Pugh C.
        • Plachta S.
        • Auyang E.
        • et al.
        Outcome measures for surgical simulators: is the focus on technical skills the best approach?.
        Surgery. 2010; 147: 646-654
        • Peters J.H.
        • Fried G.M.
        • Swanstrom L.L.
        • et al.
        Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery.
        Surgery. 2004; 135: 21-27
        • Glaser B.G.
        • Strauss A.L.
        The Discovery of Grounded Theory: Strategies for Qualitative Research.
        Aldine de Gruyter, 1967
      3. Eagan BR, Rogers B, Pozen R, et al. rhoR: Rho for Inter Rater Reliability. 2016.

      4. Eagan BR, Rogers B, Serlin R, et al. Can we rely on Reliability? Testing the assumptions of inter-rater reliability. Presented at the International Conference on Computer-supported Collaborative Learning, Philadelphia, PA.

        • Kuhn M.
        • Johnson K.
        Applied Predictive Modeling.
        Springer, 2013
        • Siebert-Evenstone A.L.
        • Arastoopour G.
        • Collier W.
        • et al.
        In search of conversational grain size: modeling semantic structure using moving stanza windows.
        in: Transforming Learning, Empowering Learners: The International Conference of the Learning Sciences (ICLS) 2016. vol. I. 2016: 631-638
        • Gardner A.K.
        • Abdelfattah K.
        • Wiersch J.
        • et al.
        Embracing errors in simulation-based training: the effect of error training on retention and transfer of central venous catheter skills.
        J Surg Educ. 2015; 72: e158-e162
        • Lorenzet S.J.
        • Salas E.
        • Tannenbaum S.I.
        Benefiting from mistakes: the impact of guided errors on learning, performance, and self-efficacy.
        Hum Resour Dev Q. 2005; 16: 301-322
        • Gully S.M.
        • Payne S.C.
        • Kiechel Koles K.L.
        • Whiteman J.-A.K.
        The impact of error training and individual differences on training outcomes: an attribute-treatment interaction perspective.
        J Appl Psychol. 2002; 87: 143-155
        • Dror I.
        A novel approach to minimize error in the medical domain: cognitive neuroscientific insights into training.
        Med Teach. 2011; 33: 34-38
        • Law K.E.
        • Ray R.D.
        • D'Angelo A.-L.D.
        • et al.
        Exploring senior residents' intraoperative error management strategies: a potential measure of performance improvement.
        J Surg Educ. 2016; 73: e64-e70