Highlights
- •Comments provided on oral examination score sheets reflect qualities examiners considered when evaluating performance.
- •Oral examinations test qualities not readily captured by other assessment tools.
- •Assessment of cognitive processing distinguishes strong from weak performance.
Abstract
Background
Given the long tradition of oral examinations in surgical training, surgical clerkships
continue to use oral examinations to evaluate medical students even though the value
of oral examination at the post-graduate level has been questioned. The key issue
in the context of undergraduate surgical training then is to understand value of the
oral examination in assessment. The goal of this study is to clarify what oral examinations
do, or appear to, test and how this complements other methods of assessment.
Methods
The study is a retrospective, qualitative study of comments provided by examiners
on the oral examination score sheets evaluating performance of students completing
their core surgery clerkship at an academic medical center. Through immersion in and
initial familiarization with the data we develop a scheme of codes for labeling the
data for subsequent synthesis. Using these inductive codes, all comments were reviewed
and analyzed to determine what qualities examiners detect, or naturally comment on,
when administering and scoring the oral examinations.
Results
Thirteen substantive codes (Communication, Critical Thinking, Decisiveness, Demeanor,
Differential Diagnosis, Focus, Knowledge, Management, Organization, Pace, Prompting,
Thoroughness, and Work Up) and three valence codes (Negative, Neutral, and Positive)
were developed and used to code the data. The most universal code was ‘Knowledge’,
used by 43 (100%) of examiners; the most frequently used code was ‘Work Up’, applied
to the comments 437 (21.1%) times. Overall, positive valence was attached to 1146
(55.2%) of codes and negative valence to 879 (42.3%) codes. The most discriminating
codes in grading were ‘Demeanor’, ‘Focus’, and ‘Organization’.
Conclusions
Oral examinations provide rich opportunity for testing qualities readily tested on
other examinations but also many intangible qualities that are otherwise less well
or not well tested. As such, the ‘value-added’ by oral examinations likely justifies
their continued use in the evaluation of surgical trainees. The identification of
testable qualities should aid in the development of a standardized scoring rubric,
the use of which may aid in minimizing subjectivity and bias in what otherwise is
a rich assessment tool.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to The American Journal of SurgeryAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- The American Board of Surgery in the 20th Century – Then and Now.ABS Office of the Secretary, Philadelphia, PA2004
- The necessity, practicality, and feasibility of modern educational and evaluative methods for residency training: economic and governing body perspectives. The 1994 Association for Academic Surgery Panel on Education.Curr Surg. 1997; 54: 261-269
- (Last accessed 12/19/2019)
- Standardized oral examinations.(Last accessed 9/7/2011)
- Effect of ethnicity and gender on performance in undergraduate medical examinations.Med Educ. 2005; 39: 1126-1128
- Impact of student ethnicity and primary childhood language of communication skill assessment in a clinical performance examination.J Gen Intern Med. 2007; 22: 1150-1160
- The oral board examination for plastic surgery: seeking a better way.Med Teach. 2006; 28: 360-364
- A surgery oral examination: interrater agreement and the influence of rater characteristics.Acad Med. 1995; 70: 1044-1046
- Is the American board of surgery certifying examination archaic?.Curr Surg. 2004; 61: 405
- Carrying out qualitative analysis.in: Ritchie J. Lewis J. Qualitative Research Practice. SAGE Publications, London, UK2009: 219-262
- Qualitative research in Health Care: analysing qualitative data.BMJ. 2000; 320: 114-116
- Do global rating forms enable program directors to assess the ACGME competencies?.Acad Med. 2004; 79: 549-556
- Mini-clinical evaluation exercise as a student assessment tool in a surgery clerkship: lessons learned from a 5-year experience.Surgery. 2011; 150: 272-277
Article info
Publication history
Published online: January 02, 2020
Accepted:
December 30,
2019
Received in revised form:
December 19,
2019
Received:
October 24,
2019
Identification
Copyright
© 2020 Elsevier Inc. All rights reserved.
