Judgment analysis: a method for quantitative evaluation of trainee surgeons’ judgments of surgical risk

Jacklin, Ros; Sevdalis, Nick; Harries, Clare; Darzi, Ara; Vincent, Charles

doi:10.1016/j.amjsurg.2007.09.031

« Previous Next »The American Journal of Surgery
Volume 195, Issue 2 , Pages 183-188, February 2008

Judgment analysis: a method for quantitative evaluation of trainee surgeons’ judgments of surgical risk

Ros Jacklin, M.B.B.S., M.A., M.R.C.S.
- Clinical Safety Research Unit, Department of Bio-Surgery and Surgical Technology, Imperial College London, 10th Floor, QEQM, St. Mary’s Hospital, South Wharf Rd., London, W2 1NY, UK
- Corresponding author. Tel.: +44-020-7886-6311; fax: +44-020-7886-1810.
Nick Sevdalis, B.Sc., M.Sc., Ph.D.
- Clinical Safety Research Unit, Department of Bio-Surgery and Surgical Technology, Imperial College London, 10th Floor, QEQM, St. Mary’s Hospital, South Wharf Rd., London, W2 1NY, UK
- National Patient Safety Agency, London, UK
Clare Harries, B.Sc., Ph.D.
- University College London, London, UK
Ara Darzi, M.B.B.S., M.D.
- Clinical Safety Research Unit, Department of Bio-Surgery and Surgical Technology, Imperial College London, 10th Floor, QEQM, St. Mary’s Hospital, South Wharf Rd., London, W2 1NY, UK
Charles Vincent, B.A., M.Phil., Ph.D.
- Clinical Safety Research Unit, Department of Bio-Surgery and Surgical Technology, Imperial College London, 10th Floor, QEQM, St. Mary’s Hospital, South Wharf Rd., London, W2 1NY, UK

Received 23 April 2007; received in revised form 25 September 2007 published online 10 December 2007.

Abstract

Background

Surgical judgment and decision making require valid methods of assessment. The aim of this study was to evaluate the use of judgment analysis as a technique for quantitative evaluation of surgeons’ risk estimates.

Methods

Thirty trainee surgeons’ estimates of conversion risk in laparoscopic cholecystectomy were investigated using judgment analysis. Hypothetical cases were created, differing in relevant risk factors. Twenty repeat cases were incorporated to test for reliability. Surgeons’ estimates were compared with an outcome-derived gold standard from the published literature.

Results

The mean reliability was .77 (range, .47–.98), and regression models indicating the weighting of variables had a mean adjusted R² value of .53 (range, .12–.76). Variables were subject to wide variation in weighting. The mean correlation to the gold standard model was .48 (range, .08–.72).

Conclusions

Judgment analysis allows detailed quantitative evaluation of the consistency of surgeons’ risk estimates and the influence of different variables on them. Comparison with a gold standard model enables accuracy to be measured.

Keywords: Judgment analysis, Surgery, Risk, Judgment, Decision making