•  
  •  
 

Impact of Structured Feedback on Examiner Judgements in Objective Structured Clinical Examinations (OSCEs) Using Generalisability Theory

Abstract

Purpose: In the context of health professions education, the objective structured clinical examination (OSCE) has been implemented globally for assessment of clinical competence. Concerns have been raised about the significant influence of construct irrelevant variance arising from examiner variability on the robustness of decisions made in high-stakes OSCEs. An opportunity to explore an initiative to reduce examiner effects was provided by a secondary analysis of data from a large-scale summative OSCE of the final-year students (n > 350) enrolled in a graduate-entry four-year Bachelor of Medicine/Bachelor of Surgery (MBBS) program at one Australian research-intensive university. The aim of this study was to investigate the impact of providing examiners with structured feedback on their stringency and leniency on assessing the final-year students’ clinical competence in the pre-feedback (P1) OSCE and post-feedback (P2) OSCE. Method: This study adopted a quasi-experimental design to analyse the scores from 141 examiners before feedback was provided for the P1 OSCE, and 111 examiners after feedback was provided for the P2 OSCE. This novel approach used generalisability theory to quantify and compare the examiner stringency and leniency variance (Vj) contributing to the examiners’ scores before and after feedback was provided. Statistical analyses conducted were controlled for differences in the examiners and OSCE stations. Results: Comparing the scores of the 51 examiners who assessed students in both P1 and P2 OSCEs, the Vj reduced by 35.65% and its contribution to the overall variation in their scores decreased by 7.43%. The results were more noticeable in the 26 examiners who assessed students in both OSCEs and in at least one station common across both OSCEs. The Vj reduced by 40.56% and its contribution to the overall variation in their scores was also decreased by 7.72%. Conclusion: The findings might be suggested that providing examiners with structured feedback could reduce the examiner stringency and leniency variation contributing to their scores in the subsequent OSCE, whilst noting limitations with the quasi-experimental design. More definitive research is required prior to making recommendations for practice. © 2020 King Saud bin Abdulaziz University for Health Sciences

This document is currently not available here.

Share

COinS