Validation of the behavior and concept based assessment of professionalism competence in postgraduate first-year residents

Background: The evaluation of professional behaviors and concepts of postgraduate first-year (PGY₁) residents has been identified as an area for development. This study examined the efficiency of the professionalism-assessing objective structured clinical examination (OSCE), 360° evaluation, and mini-Clinical Examination Exercise scores (mini-CEX; p-OSCE, p-360° evaluation, and p-mini-CEX scores). Methods: Between January 2009 and January 2012, 189 PGY₁ residents were evaluated for behavior- and concept-based professionalism competence based on the above three methods using two checklists unique to each case. Data were analyzed for reliability, inter-rater agreement, interval changes, and gender-related difference for each method. Results: The test reliabilities of p-OSCE, p-360° evaluation, and p-mini-CEX were acceptable. Further, the reliability of concept and combined p-OSCE was higher than that of behavior p-OSCE. In addition, the concept OSCE p-scores and behavior 360° evaluation p-scores were significantly improved after 6 months of training. The inter-rater agreements were relatively good in p-OSCE and p-360° evaluation. Interestingly, male PGY₁ residents had higher behavior 360° evaluation p-scores from nurses than those of females, whereas female PGY₁ residents had higher behavior 360° evaluation p-scores from patients than those of males. Behavior and concept OSCE p-scores were positively correlated with behavior 360° evaluation p-scores. In comparison with p-360° evaluation, the combination of p-360° evaluation + OSCE + mini-CEX significantly increases their reliabilities. Conclusion: The current study suggests that the p-OSCE, p-360° evaluation, and p-mini-CEX are feasible methods for evaluating professionalism in clinical training of PGY₁ residents. Combination of the above three evaluations, participation, and support from multiple constituencies and multiple representatives provides good reliability and adds credibility in the assessment of professionalism competence.