Alignment and analytic quality of two interprofessional collaboration tools for a residency training evaluation

Context: Many medical schools are retooling their training programs to cultivate competencies in interprofessional collaboration (IPC) for patient-centered, team-based collaborative care and have a need to evaluate program impact.

Objective: Determine the appropriateness of two self-report instruments used to evaluate a residency training program’s impact on IPC.

Study Design: To assess alignment between the instruments and program goals, a survey matrixing approach compared each instrument’s targeted constructs (evidenced in technical reports, validating literature and item content) to stated evaluation needs. To assess the instruments’ ability to produce informative data, a Rasch measurement approach was used.

Dataset: Survey responses were analyzed from the Interprofessional Collaborative Competencies Attainment Survey (ICCAS) and the Attitudes Towards Health Care Teams (AtHCT).

Population studied: Over 20 residents at varying training stages.

Outcome Measures: Comparison of instrument items to IPC-related outcomes; Rasch analysis of item measurement functioning.

Results: The ICCAS is a well evidenced assessment of change in IPC behaviors. Alignment with program goals was high. However, Rasch analysis revealed that item difficulty was low given residents’ ability levels. Roughly half of all respondents reported an ability level above the targeted ability level of the most difficult item. Also, a lack of “randomness” in responses (e.g., over-selection of “Moderately agree” response option) indicated survey fatigue or lack of engagement. The extant literature also supported the AtHCT’s validity in assessing change in attitudes toward working in healthcare teams. However, the instrument was not well-aligned with our training program’s goal of increasing confidence and abilities to work in interprofessional teams—rather than interest in doing so. Rasch analysis revealed that while some subscales performed better than others, there was again issues with item targeting (e.g., in pre-intervention, over 50% of students strongly endorsed all items on the “Quality of Care” subscale) and possible survey fatigue.

Conclusion: Despite having a strong literature base, one instrument was misaligned with evaluation goals and neither allowed informative programmatic data. Revisions to the evaluation plan have been ongoing, including use of focus groups and external raters (rather than survey of residents themselves) to assure training on best processes.