| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH |
|
|
|
|||||||
|
|||||||||
TRACK to:
|
|
Electronic letters published:
![[Read Comment]](/icons/misc/sectionDOWN.gif){kind=icon}
More questions than answers—evidence of study value
Trained Evaluators Increase Validity of Outcomes
Unique perspective of standardized patients
Important Standardized Patient Research
|
|
|||
|
Larry B. Mauksch, Seattle Senior Lecturer, University of Washington
Send response to journal:
|
I want to thank Dr. Fiscella et al for this stimulating study. Your discussion was provocative and highlighted the infancy of the field by generating many questions. How does the value of assessment method vary as a function of assessment purpose—formative or summative? In high stakes (summative) situations, medical students, residents or practicing physicians might be judged to achieve a desired level of competency. This may lead to scholastic advancement or licensure. For these uses your research suggests that including unannounced standardized patient feedback has significant benefit over using only real patient feedback. But is using unannounced SPs, even at the much lower numbers than real patient feedback methods require, practical in the real world? How much accuracy is lost and money saved by using announced SPs? One variant of survey use is the commonly used approach of calculating percent of excellent responses. Organizations like the American Medical Group Association use this method of scoring, with very few global questions, to deal with psychometric challenges that you identify. While this method of scoring may not be applied to the highest stakes situations—advancement or licensure—it is used by medical groups to determine salary incentive (pay for performance at the individual level) and in extreme cases, job retention. Would this scoring method applied to instruments such as the HCCQ or the PPPC increase psychometric strength? Equally important is the use of assessment to improve physician communication-formative feedback? As you suggest, using unannounced SPs is more problematic if not unworkable, and some earlier research by some of you suggests that announced SPs has more advantage1. How can we use feedback from SPs and/or patients to improve physician performance in chronic illness care and mental illness care? A recent review suggests we have a lot to learn2. What other methods of formative feedback might be as effective or even more effective such as feedback from trained peers? While some feedback approaches can be shown to improve patient satisfaction, can feedback on physician communication be shown to improve health outcomes? 1. Epstein RM, Levenkron JC, Frarey L, Thompson J, Anderson K, Franks P. Improving physicians' HIV risk-assessment skills using announced and unannounced standardized patients. J Gen Intern Med. Mar 2001;16(3): 176-180. 2. Evans RG, Edwards A, Evans S, Elwyn B, Elwyn G. Assessing the practising physician using patient surveys: a systematic review of instruments and feedback methods. Fam Pract. Jan 29 2007. Competing interests: None declared |
|||
|
|
|||
|
Lisa D Howley, Charlotte, NC USA Association of Standardized Patient Educators (ASPE)
Send response to journal:
|
I very much enjoyed reading your article. These findings highlight an important (and often disregarded) aspect of standardized patient methodologies, that “trained evaluators” can more accurately measure performance than laypersons. As you and others demonstrate, training can directly influence the validity of an assessment and the outcome of research (Woehr, 1994; Heine, Garman, Wallace, Bartos, & Richards, 2003; Wallace, 2007). I also want to thank you for promoting the use of multiple methods to evaluate the construct of health care climate. Such complex constructs require multiple, mixed and higher order methods of assessment to support valid interpretations (Howley, 2004). On behalf of the Association of Standardized Patient Educators (ASPE), I commend your efforts and call on all researchers to increase the quality of delivery and reporting of rater training. Specifically, we request that all authors increase their transparency regarding what training methods were used to reduce rater errors and what quality control measures were used throughout the administration to increase rater accuracy. This information will ensure increased objectivity and enhanced measurement outcomes. Thank you for this important SP research! Heine, N., Garman, K., Wallace, P., Bartos, R., & Richards, A. (2003). An analysis of standardized patient checklist errors and their effect on students' scores. Medical Education, 37, 990-104. Howley, L.D. (2004). Assessment of clinical competence in medical education: Where we’ve been and where we’re going. Evaluation and the Health Professions, 27 (3), 285-303. Wallace, P. (2007). Coaching Standardized Patients for Use in the Assessment of Clinical Competence. Springer Publishing Company: New York, NY. Woehr, D.J., Huffcutt, A.I.(1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189-205. Competing interests: None declared |
|||
|
|
|||
|
Kevin Fiscella, Rochester, USA University of Rochester
Send response to journal:
|
We very much appreciate the comments from Dr Carney who has conducted much of the seminal research using standardized patients (SPs). Our own work has convinced us of the value of SP ratings of care and hope that trained SPs eventually become an additional component of clinician performance evaluation. Competing interests: None declared |
|||
|
|
|||
|
Patricia A. Carney, Portland, OR Professor of Family Medicine, OHSU
Send response to journal:
|
The study by Dr. Ficella and colleagues contributes importantly to research on the use of standardized patients (SPs). This methodology provides information that is difficult if not impossible to get using other approaches. Because communication is a hallmark of pimary care, comparing standardized patient ratings with real patient ratings is quite novel. Initially, I expected the ratings would be similar but after reading how different they appear to be, it further strengthens the methodology in my mind. As the authors point out, SPs are more objective and are trained to evaluate aspects of communication on a typical checklist. It is easy to see how this training might have influenced their completion of a standardized assessment instrument (e.g., HCCQ). In addition, they are likely not as influenced by the halo effect, which may explain their lower overall ratings compared to real patients. Importantly, the psychometric properties of the SP ratings were quite high indicating the strengh of an "instrument" that walks and talks, with alpha coeffecients similar to (if not higher than many) standardized ratings of paper and pencil instruments. Excellent work by these investigators!!! Competing interests: None declared |
|||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH |
TRACK: