LISTENING TO THE DIPLOMATES: PHYSICIANS’ FEEDBACK ON SELF-ASSESSMENT MODULES

Michele Mason; Jason Rinaldo

doi:10.1370/afm.398

The Self-Assessment Module (SAM) is an integral part of the ABFM’s Maintenance of Certification for Family Physicians (MC-FP) process as well as an important learning opportunity. Contrary to what some may think, the SAMs are not tests, but rather self-evaluations and self-teaching experiences. A SAM consists of 2 parts: a knowledge assessment and a clinical simulation. Once a Diplomate completes the knowledge assessment portion for the first time, the missed questions will appear on second and subsequent tries with both references and critiques, making the SAM much more of a learning process than any type of examination. Whenever a physician completes a SAM, we solicit feedback regarding their experience to continually improve these modules. The results of this feedback for our first 4 modules reflect a strong positive response. In the areas of (1) relevance of information to clinical practice, (2) currency of information, (3) usefulness of information, and (4) overall value of the module, the average ratings fall at 5.4 on a 6 point scale (Figures 1⇓–4⇓⇓⇓).

Figure 1.

Physicians’ rating of the hypertension self-assessment module.

Figure 2.

Physicians’ rating of the coronary artery disease self-assessment module.

Figure 3.

Physicians’ rating of the diabetes self-assessment module.

Figure 4.

Physicians’ rating of the asthma self-assessment module.

Perhaps the most important feedback indicator is whether the information relayed in these educational modules effects positive outcomes by improving practice. In the case of the first 2 health topics offered in 2004, for which we have at least 1 year’s record of physician evaluation, 54% of those taking the diabetes module indicated that they would change their practice as a result of participating. Ninety percent of those physicians volunteered written comments describing what changes they would make. For the hypertension module these values were within 1% of the diabetes module (nearly identical), providing a strong suggestion that both are high-quality educational materials.

These results bear out the goals established in the development of the SAMs as the primary facet of Part II of MC-FP, Self-Assessment and Lifelong Learning, namely, to enhance family physicians’ knowledge and skills in areas that are of greatest importance to them and to provide continuous opportunity for improvement in the quality of care they deliver to the American public.

ANY COMMENTS?

Physicians are also offered the opportunity to make an open response comment at the end of the SAM evaluation. The ABFM has used this feedback—both positive and negative—to modify and update the SAM during the first 18 months to improve both user comfort and the functionality of the SAM itself. Examples of physicians’ comments regarding the SAM’s relevance as an educational tool are as follows:

“I think that this is an excellent tool for assessing and attaining knowledge in a non-biased format. Excellent update on clinical guidelines, as well as a review on the important basic science aspects of diabetes. I was very impressed with the whole set.”

“The knowledge assessment portion was quite decent and appropriate. Clearly an educational activity.”

”I think improvements have already been made since I took the SAM on diabetes last year. I really liked the way the reference came up with the question in the review instead of having to shift from page to page. That reinforces learning in a friendly way.”

”It was excellent ... the immediate critique of missed answers was most helpful.”