A MESSAGE FROM THE ABFM PRESIDENT

Unlinking the Clinical Simulation From the Knowledge Assessment in the SAMs

We have just completed an exhaustive review of all of the evaluations that our Diplomates provided after completion of their Performance in Practice Modules (PPMs) for Part IV and the SAMs for Part II. We shared with our Diplomates the preliminary results from the very positive feedback that was provided with respect to the PPMs last winter. This data has now been fully analyzed and a peer-reviewed manuscript has been accepted for publication this year. More importantly, our research staff has just finished a thorough analysis of the SAM data. Unlike the PPM data, however, this data was somewhat more concerning.

While very positive feedback on the knowledge assessment portion of the SAMs was apparent, the assessment of the utility of the clinical simulation component was less favorable. Not only were the quantitative evaluations significantly lower, the qualitative analysis of over 5 million open-ended feedback comments from 325,000 completed SAMs also revealed several important concerns about technical and navigation issues in the simulations. In an effort to determine whether Diplomate familiarity and periodic technical improvements had affected the ratings over time, we analyzed a second data set from almost 100,000 SAMs completed more recently in 2013 and 2014.

The findings were essentially unchanged. Diplomates consistently rated the knowledge assessments more favorably than the clinical simulations. The majority of the negative comments about the clinical simulations revolved around 4 major issues: difficulty in ordering or scheduling tasks; inadequate recognition of questions or language by the simulator; limited medication, treatment, and diagnostic options; and the lack of “realness” in the simulation environment. We provided this data to our Board of Directors for their review at their April 2015 and October 2015 meetings. Between those 2 meetings, our Clinical Simulation Team, led by Senior Vice President Michael Hagen, undertook the task of making several technical improvements to the clinical simulation interface.

While these changes resulted in improvement in the clinical simulation evaluations during this brief period of time, our Directors endorsed unlinking the clinical simulation and knowledge assessment components of each SAM, thereby making the clinical simulations optional, effective this year. Accordingly, the mandatory Part II requirement that at least 1 SAM be completed during each stage of MC-FP will be modified this year to mandate that at least 1 knowledge assessment, to be renamed Knowledge Self-Assessment (KSA), be completed in each stage in addition to at least 1 Part IV activity. In order to avoid the confusion of the multiple permutations of Part II and Part IV activities that could be combined to meet our stage requirements, we will also be transitioning all Diplomates to a point system to simplify how they meet their stage requirements.

Transitioning All Diplomates to the MC-FP Point System

Those Diplomates that have entered continuous MC-FP (those initially certifying or maintaining their certification in 2011 and thereafter) are by now very familiar with this system. For those that are not, it simply requires accumulation of 50 MC-FP points in each stage with completion of at least 1 Part IV activity and 1 Part II activity. That mandatory Part II activity will now be a KSA instead of a SAM (knowledge assessment plus clinical simulation); the SAM terminology will no longer be used. Most Part IV activities (but not all) are valued at 20 points; the KSAs will now be worth 10 points each, and the clinical simulations, to be renamed Clinical Self-Assessment (CSA), will be valued at 5 points. After completing at least 1 Part IV activity and 1 KSA, Diplomates will be able to mix and match any additional Part IV activities, KSAs, or CSAs to reach the 50-point requirement.

As might be suspected, these major changes for the MC-FP Part II requirement and conversion of all Diplomates to the MC-FP point system will require significant reprogramming of our website and individual physician portfolios. We will hope to have all of these changes in place by July 1, 2016.

Development of a New Continuous Knowledge Self-Assessment for Part II

Three major events occurring over the past 8 months have significantly influenced our plans to revise our Part II Platform for Maintenance of Certification for Family Physicians (MC-FP) beginning this year. The first was the testing summit held by the American Board of Pediatrics (ABP) late spring 2015. A number of assessment experts brought the meeting participants up to date with best practices in the clinical assessment arena; a single, high-stakes examination was not at the top of their list. The consensus opinion by the end of the summit was that frequent, smaller-scale assessment was a much more accurate way to measure knowledge and ability.

The second event was a presentation by the American Board of Anesthesiology (ABA) at the American Board of Medical Specialties (ABMS) meeting held in Chicago in June 2015. They provided results from their MOCA (Maintenance of Certification – Anesthesiology) Minute format that they have used for the past few years as their Part II platform. This format has been overwhelmingly popular with their Diplomates; it essentially pushes out a couple of questions per week and Diplomates have 1 minute each to answer. After submitting their answer, their Diplomates receive immediate feedback on whether they answered the question correctly and the rationale for the correct answer. The ABA has gathered very useful information on the knowledge gaps of their Diplomates, and they have shared this information with their professional society for purposes of CME development. This activity has been so successful that they are now embarking on a pilot that has been approved by the ABMS Committee on Continuous Certification (CCC) to use this same format to make summative decisions on the knowledge of their Diplomates for Part III. We will be watching the outcome of this pilot very closely.

The third event occurred in October 2015 when a blue ribbon task force created by the American Board of Internal Medicine (ABIM) released its white paper on Assessment 2020. This separate group of experts independently arrived at similar recommendations on assessing medical knowledge: multiple, frequent assessments over time are much better than a single, high stakes examination.

Armed with this information, we sent several members of our staff to the 2015 MOC Summit hosted by the ABA and ABMS at the ABA offices to learn more about the ABA’s experience with continuous longitudinal assessment. Thirteen other ABMS Member Boards attended and most were very interested in learning how this format could be used to replace their high stakes maintenance of certification (MOC) examinations. We know that several are considering moving in this direction, most notably the ABA and the ABP. However, after reviewing all of the information from the meeting, we do not believe this format is quite ready for prime time in Part III yet. The major obstacles in doing so are security, creation of sufficient secure assessment items, and uncertainty about the psychometric methodology that will underlie summative decision making.

Accordingly, our Board of Directors recently approved an alternative Part II activity that would use the MOCA Minute format and which we would use to provide targeted feedback to our Diplomates about their specific knowledge gaps as well as the probability that they would pass the MC-FP examination. This offering will allow Diplomates to choose completing KSAs or this alternative activity, which will be named Continuous Knowledge Self-Assessment (CKSA), to meet their MC-FP Part II requirements. We are planning to make this option available in early 2017. Over the course of the following 3 years, we hope to use data gathered from those Diplomates choosing this Part II offering to assess the feasibility of using this format to replace the current Part III examination.

Discounting the Cost of MC-FP for Diplomates Aged Over 70 Years

We have mentioned previously the significant number of Diplomates who continue to participate in MC-FP well into their 70s, 80s, and 90s despite the fact that they are no longer practicing. In recognition of the dedication and commitment that these Diplomates have made to our specialty, we have chosen to offer each of them a 50% discount on their MC-FP fees if they wish to continue to maintain their certification.

Transforming Clinical Practice Initiative

Before closing, we would be remiss in not mentioning 2 other important issues that will demand our attention this year. The first was our recent selection as 1 of 39 health care collaborative networks selected to participate in the federal Transforming Clinical Practice Initiative (TCPI). This initiative was designed to help physicians transform their practices to enhance care coordination and expand information sharing. We will partner with the American Academy of Family Physicians (AAFP) on this effort and will receive as much as $538,000 to help offer the tools, information, and network support needed to assist physicians improve the quality of care they provide, increase patients’ access to information, and ensure more judicious use of health care dollars. Our clinical data registry will be an integral part of our plan to strengthen quality of care and develop comprehensive quality improvement strategies for those participating in these networks.

Physician Burnout

The final, but not least important issue, is the increasing rate of physician burnout. We are currently collecting data to understand how prevalent this phenomenon is in board certified family physicians. The findings from the data that we collect will inform our decisions on how we can further enhance MC-FP to create added value and less burden for practicing family physicians in keeping with our promises that we have made with our Diplomates.