Evolving Certification to Meet Today’s Needs: The ABFM’s KSA Revision Initiative

Why Knowledge Self-Assessment?

The American Board of Medical Specialties (ABMS) implemented the Maintenance of Certification Program as a way to provide a comprehensive approach to lifelong learning, self-assessment, and quality improvement for physicians in all specialties.¹ As demonstrated in a systematic review of 62 studies, physician knowledge, skills, and adherence to evidence-based medicine tend to decline as a function of time from initial training.² In addition, there is robust evidence that physicians, like other experts, are inaccurate in their own self-assessment—we simply do not always realize what we don’t know.³ Thus, a key role of Board Certification is to provide a guided, objective, independent, self-assessment program that goes beyond traditional, self-selected continuing medical education (CME). Ongoing self-assessment also supports acquisition of new knowledge and skills as changes in medical knowledge necessitate changes in practice, and it reinforces a broad knowledge base when physicians’ scope of practice narrows over time. Participation in continuous certification empowers physicians to develop their own learning strategies supported by tools that can help direct their lifelong learning, while at the same time helping to assure the public they have demonstrated that they have the tools to provide high-quality care.

In 2004, the American Board of Family Medicine (ABFM) introduced self-assessment modules (SAMs), which consisted of a 60-item knowledge assessment followed by a corresponding computerized clinical simulation. Each SAM was divided into specific competencies and diplomates had to correctly answer 80% of the items in each competency to successfully complete the requirement. Diplomates were also presented with rationales and references for questions they missed and were allowed multiple attempts to complete the activity successfully. SAMs were purposefully created to present an in-depth and challenging exposure to the content area in order to assist in the identification of knowledge gaps, not to simply reinforce what one already knew.⁴ By 2012, SAMs covering 16 different topic areas had been created, drawn from the 20 priority areas identified by the Institute of Medicine.⁵

In 2016, the ABFM announced several changes in the continuous certification process, resulting from feedback provided by diplomates and the commitment to ensure that its programs were aligned with best practices in assessment, measurement, and quality improvement. Feedback on the knowledge assessment portion of the SAMs was very favorable; however, diplomates were less favorable about the clinical simulation.^6,7 Therefore, the knowledge assessment portion of the SAMS was uncoupled from the clinical simulation and the simulations were made optional. The knowledge assessments were renamed Knowledge Self-Assessments (KSAs) and the simulations were discontinued in 2019.

In the current continuous certification program, physicians are required to earn 50 certification points and acquire 150 CME credits during each 3-year stage. Self-Assessment and Lifelong Learning requirements now offer 2 options: (1) completing at least 1 ABFM-developed 60-question, topic-specific KSA activity, or (2) completing 4 quarters of Continuous Knowledge Self-Assessment (CKSA), consisting of 25 questions each quarter across a broad range of topics.

Improving KSAs

As part of ABFM’s commitment to continual improvement of our certification activities, in 2019 we began evaluating and substantially revising the KSAs. A review of diplomate feedback over time has suggested the need for a more robust and transparent updating process and revealed frustration with the multiple true-false question format. At the same time, our Board of Directors were eager to add topics that would be more relevant, supportive of a broader scope of care, and reflective of changes in health care. We have made a number of changes to the KSAs activities, including transitioning all questions to a single best answer multiple-choice format and completing a thorough review of all questions including updates of the critiques and references. Some KSAs were combined into a single activity (eg, Well Child Care and Childhood Illness became a single KSA on Care of Children). Other topics such as Hypertension and Diabetes are in the process of a content overhaul to ensure that it is up to date and relevant. This internal evaluation also identified opportunities to cover important areas that could become new modules, such as palliative care, addiction medicine, and high-value care. Many of these topics were suggested by diplomates in surveys and focus group conversations at state chapter and other meetings.

The new process maintains the annual review of topics for evidence updates and a substantial review of each KSA every 3 years that includes revision or replacement of up to 20 items. Comments made by participants are regularly reviewed and considered at each step in the revision process. Additionally, we have instituted an internal peer-review process and enhanced our medical librarian support to ensure effective search capabilities across topics. In some cases, we have partnered with external experts for peer review and/or development of new KSAs. For example, the new Palliative Care KSA released earlier this year was accomplished by a knowledge development team through the Interstate Postgraduate Medical Association (IPMA). The American Academy of Family Physicians’ (AAFP) Advanced Life Support in Obstetrics course will be added as a KSA-qualifying option in early 2021. A reliable mechanism for more frequent review of diplomate comments on KSA questions, and, when indicated, real time revision or replacement of questions was also established. We believe that this creates an ideal model of partnering with diplomates to help ensure up-to-date content. A schedule of the KSA revision initiative is shown in Table 1.

Lastly, the revision process also includes improvements in the KSA platform, providing participants with the ability to review critiques to any question they have already answered, not just the ones they answered incorrectly. Additional platform updates were designed to enhance the ease of navigation and improve the user experience for diplomates. The standard for successful completion was also changed from 80% correct in each competency area to 80% overall (48 out 60), in as many attempts as necessary.

What Are We Learning So Far?

In April of this year, the new Care of Children KSA was made available as the first newly revised KSA, combining content from the Well Child Care KSA and Childhood Illness KSA. As with all KSAs, completion of the Care of Children KSA fulfills the self-assessment requirement and earns the diplomate 10 certification points and 8 CME credits. It has been completed by 955 diplomates and 601 residents to date, with more than 400 currently in progress. Thus far this new activity is receiving higher ratings from participants with respect to relevance to practice (89%), usefulness (94%), currency of information provided (93%), and overall favorability (92%). Moving to a single best answer multiple-choice format for all 60 items, eliminating multiple true-false questions, and the new standard for successful completion have helped to increase the successful completion rate on the first attempt by 3.15 percentage points compared to the average rates on the 2 former KSAs (9.63%, compared to 5.55% and 5.15% for Well Child Care and Childhood Illness, respectively). The commitment to keeping the KSA challenging, with a goal of identifying knowledge gaps, was preserved in this revision process, but seems to have eliminated the frustration of the multiple true-false format, which previously comprised 70% of KSA questions.

The new Palliative Care KSA has been completed by 612 diplomates and 163 residents, with a 12.5% successful completion rate on the first attempt. Nearly 25% of all participants are physicians with a certificate of added qualifications (CAQ) in Hospice and Palliative Care. Feedback about currency of content (94%), relevance to practice (84%), usefulness (85%), and overall favorability of the activity (92%) has been very positive. Similarly, the new Heart Disease KSA, a combination of the former Heart Failure and Coronary Artery Disease KSAs, has been completed by 301 diplomates and 62 residents, with similarly higher ratings from early participants.

Next Steps

All KSAs will be updated by mid-2021, after which they will be scheduled into the new, ongoing revision process. We are already analyzing diplomate practice data and feedback to inform the choice of new KSA topics that could be added in the coming years. In partnership with the AAFP, we expect to add an alternative self-assessment activity on Health Equity in 2021. We will more explicitly link KSAs to a corresponding performance improvement activity in order to facilitate the translation of new knowledge into practice and increase the relevance of both activities. Finally, a journal article activity will be piloted in 2021, where diplomates will be able to read and answer questions about selected evidence-based, practice-changing articles as a self-assessment activity. In all of this, we will continue to engage diplomates and key partners to continually improve the suite of self-assessment options available to diplomates in Family Medicine Continuing Certification.