“THE END OF THE BEGINNING” FOR CLINICAL SIMULATION IN THE ABFM SELF-ASSESSMENT MODULES (SAMS) ================================================================================================= * Michael D. Hagen * Walton Sumner * Guy H. Roussel “Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.”1 With these words, Winston Churchill marked the Allied victory of Second El Alamein that represented a major turning point in the struggle against the Axis powers in World War II. ABFM has likewise reached a major turning point in the Maintenance of Certification for Family Physicians (MC-FP) program. The clinical simulation program began at ABFM in 1992; the ABFM Board intended at the outset to develop the simulation technology as a potential replacement or enhancement for the MC-FP examination. When ABFM embarked on the MC-FP program in 2004, the Board elected to include simulations in the self-assessment process as a means to familiarize Diplomates with the interface and functionality in preparation for simulations’ appearance in the part III examination. Since that deployment, ABFM has delivered over 500,000 simulation instances. Over the past year, ABFM has embarked on several initiatives that have led the Board of Directors to reconsider the role of simulation in MC-FP. The DAIQUERI and TRADEMaRQ registry projects2 will simplify Diplomates’ participation in MC-FP and, potentially, provide performance information that might have previously derived from simulations. Additionally, ABFM has begun development of a continuous knowledge self-assessment (CKSA) process that will involve sending out periodic “mini-quizzes” of 1 or more items (including references and critiques) keyed to the examination content blueprint.3 This process will provide ongoing prospective feedback for Diplomates and serve as an alternative option for completing MC-FP self-assessment requirements. Given these new options, the ABFM Board of Directors voted at its October, 2015 meeting to de-link the knowledge assessment component of the MC-FP SAMs from the associated clinical simulations. This means that the knowledge assessments and simulations will now serve as independent options for completing the MC-FP self-assessment requirement. This new role for simulations provides an opportunity to refocus the simulations’ operation and functionality to present a much more formative, rather than summative4, emphasis. Development efforts heretofore have stressed summative scoring models and functionality to support a possible role for simulation in the MC-FP examination. In this summative role, simulation scoring would have necessarily mapped to the existing examination pass-fail threshold, and would not serve easily to motivate and direct higher levels of performance. We can now work to implement formative features such as context sensitive feedback, quick quizzes, short lectures/discussions, and competitive gaming features that represented inappropriate components for a possible high-stakes examination environment. In this new role, scoring can provide feedback on performance, and *can* motivate higher performance levels. The simulator interface can now evolve to include much more guidance (eg, more use of drop-down menus and pop-up balloons) than would have been appropriate in the context of using simulation within the examination. In addition to this more formative emphasis, ABFM has, along with colleagues from Virginia Commonwealth University, completed recently an extensive review of structured and unstructured SAM feedback from the past 10 years’ experience (ABFM internal reports.)5,6 That review identified a number of Diplomate suggestions for improvements to the simulation interface. During the summer 2015, these suggestions guided multiple interface revisions that ABFM deployed in August. Subsequent feedback indicated favorable response to the revisions (internal report).7 Work remains, however, on several interface issues, including more responsive natural language processing, easier access to diagnostic studies and therapies, and greater use of media resources. The development team met in Lexington, KY, February 23–24, 2016, to begin work on implementing responses to this feedback and the formative features mentioned earlier. The ABFM has additionally engaged external consultant experts to aid in simulation interface re-design. The October, 2015 Board action changes the role originally envisioned for the ABFM clinical simulation system. This change, however, clearly represents the “end of the beginning,” not the end of clinical simulation in the ABFM MC-FP program. * © 2016 Annals of Family Medicine, Inc. ## References 1. Churchill WS. The Bright Gleam of Victory. November 10, 1942. [http://www.winstonchurchill.org/resources/speeches/1941-1945-war-leader/987-the-end-of-the-beginning](http://www.winstonchurchill.org/resources/speeches/1941-1945-war-leader/987-the-end-of-the-beginning). Accessed Feb 26, 2016. 2. Phillips R. ABFM to simplify maintenance of certification (MOC) for family physicians and make it more meaningful: a family medicine registry. J Am Board Fam Med. 2015;28(3):431–433. [FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6ODoiMjgvMy80MzEiO3M6NDoiYXRvbSI7czoyMzoiL2FubmFsc2ZtLzE0LzMvMjgxLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 3. The Phoenix, A Diplomate’s Newsletter. Lexington, KY: American Board of Family Medicine; 2016. [https://www.theabfm.org/about/newsletter.aspx](https://www.theabfm.org/about/newsletter.aspx). 4. Eberly Center for Teaching Excellence & Educational Innovation, Carnegie Mellon University. What is the difference between formative and summative assessment? [https://www.cmu.edu/teaching/assessment/basics/formative-summative.html](https://www.cmu.edu/teaching/assessment/basics/formative-summative.html). Accessed Mar 1, 2016. 5. Etz R, Gonzales M, Maternik A, Reves W, Winship J. American Board of Family Medicine Self Assessment Module Feedback Survey Analysis January 2004 – April 2013 Final Report. Richmond, VA: Virginia Commonwealth University; 2015. 6. Etz R, Gonzales M, Winship J, Johnson N, Middleton H, Reves W. American Board of Family Medicine SAM Feedback Survey Analysis, Dataset 2 Preliminary Findings. Richmond, VA: Virginia Commonwealth University; 2015. 7. Eden A, Peabody M. SAM Feedback Survey Revision: Pilot Validation and Results. Lexington, KY: American Board of Family Medicine; 2015.