American Board of Family Medicine In-Training Examination

Kathy Banks

doi:10.1370/afm.1024

The American Board of Family Medicine administered the first in-training examination (ITE) in 1979 and has continued since then to offer the exam to ACGME-accredited family medicine residency programs on the first Friday in November. The purpose of the examination is to provide an assessment of each resident’s progress, while providing programs with comparative data about the program as a whole.

The ITE was offered solely in the traditional pencil and paper format until the 2006 administration when the ABFM asked residency programs to voluntarily participate in an Internet-based testing ITE pilot project. The pilot was planned as the initial exploration into the possibility of eventually delivering the ITE exclusively in an online format. The initial pilot examination involved 41 ACGME-accredited residency programs and 633 resident examinees. Two international groups from Australia and Macau also participated with a combined total of 12 examinees. The pilot examination was a success with minimal problems being reported including temporary system lockups and delays in progressing from item to item.

The ABFM continued to offer the Internet-based exam option for both the 2007 and 2008 ITE administrations. In 2007, the ABFM chose to outsource the exam by using a national testing vendor instead of hosting the exam on the ABFM servers, which opened the online version to an even greater number of residents. Unfortunately, significant technical problems were experienced with this exam despite the fact that a trial exam had been undertaken. The official exam day progressed very well until the programs on the west coast began to log on around 11:00 am EST. At that point, all programs began to experience latency in moving from item to item and many of the west coast programs were unable to log on at all. Many programs had no choice but to opt out of the online exam and administer the remainder of the items by paper and pencil.

A review of the configuration of the vendor’s Web servers, application, database, and error logs was completed by the Microsoft Server Support Team and it was discovered that the vendor was not at fault. Further investigation indicated that the majority of the problems could be traced to Internet Explorer 6. Microsoft determined that the delay resulted from a hard-coded setting in Internet Explorer 6 that was invoked following a network delay of greater than 90 seconds. The problem could not be replicated when using Internet Explorer 7; the longest delays were 5 seconds. Microsoft and the testing vendor suggested that future online participants would be required to use Internet Explorer 7 as their Web browser in order to avoid the same issues experienced with the 2007 exam.

The testing vendor has since made many upgrades to their testing system including installation of a new load balancer solution for Web traffic and IBT database server upgrades. The system was load tested in February 2008 and it was determined that it can support 10,000 concurrent users.

The 2008 administration of the online ITE was largely considered a great success with approximately 1,100 residents testing successfully. Minimal problems were experienced mostly due to pop-up blockers not being turned off on the workstations and some programs still utilizing Internet Explorer 6 despite the ABFM’s advice to use Internet Explorer 7 only.

The ABFM’s ultimate goal is to transition entirely to online delivery of the in-training examination by 2014. Rest assured that we will continue to work closely with the testing vendor to strive toward administering a trouble-free exam. We believe that the online test delivery method is in the best interest of your residents as it will allow them to experience the same computer-based format they will encounter when sitting for their first certification examination upon completion of residency.