Physicians Answer More Clinical Questions and Change Clinical Decisions More Often With Synthesized Evidence: A Randomized Trial in Primary Care

Recruitment and Inclusion Criteria

Participants were recruited by 3 methods: (1) online notices to all new registrants (for a 30-day trial of the database) who self-identified as health care professionals (ie, not trainees), (2) postal mailings describing the trial and inclusion criteria sent internationally to about 5,000 members of the Medical Library Association, and (3) similar e-mails sent to about 400 US family medicine residency programs seeking graduates and faculty who met inclusion criteria. It was not practical to document the eligible population because it is unknown how many clinicians were contacted by medical librarians and residency programs.

After registering and requesting participation in the study, clinicians were included if they identified themselves as primary care clinicians who spent at least 20 hours per week in direct patient care and had English fluency, high-speed Internet access, a willingness to enter 20 of their own clinical questions, no previous relationship with DynaMed, and no editorial or financial relationship with other medical references. Participants provided informed consent through online enrollment and received a 1-year subscription to the database. Previous users of DynaMed were excluded. For participants who listed a specialty other than family medicine, internal medicine, or pediatrics, an author who was blinded to the results (DSW) evaluated their questions for content. If more than one half of their questions were considered specialty focused, all of their questions were excluded.

Randomization and Procedures

Participants were randomized on a per-question basis to be allowed to use DynaMed (A) or not (N). In either case, they could use their usual information sources. No restrictions on sources used were made except that DynaMed use was prohibited for N questions. Questions were assigned in random order uniquely for each participant. As each participant enrolled, a computer-generated random allocation sequence was assigned in blocks of 10 questions (5 A, 5 N) to provide balanced assignments for within-participant analyses. Question order was unaffected by other participants or timing of question entry. To maintain allocation concealment, participants were informed of individual question allocation only after each question was entered. Assignment for individual questions was concealed from investigators until participants had entered data.

Participants were asked to enter clinical questions for searching during practice but were able to enter questions at any time. Online instructions stated, “Please enter the clinical question you wish to search for. Be concise, but add sufficient information so that others can repeat the search. Once you have entered the question, click Start.” Data were collected only for questions formally entered as part of the study. Participants entered a clinical question through a free-text dialog box. On clicking the Start button, they were informed whether they could use DynaMed. If allowed (A), a button appeared that would open DynaMed in a new browser window. In either case (A or N), participants had a button that would open a new browser window with the Web site address of the participant’s choice entered during study enrollment. Opening a new browser window started the timer (which was not displayed). Participants were instructed to search, read, and think as appropriate for the situation until they believed they either had found an adequate answer or would not find an answer in a reasonable time. On search completion, participants returned to the study window and selected Answer Found or Answer Not Found, automatically stopping the timer.

Participants were presented with their search time, in minutes and seconds, and could change the time recorded if they were interrupted during searching. Participants were asked, “Did the answer make a difference in your clinical decision making?” (if Answer Found) or “Do you think an answer would have changed your clinical decision making?” (if Answer Not Found) and selected Yes or No. All data were recorded by the participants. The content of answers found by participants was not recorded.

Statistical Analysis

Based on funding, target enrollment was 50 participants who would each ask 20 questions. We kept the study open as long as possible within the timeline allowed by the grant funding.

For each outcome, we conducted a per-question analysis comparing all A questions with all N questions and per-participant analysis of differences in outcomes within each participant. The results of the per-question analyses are provided as descriptive statistics only because outcomes are not independent of participant, thus violating assumptions used for inferential statistics. This decision was made with statistical consultation before the protocol was submitted for funding, and thus before data collection.

We analyzed the proportion of questions for which an answer was found. For the effect of searching on clinical decision making, we analyzed the proportion of questions for which an answer was found and made a difference, the proportion of questions for which an answer was not found and would have made a difference, and total points assigned for overall impact on decision making. We counted +1 point if an answer made a difference, 0 points if an answer did not change clinical decision making, 0 points if an answer was not found but would not have changed clinical decision making, and −1 point if an answer was not found when an answer would have made a difference. We analyzed median search times for all questions, questions for which answers were found, and questions for which answers were not found. Median times were used because times did not follow a normal distribution.

For the per-participant analysis of proportion of questions answered, we determined the proportion of A questions answered, the proportion of N questions answered, and the difference between these proportions for each participant. This difference was used for statistical analysis. Because this difference was based on paired, nonnormally distributed data, we used the Wilcoxon signed rank test,¹³ a nonparametric analog of the paired t test. Per-participant analyses for all other outcomes (eg, proportions of questions for which answers made a difference, median search times) were conducted in the same manner. A normality test was performed on each outcome before Wilcoxon signed rank tests were used, and no outcomes followed a normal distribution. Our null hypothesis for each analysis was that the median difference equals zero and the sum of positive differences equals the sum of negative differences. All analyses were conducted using intention-to-treat principles based on random assignment, regardless of whether participants used the DynaMed database.

To avoid conflicts of interest, the coinvestigator (DSW) and statistician (BG) had full access to all the data and veto-level involvement in inclusion and exclusion decisions for all participants and data. Analysis was conducted by the statistician at an academic center.

This study was approved by the Health Sciences Institutional Review Board of the University of Missouri-Columbia and by the National Science Foundation.

Participants and Clinical Questions

Eighty-two participants enrolled between January 20, 2004, and April 28, 2004. Sixteen entered no questions. Two previous DynaMed users were excluded. Four participants were excluded because they did not understand instructions, recorded inaccurate data, and did not respond to requests for clarification. The remaining 60 participants entered 780 questions between January 22, 2004, and June 23, 2004. Eight participants (entering 82 questions) were excluded because they were not primary care clinicians and more than 50% of their questions were pertinent to their specialty—cardiology, emergency medicine, general surgery (2 participants), gynecologic surgery, neurology, occupational health and safety, and ophthalmology. Participant and question flow are shown in Figure 1⇓.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Participant and question flow.

A = DynaMed allowed; N = DynaMed not allowed.

Most of the 52 participants were family physicians practicing in the United States who were familiar with computer searching and who highly valued EBM based on self-report (Table 2⇓). Participants used a range of information sources (Table 3⇓). Their questions varied widely in subject area, complexity, and clarity (ie, participants often typed in brief phrases insufficient to determine the actual question). Questions were classified as relating to treatment (45%), diagnosis/evaluation (22%), etiology (4%), adverse effects of treatments or exposures (4%), epidemiology (4%), screening (3%), prognosis (3%), and prevention (2%). Eighty-two questions (12%) were not easily classified.

Finding Answers to Clinical Questions

Among the 46 participants who asked at least 1 question in the A state and at least 1 question in the N state (689 questions total), a larger proportion of participants found answers to more questions in the A state (Table 4⇓).

Participants answered 263 (75.8%) of 347 A questions and 250 (71.2%) of 351 N questions (Table 5⇓). Of the 347 A questions, participants searched DynaMed for 275 (79.3%). Answers were found for 196 (71.3%) of the 275 questions when DynaMed was searched and specifically found with DynaMed for 190 (69.1%).

Effects on Clinical Decision Making

Among the 46 participants who asked at least 1 question each in the A state and the N state, significantly more participants found answers that changed clinical decision making in the A state (Table 4⇑). Of the 263 answers found in the A state, 224 (85.2%, or 64.6% of all questions randomized to A) were noted to make a difference in clinical decision making (Table 5⇑). Of the 250 answers found in the N state, 209 (83.6%, or 59.5% of all questions randomized to N) were noted to make a difference in clinical decision making.

Among the 185 questions for which answers were not found, answers to 150 (81.1%) would have potentially made a difference in clinical decision making (81.0% for A questions, 81.2% for N questions). The per-participant analysis found no significant differences between A and N states.

For the overall impact on clinical decision making, participants generated 156 points from 347 questions in the A state and 127 points from 351 questions in the N state. Per-participant analysis for this measure highly favored the use of DynaMed (Table 4⇑).

Speed of Finding Answers

Time data were invalid (and not recalled by the participant) for 3 questions. Among the remaining 695 questions, the median time for overall searching (whether or not an answer was found) was similar in A and N states (4.95 vs 4.98 minutes) (Table 5⇑). Median times were similar in A and N states for both successful and unsuccessful searches. Per-participant analyses did not show significant differences (Table 4⇑).

Strengths and Weaknesses of the Study

Allocation concealment with per-question randomization is a major advance for studies evaluating use of databases for answering clinical questions. Trials using per-participant randomization may be confounded by variations between participants and the effects of resource availability on question recognition and selection. Participants in this study could still introduce bias if they applied differing levels of search thoroughness or answer interpretation based on whether they could use DynaMed. Excluding previous DynaMed users and participants who have relationships with medical references avoids known biases, but unrecognized confounders cannot be excluded.

One would expect that adding a resource without limiting use of existing resources would increase the proportion of questions answered. But some participants recorded No Answer Found after searching DynaMed without searching their usual sources. Some participants appeared satisfied that no answers would be found, whereas others appeared to approach the trial as a comparison with usual information sources instead of a comparison with the addition to these sources.

The study was not designed to evaluate degrees of clinical relevance of answers. Participants were asked, “Did the answer make a difference in your clinical decision making?” and were forced to make a dichotomous Yes or No choice. No additional instructions were provided for assisting participants in the interpretation of this item.

The study was not designed to measure the amount of information transmitted per unit time. Times recorded represented total time using information resources and did not separate time spent finding specific answers from time spent reading additional information of interest. Some participants found answers to their questions quickly and then spent more time reading additional information of interest, thus increasing the times recorded.

Participants chose their own questions and used different questions in A and N states. This approach more accurately reflects actual clinical questions than approaches whereby researchers interpret questions asked by clinicians, or clinicians search using questions provided to them. The A and N states were tested using different questions. Per-participant analyses may control for biases between participants but cannot control for within-participant biases introduced by using different questions. Data entry was minimized to facilitate research during clinical practice, so there are insufficient data to determine whether question characteristics differed. With a large sample, randomization should reduce such bias.

Generalizability for Clinicians

Clinicians in this study were using familiar search strategies with familiar sources in the control state. The study evaluated the addition of an unfamiliar source. Speed and success rate might increase with practice.

Our findings are likely generalizable to primary care clinicians who use the Internet to answer clinical questions and who value EBM. This study did not include any participants who expressed a low value of EBM. Although 8 specialists were inadvertently entered in the trial (perhaps because of the lack of a definition of primary care on study entry forms), there are insufficient data to analyze the ability of DynaMed to answer questions in subspecialties.

Future Research

The principal investigator was the database developer, an approach mandated by the terms of grant support. Multiple procedures (veto-level involvement of coinvestigator, independent statistical assessment, determination of outcome data by participants instead of investigators) were in place to limit the effect of competing interests. This study’s methods and mechanisms can be adapted for use by independent investigators.

Finding more answers and changing clinical decision making more often would not be beneficial if the answers found were inaccurate. A study using questions from this trial compared the validity of answers found in DynaMed with that of answers found in a combination of 4 other “just-in-time” medical references (see the Supplemental Appendix, available online-only at http://www.annfammed.org/cgi/content/full/3/6/507/DC1). That study found the level of evidence of answers found in DynaMed to meet or exceed that of answers found in the combination for 87% of questions. Given the subjective evaluations inherent in assigning level-of-evidence labels to clinical reference content, however, this study requires confirmation by independent investigators.

Our study design could be adapted by changing allocation parameters to support head-to-head and multiple comparisons. Adding question classification items to data entry and conducting a larger study could allow determination of clinical reference performance specific to question types.

This study was not able to determine which of 8 design and process principles of DynaMed (Table 1⇑) are responsible for providing valid answers to more clinical questions. Determining which factors are important would help inform development of clinical references. We believe that systematic literature surveillance is a critical component. EBM traditionally requires systematic reviews to ensure that relevant evidence is not missed when answering clinical questions. Systematic reviews alone do not have the capacity to meet most information needs during practice, and global efforts expended on systematic reviews are outpaced by research production.¹⁵ Systematic literature surveillance provides an information-processing efficiency unattainable with systematic reviews¹⁶ and may allow capture of relevant evidence for many more current questions; used alone, however, it may miss earlier research. We believe that in combination with systematic reviews, systematic literature surveillance makes it possible for the primary care knowledge base (ie, the sources primary care clinicians use to access clinical reference information) to reliably represent current best evidence. We would be pleased to be contacted by anyone wishing to advance this research agenda.