Number Needed to Benefit From Information (NNBI): Proposal From a Mixed Methods Research Study With Practicing Family Physicians

INTRODUCTION

Electronic knowledge resources help physicians find answers to clinical questions, thus enabling the use of clinical information in routine professional practice. Hereinafter, clinical information refers to research-based articles or summaries or synopses, and knowledge syntheses for clinical practice, eg, guidelines—not patient-specific data.^1–4 In observational studies, information retrieved from electronic resources has been associated with a positive cognitive impact on physicians.⁵ No study, however, has systematically described patient health outcomes associated with the use of information directly retrieved by family physicians.⁶ Our research question therefore was for how many patients does a physician need to retrieve clinical information from electronic resources to report a health benefit for 1?

We critically reviewed 29 qualitative and quantitative studies on information use and subsequent patient health outcomes.⁶ No study was grounded in a conceptual framework or a theoretical model. No study systematically tracked information objects retrieved by physicians, such as Web pages, and comprehensively assessed their use and subsequent benefits. Moreover, these 29 studies reported qualitative and quantitative data separately. None systematically linked quantitative data by tracking information use in log files with qualitative data to produce trustworthy clinical stories.

In contrast, we conducted a pilot mixed methods study where searches conducted by 17 family medicine residents were systematically tracked for 2 months.⁷ Among 84 searches for patients, 12 were associated with patient health benefits (reported by residents). This finding suggested a new concept we refer to as the number needed to benefit from information (NNBI). We define the NNBI as the number of patients for whom clinical information (eg, the answers to physicians’ clinical questions) was retrieved for 1 patient to benefit. In our pilot study, the NNBI was 7.

The purpose of the present mixed methods study is twofold. We first describe family physicians’ use of information from an electronic knowledge resource for answering clinical questions and their perception of subsequent patient health outcomes, and we then estimate the NNBI. Our study is based on a theoretical model and merges complementary quantitative and qualitative data on searches done by practicing physicians for clinical information. Given that one-half of all clinical questions in practice are not pursued by family physicians,^8,9 the wealth of information available to guide primary health care practice remains unused to its full potential. Thus, knowing that their patient can benefit may stimulate physicians to search more often when they feel the need.

METHODS

We undertook a mixed methods study¹⁰ in which we combined a quantitative longitudinal study with a qualitative multiple case study,¹¹ the design of which is described in Figure 1. Cases were searches for information by practicing physicians where information was used for specific patients. To obtain a systematic and comprehensive description of these cases, quantitative and qualitative data were integrated at both collection and analysis stages.¹² Both types of data were merged into clinical stories (hereinafter called vignettes). Concepts and terms from mixed methods research and information studies are defined in Supplemental Appendix 1, available at www.annfammed.org/content/11/6/559/suppl/DC1, and a detailed methods section is available in Supplemental Appendix 2.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Mixed methods convergence design.

IAM=Information Assessment Method.

This study is based on a theoretical model called the acquisition-cognition-application/levels-of-outcomes model (hereinafter the ACA-LO model),¹³ which can be summarized as follows. Clinicians may find one or more information objects to fulfill a search objective, eg, a Web page for addressing a clinical question (acquisition/level 1). Next, they integrate with their previous knowledge the information objects deemed relevant for achieving this objective (cognition/level 2). Only a fraction of this information may be used for 1 specific patient (application/level 3). On occasion, 1 information object is applied to the management of this patient and associated with health benefits (patient outcomes/level 4).

In the quantitative longitudinal study, 41 Canadian family physicians used Essential Evidence Plus (Wiley-Blackwell) on a handheld computer to search for clinical information, and rated their searches using the Information Assessment Method (IAM), which operationalizes the ACA-LO model (www.mcgill.ca/iam).^14,15 For all searches, physicians were prompted to complete the 2007 version of the IAM questionnaire (Figure 2).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

The 2007 Information Assessment Method (IAM) questionnaire.

In the multiple case study, qualitative data consisted of observations, log reports, archives, and interviews with respect to a subsample of most recently rated searches conducted over an average of 86 days. Physicians were then interviewed on 2 occasions regarding their most recent searches (midstudy and end of study). Before each interview, log reports containing IAM ratings were analyzed by the interviewer to identify searches rated as “I used (or will use) this information for a specific patient”; then, to stimulate recall of past events, log reports and IAM ratings were reviewed by telephone with each physician. Cases were critical searches that were identified using the critical incident technique.^16–18 In line with this technique, a search was critical when the clinical situation and the use of information were clearly described.

We then analyzed data on critical searches (cases) to describe physicians’ use of clinical information and subsequent effects on patient health. We integrated qualitative and quantitative data using specialized software (NVivo 7, QSR International). For each case, data were comprised of content from the information objects, ie, pages from Essential Evidence Plus, the corresponding log file containing IAM ratings, and interview transcripts. We combined a deductive with an inductive thematic analysis.^19,20 We assigned data to predefined themes derived from IAM items, such as types of information use and health benefits. These items were based on prior literature reviews, qualitative studies, and mixed methods studies.¹⁴ Data were also analyzed to explore new themes (emerging types of information use and benefits). A vignette contains both qualitative and quantitative data on the same case. Finally, to estimate the NNBI, we divided the total number of searches for a specific patient (reported via IAM ratings) by the number of critical searches with at least 1 patient health outcome (reported in vignettes).

Ethics approval (A09-E28-07B) was obtained from the McGill University Institutional Review Board.

RESULTS

We interviewed 39 physicians from 9 of 10 Canadian provinces about their searches. Twenty-six (66.7%) reported that they had a faculty appointment and were involved in teaching or research. There were 23 men and 16 women, all in active practice, ranging in age from 28 to 70 years (median = 44 years). Over an average of 320 days, 2,131 searches for clinical information were documented, and 1,767 (82.9%) searches were rated; of those, 1,193 (56.0%) were covered by interviews (Figure 3). In 715 (59.9%) of 1,193 searches, the search objective was directly linked to a patient (to address a clinical question, or to share information with a patient, or to plan/manage aspects of patient care with other health professionals). Of 365 (30.6%) searches rated “I used (or will use) this information for a specific patient,” 188 (15.8%) were critical searches. The 188 vignettes corresponding to these cases are available online (iamclinicalvignettes.mcgill.ca). A sample vignette is shown in Table 1. Results are presented in accordance with the 188 cases.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Mixed methods data collection: flow diagram.

IAM=Information Assessment Method.

Searches were conducted before, during, and/or after the patient-physician encounter in 21 (11.2%), 71 (37.8%), and 98 (52.1%) cases, respectively. The sum is superior to 100% because in 2 cases the same information was consulted twice: once with the patient and once without, ie, during and after seeing the patient. Before the encounter referred to situations where physicians prepared for the patient visit. After the encounter referred, for instance, to situations where physicians reviewed laboratory results after the visit. Searches were conducted in a clinic or office setting in 146 (77.7%) cases, or outside a professional setting, eg, at the physicians’ home, in 42 (22.3%) cases. Physicians searched most often by themselves, less commonly in the presence of the patient, or with a colleague in 127 (67.5%), 34 (18.1%), and 27 (14.4%) cases, respectively. In 133 (70.7%) cases, Essential Evidence Plus was the only resource consulted by the physician. When another source was consulted, interviewees consulted 1 or more than 1 type of source, such as another electronic resource, textbooks or journals, and other professionals in 41 (21.8%), 11 (5.9%), and 4 (2.1%) cases, respectively.

In all cases, information from Essential Evidence Plus was relevant and met the physician’s objective. The most frequently reported search objective was to address a clinical question about a specific patient (181; 96.3%). The most frequently reported cognitive impact was confirming that the physician did the right thing (132; 70.2%). The most frequent use of the information was confirming that the patient was being managed correctly (109; 58.0%). The most frequently anticipated health benefit was to avoid an inappropriate diagnostic procedure or treatment (28; 14.9%).

These 28 perceived benefits resulted from 4 types of patient advice: (1) the patient was not asked to do something that the physician had planned to do before the search (20; 10.6%), eg, consult another specialist; (2) the patient was asked to avoid something (4; 2.1%), eg, planned use of a natural product that could interact with prescribed medication; (3) the patient was asked to do something different, compared with what was planned before the search (2; 1.1%), eg, to reduce a drug dose; and (4) the patient was asked to do something more appropriate, which the physician had not planned to do before the search (2; 1.1%), eg, further testing.

Paths and NNBI

The 188 vignettes are represented in the path diagram (Figure 4), where paths are characterized by types of patient outcome (patient behavior vs patient knowledge or satisfaction vs no outcome). The 53 vignettes with outcomes are listed in Supplemental Appendix 3 (www.annfammed.org/content/11/6/559/suppl/DC1). Of those vignettes, 42 describe types of patient behavior outcome: in 28 cases (14.9%), interviewees associated the use of retrieved information with avoidance of an unnecessary or inappropriate diagnostic procedure, treatment, or preventive intervention; in 18 cases (9.6%), interviewees associated information use with prevention of disease or health deterioration; and in 13 cases (6.9%), interviewees associated the use of retrieved information with improvement of patient health or functioning or resilience. In summary, interviewees conducted 715 searches for a specific patient and associated the retrieved clinical information with health benefits in 53 (7.4%), ie, they retrieved information on average for 13.5 patients to report health benefits for 1. This outcome suggests an NNBI of 14.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

Physician perceptions of patient outcomes.

Note: This diagram and corresponding vignettes are available at iamclinicalvignettes.mcgill.ca.

DISCUSSION

Based on a new concept (NNBI) and a mixed methods study, our results suggest how frequently patients may benefit when their physicians search for information in electronic knowledge resources. The NNBI will vary depending on the type of electronic knowledge resource and the clinicians’ characteristics, such as information-seeking skills (acquisition), knowledge (cognition), and behavior in specific clinical situations (application). In a concomitant study with similar mixed methods, we documented searches for treatment recommendations (summaries) conducted by 30 primary health care professionals (10 nurse practitioners, 10 family health team pharmacists, and 10 family medicine residents), using another resource (e-Therapeutics+) on a desktop computer: the NNBI was lower (NNBI = 7), ie, expected benefits were more frequently reported.¹³ In our pilot study involving family medicine residents, the NNBI was also 7.

In a qualitative research study, Australian researchers interviewed 29 clinicians (13 hospital-based physician specialists and 16 clinical nurse consultants) about their use of an electronic knowledge resource called the Clinical Information Access Program.²¹ Interviewees described 85 searches. Of those, 19 (22.4%) were associated with patient health benefits, such as health improvement (eg, quality of life improved) and avoidance of an unnecessary procedure or medication. Their findings suggests an NNBI of 4. The main reason for the difference between this lower (better) NNBI compared with our results might be that in the Australian study, interviewees were asked to recall examples of searches that were helpful or unhelpful in their clinical work. Thus, participants selectively reported on searches with potential health benefits. In contrast, in our study all searches were systematically tracked whatever their helpfulness, and interviews of clinicians were informed by a log report of rated searches.

In the present study, our estimation of the NNBI may be too high for 3 reasons. First, to be conservative, we considered that forgotten searches never led to patient benefits (n = 130) (Figure 3). Forgotten searches that were beneficial for patients would have led to a lower NNBI. Second, we interviewed participants only once with respect to only 1 patient, but follow-up times needed to observe a health benefit vary as outcomes are time dependent.²² Had participants observed benefits for patients after the interview, the NNBI would have been lower. In addition, longer follow up on all searches could yield additional evidence of benefit.²³ Longer follow-up may also allow the detection of other patients for which the same information could be used; observed benefits for other patients would have also led to lowering the NNBI. Third, it is likely the 2008 version of Essential Evidence Plus was somewhat difficult to use for searches on a small hand-held computer. Our results indicate that participants rated about only 1 search per week (Figure 3), while some participants reported technical problems, making it likely that searches were occasionally done at a PC workstation (and therefore not tracked). Just after our data collection ended, Essential Evidence Topics was added to Essential Evidence Plus. This resource was specifically created to make it easier for physicians to find relevant information.

One implication of this study is that further improvements are needed to facilitate information seeking at the point of care. Improvements can be done by optimizing search engines and by making access to electronic knowledge resources easier. Such improvements may increase the frequency of searches, the relevance of information for patient management, and patient health benefits. For example, we propose developing a system to make it easier for family physicians to search for answers to their clinical questions within the electronic health record and to document the outcome of these searches using the IAM.²⁴ Such a system will facilitate searches by embedding links to favorite knowledge resources directly within the patient record and help physicians to automatically earn continuing education credit for their searches. Such improvements can make it worth the time and energy to search for answers to clinical questions.

What is encouraging in our work is that searches for information are often associated with educational outcomes for physicians themselves (published elsewhere).¹⁵ Considering all participants’ searches (n = 2,131), our results indicate that information was relevant in 76% of rated searches. Our results also indicate that information often had positive cognitive impact. Participants rated 3,300 information objects associated with these searches. Regarding 52% of them (n = 1,708), they reported an actual use or an intention to use for a specific patient. Regarding 46% (n = 1,516), they reported, “This information confirmed I did (will do) the right thing.” Regarding 38% (n = 1,246), they reported, “I learned something new.”¹⁵ Although reinforcement of knowledge and learning outcomes do not bring direct benefit to patients, they help physicians to build confidence for decision making.

We propose the NNBI concept by analogy with the number needed to treat (NNT), a widely used concept to communicate the magnitude of treatment effect, defined as the number of patients that need to be treated for 1 to benefit compared with a control in a clinical trial context. In line with this definition, further experimental research can measure the NNBI in a variety of contexts and consider potential adverse effects to arrive at a net NNBI (benefit after deduction of harm). For example, experiments can use the NNBI metric to compare electronic knowledge resources using objective outcome measures, such as data from the electronic medical record.^25,26 These resources are expensive, and such a comparison, eg, Essential Evidence Plus vs UpToDate, can help academic institutions and professional organizations select the most efficient resource in their context (resources with an optimal ratio of benefit divided by cost). Information retrieval and use are complex, thus further research may measure composite outcomes (ideally patient-oriented), eg, a combination of different kinds of health benefit associated with answers to therapeutic questions.

The lower the NNBI, the more value physicians will see in looking for information (if all searches had benefits, the NNBI would be 1). One-half of all clinical questions in practice are not pursued by family physicians; thus, a low NNBI can encourage clinicians to search for information systematically when they perceive an information need and may justify further continuing education or financial incentive for searching. Applying what is already known will have an important impact on health care and health. Moreover, a low NNBI can help to stimulate clinicians to take information retrieval training. If used by patients, the NNBI concept may enhance their confidence in asking clinicians or librarians to seek more information in high-stakes situations, eg, a treatment associated with serious adverse effects.

Our research faces at least 2 limitations. First, our results may not generalize to community physicians because of selection biases: (1) two-thirds of study physicians had a faculty appointment and were involved in teaching or research, (2) participants rated relatively few searches considering the broad scope of clinical questions that come up in primary health care, and (3) we described patient health benefits arising from the use of information directly retrieved by practicing family physicians within 1 type of electronic knowledge resource on 1 device (hand-held computer). Second, the NNBI is difficult to estimate because patient health benefits were based on physician perception rather than objective measures, such as blood pressure. Adverse effects that might result from searches for information were not reported (eg, by spending time with 1 patient searching for information, a physician may have less time for other patients with potentially adverse consequences). Our clinical vignettes document physicians’ reported behavior (information use) and their observations or expectations in terms of patient behavior, knowledge, and satisfaction (benefits). We did not prospectively follow patients to determine, for example, whether recommendations made by their physician were implemented. In addition, the association between preventive or therapeutic interventions and observed benefits are at best probabilistic because there was no control group. As mentioned by an interviewee, it is hard to determine for individual patients in routine clinical settings whether a specific intervention contributes to health outcomes.

The strength of our research was the ability to establish a chain of evidence between information retrieved in routine practice and reported benefits for a specific patient. Documenting the relative influence of all sources of information on the process of care and patient outcomes is challenging, and mixed methods allowed us to address this challenge. The combination of quantitative and qualitative data strengthened our work for at least 2 reasons. First, the computerized IAM questionnaire reduced recall bias because IAM-based log reports guided interviews and stimulated participants’ recall.²⁷ Second, the critical incident technique is known to be valid, and the triangulation of multiple sources of qualitative evidence, namely, archives, log reports, interviews, and observations, permitted us to critically examine physician interviews.

In conclusion, we report the first systematic and comprehensive examination of patient health benefits associated with the use of information directly retrieved by practicing family physicians from an electronic knowledge resource. Our mixed methods study supports a new concept, the number needed to benefit from information (NNBI), which can be used in experimental research with objective measures of patient health outcome to compare resources, to identify kinds of clinical questions that are difficult to answer, and to measure benefits of specific searches for information.