Developing an International Register of Clinical Prediction Rules for Use in Primary Care: A Descriptive Analysis
=================================================================================================================

* Claire Keogh
* Emma Wallace
* Kirsty K. O’Brien
* Rose Galvin
* Susan M. Smith
* Cliona Lewis
* Anthony Cummins
* Grainne Cousins
* Borislav D. Dimitrov
* Tom Fahey

## Abstract

**PURPOSE** We describe the methodology used to create a register of clinical prediction rules relevant to primary care. We also summarize the rules included in the register according to various characteristics.

**METHODS** To identify relevant articles, we searched the MEDLINE database (PubMed) for the years 1980 to 2009 and supplemented the results with searches of secondary sources (books on clinical prediction rules) and personal resources (eg, experts in the field). The rules described in relevant articles were classified according to their clinical domain, the stage of development, and the clinical setting in which they were studied.

**RESULTS** Our search identified clinical prediction rules reported between 1965 and 2009. The largest share of rules (37.2%) were retrieved from PubMed. The number of published rules increased substantially over the study decades. We included 745 articles in the register; many contained more than 1 clinical prediction rule study (eg, both a derivation study and a validation study), resulting in 989 individual studies. In all, 434 unique rules had gone through derivation; however, only 54.8% had been validated and merely 2.8% had undergone analysis of their impact on either the process or outcome of clinical care. The rules most commonly pertained to cardiovascular disease, respiratory, and musculoskeletal conditions. They had most often been studied in the primary care or emergency department settings.

**CONCLUSIONS** Many clinical prediction rules have been derived, but only about half have been validated and few have been assessed for clinical impact. This lack of thorough evaluation for many rules makes it difficult to retrieve and identify those that are ready for use at the point of patient care. We plan to develop an international web-based register of clinical prediction rules and computer-based clinical decision support systems.

*   clinical prediction rule
*   decision aid
*   score card
*   decision making
*   clinical decision support systems
*   primary care

## INTRODUCTION

Clinical prediction rules are tools that quantify the impact of multiple predictors from a patient’s history, physical examination, or laboratory results to inform a diagnosis, prognosis, or treatment response.1 Different methods are used to develop these rules, including univariate and multivariate analysis, neural networks, predictive nomograms, and classification and regression tree (CART) analysis.2,3 Examples of clinical prediction rules include the Centor score to predict streptococcal pharyngitis4 and the ABCD2 rule (age, blood pressure, clinical features, duration of symptoms, and diabetes history) to predict stroke.5

Before widespread clinical implementation, clinical prediction rules should pass through 3 stages of development. First, they should undergo derivation, whereby factors with predictive power are identified to develop the rule. Second, they should undergo validation, whereby the rule is tested in a new population for reliability and accuracy. This stage can be divided into 2 substages: narrow validation, in which the rule is tested in a similar population or clinical setting, and broad validation, in which the rule is tested in a different population or clinical setting. Third, the rules should undergo impact analysis, which assesses their impact on patient outcomes, physician behavior, or costs.1 The majority of clinical prediction rules have been derived but not validated, and relatively few have undergone impact analysis.6–8

Clinical prediction rules offer one way of implementing evidence-based medicine, especially if incorporated into clinical decision support systems (CDSSs), at the point of care. We are developing an international web-based register of clinical prediction rules for use in primary care, in collaboration with the Cochrane Primary Health Care field. We intend this register to be a resource for clinicians, available at the point of care, relevant to clinical practice, with rules summarized and categorized in terms of their derivation, validation, and impact analysis status. Although a number of clinical prediction rules are designed for use in primary care settings, articles describing these rules are particularly difficult to identify. This difficulty stems from the use of multiple terms to describe clinical prediction rules (eg, decision aid, score card) and primary care (eg, family medicine, general practice). Furthermore, no specific term is used to code clinical prediction rule articles in electronic search engines such as PubMed. To overcome this limitation, we developed an electronic search string to search the database for rules relevant to primary care.9 This search strategy was supplemented by searching additional relevant resources to develop a register of rules that are relevant to this setting.

The aims of this article are to describe the clinical prediction rules relevant to primary care that were retrieved for the creation of this register and to summarize them in terms of their clinical domain, stage of development, methodologic quality assessment, and the clinical setting in which they were studied.

## METHODS

### Search Strategy

We first searched the MEDLINE database (PubMed) using a search string specifically developed in house to retrieve clinical prediction rules relevant to primary care from 30 preselected medical journals between 1980 and 2009.9 No restriction was placed on language. See Supplemental Appendix 1 for details of the search string and included journals. Each article was screened on a hierarchical basis by title, abstract, and full text, if necessary. We next searched secondary sources of clinical prediction rules, including the *Journal of the American Medical Association (JAMA)* Rational Clinical Examination Series,10 and a handbook of rules.11 Personal resources were also investigated. Key experts in the field were asked for relevant articles through announcements made at 2 international conferences. In-house researchers and librarians also made their personal libraries available. For every validation article retrieved, we obtained the original derivation article; if derivation articles had not been retrieved through other search methods, we searched for them manually.

### Inclusion and Exclusion Criteria

Articles were eligible for inclusion in the register if they were considered to describe clinical prediction rules relevant to primary care. A clinical prediction rule was defined as “a clinical tool that quantifies the individual contributions that various components of the history, physical examination, and basic laboratory results make toward the diagnosis, prognosis, or likely response to treatment in a patient. [Clinical prediction rules] attempt to formally test, simplify, and increase the accuracy of clinicians’ diagnostic and prognostic assessments.”1 We excluded health questionnaire screening tools. We included rules at all stages of development (derivation, validation [narrow or broad], or impact analysis). We used the term *validation study* to describe studies that attempted to validate a rule regardless of outcome.

Primary care was defined as “... normally the point of first medical contact within the health care system, providing open and unlimited access to its users, dealing with all health problems regardless of the age, sex, or any characteristic of the person concerned.”12 Studies of clinical prediction rules conducted in non–primary care settings were eligible for inclusion if they were relevant to primary care. As the register is designed to be used internationally, we used broad inclusion criteria to acknowledge variation in the same-day diagnostic tests that are available across countries, with the understanding that not all countries will have access to the same services. The role of primary care clinicians also differs internationally.

### Data Extraction

We screened all articles for inclusion according to title and abstract. Any queries were discussed with 2 authors, 1 methodologist, and 1 general practitioner. For each rule in an article, we extracted information on the following: (1) type of article: original study, systematic review, or review; (2) stages of development of the clinical prediction rule: derivation, validation, or impact analysis; (3) type of rule: prediction rule, decision rule, or both; (4) clinical domain to which the rule applies according to the *International Classification of Primary Care,* 2nd edition (*ICPC*-2); and (5) clinical setting in which the rule was studied.

We also collected several pieces of additional information. First, we assessed methodologic quality of each article using appropriate checklists; for derivation, there were 8 criteria assessing internal and external validity (Supplemental Appendix 2a), for validation studies there were 5 criteria assessing internal and external validity (Supplemental Appendix 2b), and for impact analysis, the choice of methodologic assessment varied according to the study type (Supplemental Appendix 2c). Second, we determined the country in which the study was conducted. Third, we ascertained the term used to describe clinical prediction rules, for example, clinical prediction rule, nomogram, or score card. These findings are not the main focus of this article, but details are presented in an online Appendix 3 (available at [http://hrbcentreprimarycare.ie/?q=cpr-register-paper-supplemental-data](http://hrbcentreprimarycare.ie/?q=cpr-register-paper-supplemental-data)).

## RESULTS

### Search Strategy

The process we used to identify and include articles for analysis is shown in Figure 1. The MEDLINE PubMed search (1980–2009) retrieved 72,837 articles; a total of 1,151 full-text articles were retrieved, of which 277 met all inclusion criteria. These articles were supplemented with 66 articles from secondary sources,10,11 243 from experts, and 159 identified by searching for derivation articles from references of validation articles. We ultimately included a total of 745 unique articles in the register.

![Figure 1](http://www.annfammed.org/https://www.annfammed.org/content/annalsfm/12/4/359/F1.medium.gif)

[Figure 1](http://www.annfammed.org/content/12/4/359/F1)

Figure 1 
Flow diagram of articles on clinical prediction rules in primary care.

Note: The 745 articles included in our review contained data on 895 original studies as many articles described more than 1 clinical prediction rule study. Analyses pertain to 434 unique clinical prediction rules.

a Articles did not pertain to a clinical prediction rule, were not relevant to primary care, or both.

### Overview of Retrieved Articles

Several of the articles included in the register described more than 1 stage of development of a clinical prediction rule or described multiple rules. For example, some authors derived and validated a rule in the same article,5 and others validated and compared a number of rules in the same article.13 We therefore conducted our analysis at the level of pieces of information about a single clinical prediction rule; for example, if an article both derived and validated a rule, it was considered as providing 2 pieces of information about that rule. This unit of measurement is henceforth referred to as a *clinical prediction rule study*. In all, the 745 articles contained 989 clinical prediction rule studies on 434 unique rules. Most, 895 (90.5%), were original studies; a few were systematic reviews (3.9%) or reviews or guidelines (5.6%). Our main analysis is based on the 895 original studies; the systematic reviews and reviews/guidelines are discussed in Appendix 4 (available online at [http://hrbcentreprimarycare.ie/?q=cpr-register-paper-supplemental-data](http://hrbcentreprimarycare.ie/?q=cpr-register-paper-supplemental-data)).

Publication of original clinical prediction rule studies spanned 1965 to 2009 and became more common in recent years (Figure 2). Of the 895 original studies included in the register, the majority were published in the 2000s (68.2%), markedly more than were published in the 1990s (20.2%), the 1980s (10.8%), and 1965–1979 (0.8%). The majority of new clinical prediction rules were derived and validated in the 2000s. Of the 461 validation studies, more were narrow (57.4%) than broad (42.5%). The overall number of impact analysis studies was relatively very low.

![Figure 2](http://www.annfammed.org/https://www.annfammed.org/content/annalsfm/12/4/359/F2.medium.gif)

[Figure 2](http://www.annfammed.org/content/12/4/359/F2)

Figure 2 
Clinical prediction rule studies, split by decade reported and stage of development of the rule (N = 895).

Note: Few studies were reported before 1980; therefore, we grouped these studies into a broader time period (1965–1979).

### Clinical Prediction Rule Characteristics

#### Stage of Development

Of the 434 unique clinical prediction rules included in the analysis, 238 (54.8%) were associated with at least 1 validation study, although some were associated with more. The stages of development of all included rules are presented in Appendix 5 (available online at [http://hrbcentreprimarycare.ie/?q=cpr-register-paper-supplemental-data](http://hrbcentreprimarycare.ie/?q=cpr-register-paper-supplemental-data)).

The validation studies predominantly concentrated on certain clinical prediction rules: 34 focused on the Alvarado score for appendicitis14; 15 on the CRB-65 (Confusion, Respiratory rate, Blood pressure, aged 65 years and older) for pneumonia15; 12 on the CURB-65 (Confusion, Urea nitrogen, Respiratory rate, Blood pressure, aged 65 years and older) for pneumonia15; 12 on the Framingham Coronary Heart Disease rule16; 11 on the ABCD2 rule for stroke5; 11 on the Centor score for streptococcal pharyngitis4; 10 on the Ottawa Ankle Rule for need for radiograph after ankle injury17; and 10 on the Pneumonia Severity Index (PSI).18

Of the 434 unique clinical prediction rules, only 2.8% had been evaluated in at least 1 impact analysis study, although some had been evaluated in more (Table 1).19–34 The studies predominantly focused on a pair of rules: 4 focused on the Ottawa Ankle Rule and 2 on the McIsaac rule. The impact analysis studies included here had a mixture of study designs and outcome measures. Most studies reported improvement in primary outcomes with implementation of the rule.

View this table:
[Table 1](http://www.annfammed.org/content/12/4/359/T1)

Table 1 
Impact Analysis Studies of Clinical Prediction Rules (16 Studies Covering 12 Rules)

#### Clinical Domains

We classified the clinical prediction rule studies using *ICPC*-2 coding.35 The studies included in the register spanned 17 broad clinical domains (Figure 3 and Supplemental Appendix 3). The majority pertained to cardiovascular, respiratory, and musculoskeletal areas.

![Figure 3](http://www.annfammed.org/https://www.annfammed.org/content/annalsfm/12/4/359/F3.medium.gif)

[Figure 3](http://www.annfammed.org/content/12/4/359/F3)

Figure 3 
Broad clinical domains for the clinical prediction rule studies, split by stage of development of the rule (N = 895).

Note: Studies were classified according to the *International Classification of Primary Care*, 2nd Edition (ICPC-2).35

Differences existed in terms of the stages of development of the rules across clinical domains (Figure 3). Some areas (eg, neurologic) were associated with more derivation than validation studies, whereas others (eg, digestive) were associated with more validation than derivation studies. Just 5 of the clinical domains (digestive, cardiovascular, musculoskeletal, neurologic, and respiratory) were associated with any impact analysis, and only a small subset of the rules within each of these domains had undergone impact analysis.

The clinical domains with the largest number of clinical prediction rules derived were stroke/cerebrovascular accident (K90, 29 rules), risk factors for cardiovascular disease (K22, 26 rules), pneumonia (R81, 24 rules), influenza (R80, 17 rules), pulmonary embolism (K93, 16 rules), and osteoporosis (L95, 16 rules). Many of these rules had not gone through validation or impact analysis, however. For example, for the K90 rule for stroke/cerebrovascular accident, there were more derivation studies than validation studies (29 vs 27), and no impact analysis studies had been done. The validation studies focused on 13 rules, with the ABCD2 being the most commonly validated rule (11 studies).5 The mixed pattern observed here is similar to that observed across the remaining clinical domains.

#### Clinical Setting

The clinical prediction articles included in the register came from a range of clinical settings (Table 2). The most common settings were primary care (28.2%) and the emergency department (28.0%). A large proportion also came from the outpatient setting (13.6%) or hospital setting (16.8%). Few articles described research among patients from 2 settings, for example, primary care and the emergency department (0.7%) and primary care and the inpatient setting (0.3%).

View this table:
[Table 2](http://www.annfammed.org/content/12/4/359/T2)

Table 2 
Clinical Setting of Clinical Prediction Rule Studies (N=895)

## DISCUSSION

### Summary of Main Findings

The publication of clinical prediction rules has increased over time, particularly for cardiovascular, respiratory, and musculoskeletal conditions. Despite the large numbers of rules derived, however, only slightly more than one-half have been validated, and these validations are more likely to have been narrow than broad. Moreover, few rules have gone through impact analysis. Such limited evaluation makes it difficult to recommend clinical prediction rules uncritically for widespread clinical use. In this context, our development of an international register of clinical prediction rules relevant to primary care will help prioritize areas of research, as well as provide a platform for the development of CDSS tools that integrate clinical prediction rules into the point-of-care clinical decision support.

### Comparison With the Existing Literature

Ours is the first review of clinical prediction rules that specifically focuses on primary care. Although similar reviews have been conducted, they have generally been on a smaller scale, searching up to 6 medical journals and retrieving up to 61 rules over a short time frame.6,8,36,37 The exception is a review of clinical prediction rules for children, which retrieved 101 rules but searched only 2 resources.7 These studies have also indicated a pattern of limited validation and even more limited impact analysis. This lack of in-depth study of rules makes it difficult to determine the impact that they have on patient care, physician behavior, and health care costs.1,3,6

### Study Limitations

We took a broad approach to the clinical settings that were eligible for inclusion and included any clinical prediction rules that could be applied in a primary care setting. The majority of studies we included were conducted in primary care and emergency department settings. Some rules had been derived in hospital settings and subsequently validated in primary care (eg, the Centor score for streptococcal pharyngitis).4 The findings from these validation studies need to be considered in the context of the study setting, particularly with respect to the biases associated with patient selection in the different settings. Other rules had been derived for use in community settings but validated in hospital settings (eg, the CRB-65 for pneumonia).14 It is therefore necessary to adapt a broad approach to adequately quantify narrow and broad validation studies, and to account for differences between health care services and access to same-day technology across different countries. For this reason, not all included clinical prediction rules will apply to primary care in all countries.

Although we searched multiple resources to retrieve relevant articles for the register, the search was not designed to be a systematic review of any of the clinical prediction rules included here. Thirty primary care journals were purposively chosen from an extensive list of relevant journals as described in Supplemental Appendix 1. In addition, only 1 reviewer identified relevant articles during the article screening process using predefined inclusion criteria. As such, although the derivation article for every rule is included here, it is possible that some validation and impact analysis studies were not retrieved by our search strategy. It is also possible that we have overlooked clinical prediction rules relevant to primary care. Nevertheless, our review is substantially broader in scope than previous reviews.6–8,36,37

### Clinical Implications

The data we describe will be used to establish a web-based international register of clinical prediction rules relevant to primary care through the Cochrane Primary Health Care field, which we envisage will be available beginning in mid-2014. The coding of articles according to their clinical domain and stage of development will allow for easy navigation through the database.

The coefficients and algorithms for each of the clinical prediction rules have been extracted and will be used to develop a computer-based CDSS. This process will allow the pretest probability to be adjusted for any rules that were not derived or validated in a primary care setting. The development of a web-based CDSS should overcome some of the barriers associated with promoting the use of clinical prediction rules in routine clinical practice.38 Future work will focus on integrating such clinical decision support fully into the electronic health record, which will assist in the knowledge transfer of up-to-date clinical evidence that will be available for use at the point of care.

### Future Research

Our work highlights a number of areas for future research. The development of a register of clinical prediction rules will enable identification of clinical domains in which such rules have not yet been developed. Identifying these gaps may be particularly useful for determining where to focus research efforts to develop new rules, rather than simply adding to the mass of existing rules for a condition that is well covered. Where rules already exist, it is argued that these rules should be adapted to accommodate changes in clinical setting or to incorporate new evidence regarding predictor variables or changes in the management strategies of the population of interest.8,39 From a primary care perspective, research efforts should be focused on developing or adapting clinical prediction rules that safely rule out serious illnesses, given the lower prevalence of these conditions in this setting.

Clinical prediction rules with limited validation and impact analysis can also be identified. The systematic reviews of rules identified here offer a way to determine the predictive accuracy of the rule, which may help prioritize which rules should undergo impact analysis. The methodologic quality analysis may also be useful in this regard. Previous reviews have identified quality components, such as clinical sensibility, as important for selecting rules for impact analysis.6 These research efforts should comply with relevant methodologic quality checklists, to overcome the problems outlined here in this regard.

It is clear that establishing an international register of clinical prediction rules relevant to primary care will require checking of multiple resources. The numerous terms used to describe clinical prediction rules in the literature makes it difficult to develop a search string/term for use in electronic resources such as PubMed. Our search string has good sensitivity and specificity but was designed to search 30 key journals relevant to primary care. The range of search terms identified in the current study provides a means to develop new search strings to identify clinical prediction rules across a range of journals and clinical domains. Furthermore, the contribution of both clinical and methodologic experts on clinical prediction rules has been identified as important to the ongoing development of the register. We will continue to make appeals to the research community to keep the register up to date, in addition to searching other resources.

## Acknowledgments

We would like to acknowledge staff at the HRB Centre for Primary Care Research.

## Footnotes

*   Conflicts of interest: authors report none.

*   **Funding support:** This research was funded by the Health Research Board in Ireland under grant HRC/2007/1.

*   **Previous presentations:** Preliminary results of this article were presented as an oral presentation at a number of European and North American conferences, including the following: North American Primary Care Research Group (NAPCRG), December 1–5, 2012, New Orleans, Louisiana; Association of University Departments of General Practice in Ireland (AUDGPI) Annual Scientific Meeting 2013, March 8, 2013, Limerick, Ireland; Association of University Departments of General Practice in Ireland (AUDGPI) and the University Departments of General Practice in Scotland (ADEGS) joint meeting, January 22–23, 2011, Dublin, Ireland; European General Practice Research Network (EGPRN), October 14–17, 2010, Zurich, Switzerland; 39th Annual Scientific Meeting of the Society of Academic Primary Care (SAPC), July 7–9, 2010, University of East Anglia.

*   **Supplementary materials:** Available at [http://www.AnnFamMed.org/content/12/4/359/suppl/DC1](http://www.AnnFamMed.org/content/12/4/359/suppl/DC1)

*   Received for publication June 14, 2013.
*   Revision received September 18, 2013.
*   Accepted for publication December 3, 2013.


*   © 2014 Annals of Family Medicine, Inc.

## References

1.  McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS; Evidence-Based Medicine Working Group. Users’ guides to the medical literature: XXII: how to use articles about clinical decision rules. JAMA. 2000;284(1):79–84.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.284.1.79&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=10872017&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000087843000029&link_type=ISI) 

2.  Grobman WA, Stamilio DM. Methods of clinical prediction. Am J Obstet Gynecol. 2006;194(3):888–894.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.ajog.2005.09.002&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=16522430&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 

3.  Adams ST, Leveson SH. Clinical prediction rules. BMJ. 2012;344: d8312.
    
    [FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDQvamFuMTZfMS9kODMxMiI7czo0OiJhdG9tIjtzOjIzOiIvYW5uYWxzZm0vMTIvNC8zNTkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

4.  Centor RM, Witherspoon JM, Dalton HP, Brody CE, Link K. The diagnosis of strep throat in adults in the emergency room. Med Decis Making. 1981;1(3):239–246.
    
    [FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo1OiJzcG1kbSI7czo1OiJyZXNpZCI7czo3OiIxLzMvMjM5IjtzOjQ6ImF0b20iO3M6MjM6Ii9hbm5hbHNmbS8xMi80LzM1OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

5.  Johnston SC, Rothwell PM, Nguyen-Huynh MN, et al. Validation and refinement of scores to predict very early stroke risk after transient ischaemic attack. Lancet. 2007;369(9558):283–292.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/S0140-6736(07)60150-0&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=17258668&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000243912000032&link_type=ISI) 

6.  Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med. 2006;144(3):201–209.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.7326/0003-4819-144-3-200602070-00009&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=16461965&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000235074800007&link_type=ISI) 

7.  Maguire JL, Kulik DM, Laupacis A, Kuppermann N, Uleryk EM, Parkin PC. Clinical prediction rules for children: a systematic review. Pediatrics. 2011;128(3):e666–e677.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6InBlZGlhdHJpY3MiO3M6NToicmVzaWQiO3M6MTA6IjEyOC8zL2U2NjYiO3M6NDoiYXRvbSI7czoyMzoiL2FubmFsc2ZtLzEyLzQvMzU5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

8.  Steyerberg EW, Moons KG, van der Windt DA, et al; PROGRESS Group. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1371/journal.pmed.1001381&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=23393430&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 

9.  Keogh C, Wallace E, O’Brien KK, et al. Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish a Web-based register. J Clin Epidemiol. 2011;64(8):848–860.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2010.11.011&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=21411285&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 

10. Simel D, Rennie D. The Rational Clinical Examinations: Evidence Based Clinical Diagnosis. New York, NY: McGraw-Hill Medical; 2009.
    
    

11. Ebell M. Evidence-based Diagnosis: A Handbook of Clinical Prediction Rules. New York, NY: Springer; 2001.
    
    

12. Allen J, Gay BG, Crebolder H, Catholic JH, Svab I, Maastricht PR. The European Definition of General Practice/Family Medicine. Short version. WONCA Europe 2011. [http://www.unige.ch/medecine/umpr/Enseignement/stage3ememaster/Definition\_wonca\_tree.pdf](http://www.unige.ch/medecine/umpr/Enseignement/stage3ememaster/Definition_wonca_tree.pdf). Accessed Mar 18, 2014.
    
    

13. Fischer Walker CL, Rimoin AW, Hamza HS, Steinhoff MC. Comparison of clinical prediction rules for management of pharyngitis in settings with limited resources. J Pediatr. 2006;149(1):64–71.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.jpeds.2006.03.005&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=16860130&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000239352000020&link_type=ISI) 

14. Alvarado A. A practical score for the early diagnosis of acute appendicitis. Ann Emerg Med. 1986;15(5):557–564.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/S0196-0644(86)80993-3&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=3963537&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=A1986C028100009&link_type=ISI) 

15. Lim WS, van der Eerden MM, Laing R, et al. Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax. 2003;58(5):377–382.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToidGhvcmF4am5sIjtzOjU6InJlc2lkIjtzOjg6IjU4LzUvMzc3IjtzOjQ6ImF0b20iO3M6MjM6Ii9hbm5hbHNmbS8xMi80LzM1OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

16. Anderson KM, Wilson PW, Odell PM, Kannel WB. An updated coronary risk profile. A statement for health professionals. Circulation. 1991;83(1):356–362.
    
    [FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czoxNDoiY2lyY3VsYXRpb25haGEiO3M6NToicmVzaWQiO3M6ODoiODMvMS8zNTYiO3M6NDoiYXRvbSI7czoyMzoiL2FubmFsc2ZtLzEyLzQvMzU5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

17. Stiell IG, Greenberg GH, McKnight RD, et al. Decision rules for the use of radiography in acute ankle injuries. Refinement and prospective validation. JAMA. 1993;269(9):1127–1132.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.1993.03500090063034&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=8433468&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=A1993KN61100029&link_type=ISI) 

18. Fine MJ, Auble TE, Yealy DM, et al. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med. 1997;336(4):243–250.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1056/NEJM199701233360402&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=8995086&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=A1997WE63200002&link_type=ISI) 

19. Stanley AJ, Ashley D, Dalton HR, et al. Outpatient management of patients with low-risk upper-gastrointestinal haemorrhage: multicentre validation and prospective evaluation. Lancet. 2009;373(9657):42–47.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/S0140-6736(08)61769-9&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=19091393&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000262183800030&link_type=ISI) 

20. Farahnak M, Talaei-Khoei M, Gorouhi F, Jalali A, Gorouhi F. The Alvarado score and antibiotics therapy as a corporate protocol versus conventional clinical management: randomized controlled pilot study of approach to acute appendicitis. Am J Emerg Med. 2007;25(7):850–852.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.ajem.2007.01.012&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=17870498&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 

21. Koelewijn-van Loon MS, van der Weijden T, van Steenkiste B, et al. Involving patients in cardiovascular risk management with nurse-led clinics: a cluster randomized controlled trial. CMAJ. 2009;181(12):E267–E274.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTgxLzEyL0UyNjciO3M6NDoiYXRvbSI7czoyMzoiL2FubmFsc2ZtLzEyLzQvMzU5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

22. Montgomery AA, Fahey T, Peters TJ, MacIntosh C, Sharp DJ. Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. BMJ. 2000;320(7236):686–690.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjAvNzIzNi82ODYiO3M6NDoiYXRvbSI7czoyMzoiL2FubmFsc2ZtLzEyLzQvMzU5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

23. Pozen MW, D’Agostino RB, Selker HP, Sytkowski PA, Hood WB Jr. A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease. A prospective multicenter clinical trial. N Engl J Med. 1984;310(20):1273–1278.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1056/NEJM198405173102001&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=6371525&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=A1984SR28000001&link_type=ISI) 

24. Kline JA, Webb WB, Jones AE, Hernandez-Nino J. Impact of a rapid rule-out protocol for pulmonary embolism on the rate of screening, missed cases, and pulmonary vascular imaging in an urban US emergency department. Ann Emerg Med. 2004;44(5):490–502.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.annemergmed.2004.03.018&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=15520709&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000224768800008&link_type=ISI) 

25. Wells PS, Anderson DR, Rodger M, et al. Evaluation of D-dimer in the diagnosis of suspected deep-vein thrombosis. N Engl J Med. 2003;349(13):1227–1235.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1056/NEJMoa023153&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=14507948&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000185488800005&link_type=ISI) 

26. Bessen T, Clark R, Shakib S, Hughes G. A multifaceted strategy for implementation of the Ottawa ankle rules in two emergency departments. BMJ. 2009;339:b3056.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzMzkvYXVnMTJfMy9iMzA1NiI7czo0OiJhdG9tIjtzOjIzOiIvYW5uYWxzZm0vMTIvNC8zNTkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

27. Auleley GR, Ravaud P, Giraudeau B, et al. Implementation of the Ottawa ankle rules in France. A multicenter randomized controlled trial. JAMA. 1997;277(24):1935–1939.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.1997.03540480035035&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=9200633&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=A1997XF08700037&link_type=ISI) 

28. Stiell I, Wells G, Laupacis A, et al; Multicentre Ankle Rule Study Group. Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries. BMJ. 1995;311(7005):594–597.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTEvNzAwNS81OTQiO3M6NDoiYXRvbSI7czoyMzoiL2FubmFsc2ZtLzEyLzQvMzU5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

29. Stiell IG, McKnight RD, Greenberg GH, et al. Implementation of the Ottawa ankle rules. JAMA. 1994;271(11):827–832.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.1994.03510350037034&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=8114236&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=A1994MZ49800028&link_type=ISI) 

30. Stiell IG, Wells GA, Hoag RH, et al. Implementation of the Ottawa Knee Rule for the use of radiography in acute knee injuries. JAMA. 1997;278(23):2075–2079.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.1997.03550230051036&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=9403421&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=A1997YK98700036&link_type=ISI) 

31. Stiell IG, Clement CM, Grimshaw J, et al. Implementation of the Canadian C-Spine Rule: prospective 12 centre cluster randomised trial. BMJ. 2009;339:b4146.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1136/bmj.b4146&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=19875425&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 

32. Worrall G, Hutchinson J, Sherman G, Griffiths J. Diagnosing streptococcal sore throat in adults: randomized controlled trial of in-office aids. Can Fam Physician. 2007;53(4):666–671.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiY2ZwIjtzOjU6InJlc2lkIjtzOjg6IjUzLzQvNjY2IjtzOjQ6ImF0b20iO3M6MjM6Ii9hbm5hbHNmbS8xMi80LzM1OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

33. McIsaac WJ, Goel V, To T, Permaul JA, Low DE. Effect on antibiotic prescribing of repeated clinical prompts to use a sore throat score: lessons from a failed community intervention study. J Fam Pract. 2002;51(4):339–344.
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=11978257&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000174879500009&link_type=ISI) 

34. McIsaac WJ, Goel V. Effect of an explicit decision-support tool on decisions to prescribe antibiotics for sore throat. Med Decis Making. 1998;18(2):220–228.
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=9566455&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000073022700010&link_type=ISI) 

35. 1.  Miller G
    
    , ed. World Organization of Family Doctors. International Classification of Primary Care. 2nd ed. Wonca International Classification Committee (WICC); 2004.
    
    

36. Wasson JH, Sox HC, Neff RK, Goldman L. Clinical prediction rules. Applications and methodological standards. N Engl J Med. 1985;313(13):793–799.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1056/NEJM198509263131306&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=3897864&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=A1985ARA8900006&link_type=ISI) 

37. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA. 1997;277(6):488–494.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.1997.03540300056034&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=9020274&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=A1997WG05400037&link_type=ISI) 

38. Lang ES, Wyer PC, Haynes RB. Knowledge translation: closing the evidence-to-practice gap. Ann Emerg Med. 2007;49(3):355–363.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.annemergmed.2006.08.022&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=17084943&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000244637000020&link_type=ISI) 

39. Toll DB, Janssen KJ, Vergouwe Y, Moons KG. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol. 2008;61(11):1085–1094.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2008.04.008&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=19208371&link_type=MED&atom=%2Fannalsfm%2F12%2F4%2F359.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000259955800003&link_type=ISI)