Clinical Prediction Rules: Challenges, Barriers, and Promise

• decision support techniques
• risk

In this issue of Annals of Family Medicine, an article by Stam and colleagues focuses on the derivation and validation of a clinical prediction rule (CPR) aiming to predict which older people presenting with dizziness to their primary care physician will be more likely to have a poor prognosis.¹ Dizziness is somewhat common in primary care and can be debilitating, diagnostically challenging, and difficult to treat.² This CPR has potential to aid clinical decision making by tailoring treatment to older people at high risk of experiencing a poor outcome. Despite the great promise of CPRs, however, questions remain about whether their use will eventually lead to improvements in care delivery or improved outcomes for patients.

CPRs aim to standardize and increase the accuracy and consistency of clinicians’ diagnostic and prognostic assessments and management decisions.³ With high levels of interest in CPRs (Figure 1), large numbers of CPRs have been derived and validated but comparatively very few have been evaluated in impact analysis studies.⁴ Examples of well-known CPRs include the prognostic CHA2DS2-VASc score, designed to predict the risk of ischemic stroke in patients with atrial fibrillation, and the diagnostic Centor score, used to identify patients with group A beta-hemolytic streptococcal throat infections who would benefit from antibiotic treatment.^5,6

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Clinical prediction rule studies, split by decade reported and stage of development of the rule (N = 895)⁴

Reprinted with permission from Annals of Family Medicine.

If CPRs are to truly improve the quality of patient care then evaluation of these tools on relevant processes of care and patient outcomes is crucial. A systematic review that examined CPR impact analysis studies relevant to primary care reported that implementation was restricted to a few clinical domains, mainly musculoskeletal, cardiovascular, and respiratory.⁷ Of 18 included studies, 10 demonstrated an improvement in primary outcome with CPR use when compared with usual care, with one-half focusing on changing physician behavior in ordering imaging for patients presenting with musculoskeletal injuries. Few studies provided data about the continued use of successful interventions.⁷

The issue of methodological quality has recently been addressed with the publication of 2 standardized reporting guidelines for CPR derivation and validation studies and systematic reviews of CPRs.^8,9 These guidelines will have an important role to play in standardizing CPR research and in promoting robust validation of CPRs that should then be prioritized for evaluation in future impact analysis studies. However, uncertainty remains about the costs and potential unforeseen consequences of CPRs and the transportability of CPRs to other settings or health systems.¹⁰ Types of CPRs used in clinical practice varies, but use has been dictated by perceived clinical utility, familiarity, and local policy requirements.¹¹ Prioritizing the evaluation of select clinically useful, adequately validated CPRs with proven predictive accuracy could add to this evidence base and facilitate, if appropriate, the inclusion of these select CPRs into clinical practice guidelines and potentially widespread clinical use.

Several challenges have been identified in evaluating the effectiveness of CPRs in RCTs. First, implementing CPRs is challenging on a wide scale as integration into a clinician’s workflow is critical to improvement in care. Computerized clinical decision-support systems are effective at improving process-of-care measures but evidence for clinical and economic outcomes remains more limited.¹² A recent large-scale US study (n = 168 primary care providers, n = 40,003 patient consultations) demonstrated the promise of CPRs by finding reduced rates of antibiotic prescribing for patients presenting with respiratory symptoms and sore throat following the integration of 2 respiratory CPRs into the electronic health record.¹³ Importantly, clinicians were involved throughout the CPR process.

Other challenges important to consider include barriers to clinicians’ knowledge acquisition (eg, access to the literature, interpretation of the CPR), attitudinal barriers (eg, perceived clinical utility and confidence in the CPR), and behavioral barriers including environmental factors (eg, local contextual factors), patient factors (eg, shared decision making), institutional factors (eg, degree of support locally), and fear of litigation.^14,15 For example, the impact of the Canadian computed topography (CT) head rule was evaluated in a before-after study of patients (n = 4,531) presenting with intracranial bleeding following minor head injury.¹⁶ Despite this CPR having a sensitivity rate of 100% it did not impact on CT imaging rates. In process evaluation, clinicians’ reported unease with certain components of the CPR, and fear of missing this high-stakes diagnosis as reasons for not adopting the CPR.¹⁶

CPRs are also increasingly advocated for risk stratification at a population level to predict utilization outcomes such as 30-day readmission or emergency hospital admissions. Financial incentives have been introduced to reduce these types of events in both the United States and the United Kingdom. Risk stratification tools for emergency hospitalization will only identify a minority of patients who are at high risk of an event, in part because of the modest predictive accuracy of these tools^17,18 and because, in absolute terms, most emergency admissions occur in patients stratified as lower risk.^19,20 Hundreds of citations evaluating the predictive characteristics of risk stratification CPRs such as the LACE Index Scoring Tool for Risk Assessment of Death and Readmission exist, but there is notably less evidence evaluating their effect on care.²¹ While it is certainly possible that efforts to design systems around “risklevels” might improve care, there is a real possibility that these systems might increase costs without related benefit,²² and could inadvertently lead to less focus on more systemic improvements.²³

With increasing interest in developing and validating CPRs, it is important to consider how best to implement these tools at the point of clinical care to improve processes of care, patient outcomes, and system efficiency. Selecting CPRs in relevant clinical domains with proven predictive accuracy for impact analysis is essential. However, it is also critically important to anticipate barriers to the impact of CPRs such as clinicians’ beliefs, local contextual factors and how best to integrate the CPR into different clinical workflows. Without due consideration of these issues, the great promise of CPRs to aid clinical decision making may not be realized.

• Received for publication July 31, 2018.
• Revision received August 14, 2018.
• Accepted for publication August 14, 2018.

• © 2018 Annals of Family Medicine, Inc.