The road to effective clinical decision support: are we there yet?

Have we arrived at an understanding of what factors make clinical decision support effective? The short answer is “no.” In fact, our knowledge of this field may have taken a step backwards.

Reams of new scientific evidence are published each year, and it is increasingly difficult for practitioners to integrate this tsunami of best practice information into routine care.1 The use of electronic health records, particularly those with integrated decision support—tools through which patient specific information is intelligently processed to provide patient specific guidance—is touted as a means of promoting evidence based care.2 3 Ironically, however, evidence on what constitutes effective decision support is sorely lacking.

In a linked paper (doi:10.1136/bmj.f657), Roshanov and colleagues report a regression analysis of data abstracted from 162 randomized controlled trials. The analysis aims to identify which factors enable effective decision support.4 Most studies to date have sought simply to determine whether or not decision support tools could modify clinical process measures or outcomes. A recent systematic review found that these tools can improve the ordering of appropriate tests and treatments, and that this improvement is now apparent across community based sites with commercial decision support systems and is not restricted to academic medical centers using homegrown systems.5 However, few studies have sought rigorous evidence to determine what factors contribute to a system’s effectiveness.

To date, system design has largely been guided by expert opinion or limited interaction with intended users. To improve on this anecdotal approach, Bates and colleagues drew from years of observation and experience to generate the “ten commandments” of effective clinical decision support.6 More recently, a systematic review of 100 studies found that systems that automatically prompted users were more effective than those that required users to activate the tool themselves.7 A systematic review of data from 70 randomized controlled trials found that practice improved if decision support was provided as part of clinician workflow, if it was provided at the time and location of decision making, and if it entailed recommendations rather than just assessments.8 However, this study openly recognized that the regression model could have been “over fitted” because of a suboptimal ratio of cases relative to explanatory variables.8

Roshanov and colleagues’ meta-regression advances the field in that the authors applied rigorous methodology by contacting study authors to verify the accuracy of the data abstraction and to supply missing data. They also limited the primary analysis to six factors to maintain a favorable event-per-independent-variable ratio and tested the resultant models using different statistical methods to show that the identified associations persisted regardless of the analytic technique used.

Although many factors influence the usability and effectiveness of decision support systems, unfortunately, the relative paucity of good quality studies allowed only six factors to be assessed with statistical rigor. Consequently, the study outcomes, although robust, are limited in their ability to inform system development and implementation. The authors found that only two modifiable factors—providing advice to practitioners and patients and requiring practitioners to justify why they over-ride advice—were independently associated with improved clinical outcomes. The implications of these findings for clinical practice are that advice should increasingly be used to engage patients directly in the decision process and that practitioners may need to become more proactive in justifying why they elect not to follow advice.

Alarmingly, the study found that advice presented within electronic charting or order entry systems was negatively associated with improved outcomes. This finding could represent an unintended consequence of decision support technology and a possible step backwards. Perhaps the ramifications of this strong negative association are the most salient findings of this study. Although increasing emphasis is placed on using decision support within electronic health records, more information is needed to understand potential detrimental effects. Is the negative association due to alert fatigue? Are the integrated systems too distracting? Are such systems “user hostile” in some other way?

Although the current study has provided some insight into how best to develop clinical decision support, it has also shown that traditional research based on randomized controlled trials will not generate sufficient data to allow rigorous assessment of the dozens of factors that may contribute to effectiveness of decision support. Looking forward, we need to acknowledge that the evidence based approach will never adequately inform the fast paced and ever changing development of decision support. The time has come to strike off on a different path. The field should embrace a continuous quality improvement approach through which real world field based observations on the use of decision support, combined with qualitative contextual evaluation, can inform developers and users about what does and does not work. Accordingly, decision support systems should be increasingly created to allow this type of real time monitoring and evaluation.

So for now, the journey toward what constitutes effective clinical decision support is far from over.