Importance of Evidence Grading for Guideline Implementation: The Example of Asthma

I presented this article to 20 other Family Medicine Residents at our monthly journal club and got a good response. There were many good comments that I will summarize here.

Overall it was an "eye opener" that evidence grading guidelines in use have many flaws. Many guidelines don't give any information to help the patient make an informed decision, especially when the evidence is weak. We agree that NNT and NNH should be included in any evidence grading guidelines. As a group we decided to not take evidence grading guidelines at face value, but to do our own research and be better informed.

The article as whole was great. We did have a few questions and suggestions. 1- Why only show figures of EPR3 and SORT, and then in your concluding remarks recommend GRADE. 2- Including a figure or table of GRADE would be helpful to let us compare. 3- SORT and GRADE were not mentioned in the abstract or as key words when searching for this article.

Competing interests:   None declared

Drs. Lee Green, Kevin Pottie and Chris van Weel each amplify and extend aspects of my article. I appreciate and largely agree with their thoughtful comments.

US family physicians tend to be deferential to an expansive notion of the expertise of disease-oriented specialists, probably because these specialists played such a large role in our training as medical students and as residents. It is time to acknowledge, as Dr. Green implies, that clinical expertise in a content area is not the same thing as expertise in evidence-based critical appraisal and guideline development. I agree with Dr. Green that the transition at NIH from expert panel guidelines to evidence-based guidelines is incomplete. I would add, however, that the amount of NIH effectiveness research is disease-specific. For example, patient-oriented effectiveness studies (“large simple trials”) of heart disease have been performed since the 1960s; and death from cardiovascular disease is on the decline. On the other hand, lung disease research continues to disdain effectiveness; and death from chronic lung disease continues to rise (1).

Dr. Pottie advocates a broader assessment of asthma guidelines using the AGREE instrument, and I “agree” that this instrument merits more general use. I regret to say that I would award the EPR-3 asthma guideline high marks in only 2 of the 23 key categories (#13. The guideline has been externally reviewed by experts prior to its publication and #17. Key recommendations are easily identifiable). It would be interesting and perhaps important to research the question How much would a panel of family physicians collectively AGREE with the group of practice guidelines that we are expected to implement? One of my primary goals was to compare the EPR-3 evidence grading system to an instrument (SORT) that was familiar to (and hopefully used by) family physicians. GRADE certainly deserves more exposure in the primary care arena as arguably the most comprehensive evidence-grading system available. I also agree that there remain significant challenges to understanding and responding to patient values. An issue that particularly intrigues and perplexes me is how to approach patients who prefer not to engage in shared decision- making.

Dr. van Weel emphasizes the importance of incorporating patient preferences into guidelines, notes the deficiency of relevant effectiveness research applicable to primary care and advocates the need to change the research culture, including involving patients in the design of effectiveness research. Not all current guideline developers endorse this “holistic” approach. Analogous to the “splitters” and “lumpers” that I described in my paper, guideline “splitters” believe that patient preferences and implementation tools should be addressed separately from the guideline itself. I, along with Dr. van Weel, prefer the “lumping” approach wherein relevant research evidence is contextualized and made available for shared decision-making within a unitary document. This controversy, along with the other important issues raised by Drs. Green, Pottie and van Weel, deserves vigorous ongoing examination and I am pleased to be part of that debate.

(1) Hurd SS, Lenfant C. COPD: good lung health is the key. Lancet 2005;366:1832-4.

Competing interests:   None

For the appropriate use of evidence in practice guidelines, a critical analysis of published research papers is essential. This has been the rationale behind the professionalization of literature reviewing, as done for example by the Cochrane Collaboration. As David Hahn illustrated in the thoughtful review of the example of asthma guidelines, still a lot remains to be desired [1]. Even with an abundance of available guidelines developed by authoritative international bodies, practitioners are still presented with guidelines that are deficient for the essential clinical decisions they have to take. Hahn stresses the importance of patient preferences, and there can be little doubt of its relevance if people are to be at the centre of health care. This is the reason that patient preferences should be an integral part of guidance and not only when evidence is weak (as Hahn states [1]) or side effects are substantial.

We would like to underline this with two illustrations. First, the importance of patient preferences was clearly experienced by our research group in a study on the early treatment of asthma and COPD – the DIMCA program [2]. In this program, participants were recruited from the primary care population for minimal signs symptoms of asthma or COPD and were followed for two years to observe the symptoms progression. This was followed by a randomized controlled trial of the efficacy of inhaled corticosteroids for those who developed predefined levels of signs and symptoms [3]. Inhaled corticosteroids are the state of the art treatment of asthma, but what is interesting in this context were our experiences in the trial recruitment and participation. Although the participants had demonstrated a most valued compliance with the two years of follow-up – including spirometry and blood testing – a substantial number declined participation in the subsequent steroid trial [4]. The perceived negative effects of what clinicians see as the state of the art intervention was a main factor that stood in the way of early treatment. Secondly, , the Netherlands Asthma Foundation Found a striking contrast between the preferences of the patients with COPD and the prevailing Dutch evidence based guidelines. From this, we conclude that there is a need for shared decision making and research that addresses its effects in patient care.

The recurrent theme of Hahn’s review is the mismatch of research evidence and the challenges of patient care. Grading of evidence is helpful in identifying the most useful research. This is particularly helpful in addressing the internal validity of research – the extent to which the study design did live-up to its findings. But as Hahn makes clear, the major problem for the quality of guiding patient care is the external validity – the extent to which essential features of the clinical decisions are represented in research. The example of patient preferences is just one of where research is often failing clinicians. In particular for primary care the external validity is a problem, as most clinical research is still done in secondary or tertiary care settings. This leads to the frustrating paradox that what is seen most often is studied the least. As an example could serve the international GOLD guideline[5]. Although most patients with COPD are graded stage I and II mild to moderate disease, most of the research – and consequently most of the guideline – is directed at stages III and IV. This is inevitable under the rules of guideline development but leaves family physicians (FP)who treat most of the patients with stage I and II out in the cold. Though grading can be helpful, it can only do so much, and the review of Hahn [1] should at least be seen as well as a strong argument to change the research behind it. The reason for clinical research is to improve the care of patients by making it more effective and safer and this is the only convincing argument to expose patients to research conditions. When the reality is that clinical care is poorly supported – because researchers have studied the wrong patients, or the wrong questions – it is time to change the research culture. To address the challenges of primary care (patients’ preferences, joint decision making, comorbidity, uncertainty of the prognosis of early signs and symptoms), it is not enough that researchers base their research in the primary care population. A partnership between research and practice, between researchers and FPs, is needed to design research that will in turn deliver the evidence FPs need. This is why primary care patients need research that is done in primary care, for primary care, by primary care. And this why it is attractive to seek the involvement of patients, as partner, in the design of studies.

References 1. Hahn DL. Importance of Evidence Grading for Guideline Implementation: The Example of Asthma. Ann Fam Med 2009; 7:364-369
2. Albers JMC, Schermer TRJ, Boom G van den, Akkermans RP, Schayck CP van, Herwaarden CLA, Weel C van. Efficacy of Inhaled Steroids in Undiagnosed Subjects at High Risk for COPD: Results of the Detection, Intervention, and Monitoring of COPD and Asthma Program. Chest 2004;126:1815-1824.
3. Boom G van den, Rutten-van Mölken MPMH, Molema J, Tirimanna PRS, Weel C van, Schayck CP van. The cost effectiveness of early treatment with fluticasone propionate 250 µg twice a day in subjects with obstructive airway disease. Am J Respir Crit Care Med 2001;164:2057-66.
4. Grunsven, P.M. van, Schayck, C.P. van, Kollenburg, H.J.M. van, Bosheide, K. van, Hoogen, H.J.M. van den, Molema, J., Weel, C van. The role of "fear of corticosteroids" in nonparticipation in early intervention with inhaled corticosteroids in asthma and COPD in general practice. Eur Respir J 1998, 11, 1178-1181.
5. Rabe KF, Hurd S, Anzueto A, Barnes PJ, Buist SA, Calverley P, Fukuchi Y, Jenkins C, Rodriguez-Roisin R, Weel C van, Zielinski J. Global Strategy for the Diagnosis, Management, and Prevention of COPD - 2006 Update. Am J Respir Crit Care Med 2007;176(6):532-55

Competing interests:   We are involved in research of patients with asthma and COPD and also involved in guideline development for primary care

Dr. Hahn’s article, the Importance of Evidence Grading for Guideline Implementation, is both timely and thought provoking. His critique of the EPR-3 Asthma Guidelines illustrates both the importance and the complexity of critically appraising evidence-based guidelines.

There is clearly a need for explicit and consistent reporting of guideline development processes and the factors that result in the making of clinical recommendations. Decision makers, be they clinicians, patients, or policymakers, need to understand the rationale behind recommendations, rationales that now often extend well past expert opinion.

I certainly agree that an evidence grading system is essential, and that in making a recommendation guideline authors should explicitly comment on the balance of desirable and undesirable patient important outcomes (benefits/ harms), the quality of evidence, and pertinent values and preferences that may have played out in making the recommendation. To avoid confusion, the GRADE Group argues for the importance of clearly separating decisions relating to the quality of evidence and the strength of recommendation. (Guyatt et al. GRADE: an Emerging Consensus, BMJ, 2008)

Shared decision making means both patient and practitioner need to be able to understand the rationale behind a recommendation. The GRADE Group notes that this becomes especially important in cases where the quality of evidence is low and where there are tradeoffs between benefits and harms of an intervention, or where values and preferences of the guideline committee may have played a significant role in the making of a recommendation. I would agree with Dr. Hahn that guideline development committees can highlight value and preference considerations, and that it is the practitioner who should elicit patient preferences to support shared and informed decision making.

The science of evidence synthesis and knowledge translation (including practice guidelines) is emerging as an important resource to improve the quality of patient care. However clinical practice guideline development is a new science and confusion around terminology, synthesis and evidence grading systems, and reporting remains a problem. I would have liked to see Dr. Hahn utilize the Appraisal of Guidelines Research and Evaluation (AGREE) guideline appraisal instrument (http://www.agreecollaboration.org/) as a global framework in appraising the Asthma guidelines. AGREE consists of 23 key items organized in six domains to capture a separate dimension of guideline quality: scope, purpose, stakeholder involvement, rigour of development, clarity of presentation, and applicability. I would also have liked to see Dr. Hahn present the background to the GRADE Group (Guyatt et al 2008) rather than attempt to present both SORT (Strength of Recommendation Taxonomy) and GRADE arguments together. And I would have liked more recognition of the limitations and challenges that continue in the area of documenting, synthesizing and reporting values and preferences.

The recent publication by David Moher and colleagues, Preferred Reporting Items for Systematic Review and Meta-Analysis: The PRISMA Statement (PloS Medicine, 2009) provides yet another building black in the foundation of synthesis and knowledge translation. Engaging family physicians in the methods and reporting debates around guidelines is an important step to improving guidelines for primary care. I applaud Dr. Hahn for pushing us in this direction.

Sincerely,

Kevin Pottie MD CCFP MClSc FCFP Associate Professor, Departments of Family Medicine and Epidemiology and Community Health University of Ottawa

References: Guyatt GH, Oxman AD, Gunn EV, Kunz R, Falch-Ytter, Alsonso-Coello P, Schunemann, for the GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924-926.

AGREE: Appraisal of guidelines research and evaluation. Health Care Evaluation Unit at St George's Hospital Medical School in London 2004 Available from: URL: http://www.agreecollaboration.org/

Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097 (available at: http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1000097)

Competing interests:   None declared