| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH |
|
|
|
|||||||
|
|||||||||
TRACK to:
|
|
Electronic letters published:
![[Read Comment]](/icons/misc/sectionDOWN.gif){kind=icon}
Let's get started
Benefit-harm trade-off method has important implications
Reply to Dr. Hahn
The Personal Sufficiently Important Difference (pSID): Does it have application to clinical research?
|
|
|||
|
Bruce P Barrett, Madison, Wisconsin, USA Faculty physician & researcher
Send response to journal:
|
I couldn't agree more with Dr. Herbert's astute comments. In the interests of full disclosure I must say that I am honored and excited about working with him towards estimation of sufficiently important difference (aka. smallest worthwhile effect) for low back pain. Who knows, maybe we will meet in person some day! I imagine that Dr. Herbert and I were not the first people to realize that benefits should outweigh harms, and that randomized trials should be powered for - and interpreted in light of - the extent that they do. It may also be worth mentioning that benefit harm trade-offs are only one means towards SID/SWE estimation. Others will no doubt emerge, and I would challenge readers to think about other ways to define and investigate SID/SWE. The over- arching goal would be to have reasonable confidence that most of our treatments provide positive net benefit, rather than simply statistical superiority in a single benefit domain. Competing interests: None declared |
|||
|
|
|||
|
Robert D Herbert, Sydney, Australia Associate Professor, University of Sydney
Send response to journal:
|
The work of Barrett and colleagues has some immediate implications. It suggests that most people presenting to primary care with a common cold are unlikely to receive enough benefit from vitamin C, echinacea, zinc lozenges or pleconaril to be satisfied that the intervention was worthwhile. But the real significance of this study, and the two papers that precede it,[1,2] is that it provides a rigorous method (the "benefit- harm trade-off method") for determining what individual patients consider to be worthwhile effects ("sufficiently important differences"). Current practice is to consider that almost any health intervention shown to have statistically significant beneficial effects in a high quality randomised trial should be offered to the relevant patients, provided the intervention does not have serious side effects. This is bad medicine. A far better approach would be to offer only those interventions whose effects are likely to be big enough to be considered worthwhile by a substantial proportion of patients.[3] Now that there is a rigorous method for determining sufficiently important differences an effort should be made to systematically investigate the distributions of sufficiently important differences of all major health interventions. As Barrett and colleagues suggest, it is likely that such an exercise would show that many widely accepted health interventions, and many new interventions, have effects that are too small to make them worth implementing. If these data were taken seriously they could change the landscape of health care from an industry which provides any statistically significant intervention to one which provides only those interventions which have effects that are big enough to be worthwhile. The benefit-harm trade-off method should also have important implications for the design of clinical trials. Clinical trials should be designed to have sample sizes that are large enough to detect the smallest worthwhile effect, but the methods traditionally used to determine smallest worthwhile effects have been dubious at best. (For an exception see reference 4.) All clinical trials could now be preceded by an investigation, using the benefit-harm trade-off method, to determine what constitutes the sufficiently important difference for most patients, and powered accordingly. The sufficiently important differences identified using this method are likely to be larger than those usually nominated as the smallest worthwhile effect in most clinical trials, so the consequence would be to reduce the sample size and expense of many trials. 1. Barrett B, Brown D, Mundt M, Brown R. Sufficiently important difference: expanding the framework of clinical significance. Med Decis Making 2005;25:250-61. 2. Barrett B, Brown R, Mundt M et al. Using benefit harm tradeoffs to estimate sufficiently important difference: the case of the common cold. Med Decis Making 2005;25:47-55. 3. Herbert RD. How to estimate treatment effects from reports of clinical trials. I: continuous outcomes. Australian Journal of Physiotherapy 2000;46:229-235. 4. Willan AR, Pinto EM. The value of information and optimal clinical trial design. Stat Med 2005;24:1791-806. Competing interests: The author of this response commenced collaboration with Dr Barrett on a related project after this work was conducted. |
|||
|
|
|||
|
Bruce Barrett, Madison Wisconsin USA Family physician faculty, University of Wisconsin
Send response to journal:
|
It was gratifying to read comments from my friend and colleague David Hahn. First, I must say that I agree with the substance entirely. Certainly, there should be "a more patient-centred approach to determining clinically important effect sizes for randomized treatment trials," as Naylor and Llewellyn-Thomas wrote in 1994.1 Yes, clinical trials should be based on effects sizes that matter to patients, and yes, assessment of those SIDs should be incorporated into the work of designing, implementing and interpreting those trials. Dr. Hahn's proposal to distinguish personal SID as pSID seems reasonable. Our research suggests that each person can designate a magnitude of benefit that justifies costs and risks. When looking at SID across populations, terms such as "mean SID," "median SID," or "inter-quartile SID" can be used, and, optimally, actual distribution patterns displayed in tables or figures. In struggling with how group summaries of individual SIDs should be interpreted and used, I have conceived of "SID50," "SID10," or "SID90" as potentially important benchmarks. The SID50 is the effect size which divides the population in half, equivalent to "median SID." SID90 is the effect size that would justify treatment for at least 90% of patients. My own value judgment is that trials should be powered (and interpreted) for effect sizes in this range. I would also argue that an effect size equivalent to SID10 would speak against inclusion in common practice, and would not be an appropriate effect size for powering a trial. If 10% or less of the intended recipients would judge an intervention as worthwhile, then perhaps that intervention should be questioned if not discouraged. I would also hazard the guess is that adoption of this approach would lead to substantial monetary cost savings, as well as reduction in side effects, adverse events, and lost opportunity costs. Comparing clinicians' and patients' outcome-valuation is indeed high priority. As a first attempt at addressing this issue, we completed a survey of family physicians and researchers regarding SID in common cold, now in press in the Wisconsin Medical Journal.2 I hope I don't spoil the punch line by saying that physicians show considerable variability not only in how they value outcomes, but in how they interpret evidence. I am looking forward to Dr. Hahn’s upcoming article. I have long wondered why individual physicians and entire health systems are held accountable for delivering interventions with minimal effect size and potentially negative overall value. Mammography may be one example. Blood-sugar-lowering pills to improve diabetic outcomes may be another. Unless we adequately address the question of SID (aka. smallest worthwhile difference3;4) we will never know whether benefits attributable to interventions are worthwhile, according to the values of intended recipients. Sorting out which of the many interventions now available are actually worthwhile can and should be the next great task for medicine. Bruce Barrett June 11, 2007 Reference List 1. Naylor CD,.Llewellyn-Thomas HA. Can there be a more patient- centred approach to determining clinically important effect sizes for randomized treatment trials? Journal of Clinical Epidemiology. 1994;47:787 -95. 2. Barrett B, Endrizzi S, Andreoli P, Barlow S, Zhang Z. Clinical significance of common cold treatment: Professionals' opinions. Wisconsin Medical Journal 2007;Accepted. 3. Herbert RD. How to estimate treatment effects from reports of clinical trials. I - Continuous outcomes. Australian Journal of Physiotherapy 2000;46:239-45. 4. Yelland MJ,.Schluter PJ. Defining worthwhile and desired responses to treatment of chronic low back pain. Pain Med 2006;7:38-45. Competing interests: None declared |
|||
|
|
|||
|
David L. Hahn, Madison, Wisconsin, USA Physician
Send response to journal:
|
Evidence-based practice (EBP) aspires to combine the best clinical evidence, informed patient preferences and clinical experience to optimize the value of medical care to patients. Benefit/harm trade-offs are an integral part of EBP and are often extremely challenging to formulate because of current limitations in the production of clinical evidence and in the practice of shared decision-making. In the pursuit of EBP, some progressive medical practices have adopted the “Value Equation” as the basis for medical decision-making, where “Value” = “Quality”/”Cost” (V=Q/C). By analogy, effect sizes in clinical trials can be represented by Q, which is an incomplete metric because what patients, providers and payors ultimately value is “Value” itself (V). In this issue of the Annals of Family Medicine, Barrett et al. [1] propose a metric they term the “sufficiently important difference (SID)” that is analogous to V and has potential to more closely integrate clinical trial results and patient preferences to enhance the value of EBP. After reading their article I would like to offer three observations extending their concept of SID to related venues. First, I note the major variability in patient preferences as exemplified in Figure 1. and as commented upon by the authors. Physicians who have practiced medicine for more than a microsecond (or have read any political news recently) cannot be surprised by the large variability in what individuals value. A single measure of central tendency (e.g., mean or median SID) as a benchmark patient preference outcome cannot account for these large variations in how individuals value medical outcomes. Just as current biomedical research is beginning to aim at optimizing diagnosis and treatment based on relevant individual genetic variations, clinical research should begin to explore ways to optimize application of the SID concept in clinical research. In this pursuit, I offer up the following radical suggestion for debate: In clinical trials that measure an outcome that is amenable to SID measurement, why not ask each subject to pre-specify their “personal SID (pSID)”? The pSID value could then be used as a cut-off to determine whether the treatment (intervention/placebo) was a “success” or a “failure” for each subject. Initially, the pSID could be specified as a secondary outcome; if the pSID offers good value in these exploratory analyses, it might eventually become the primary outcome in clinical trials that inherently integrate effect sizes and patient preferences in the outcome. Second, I wonder whether measurement of SIDs in physicians and in their patients would yield equivalent results. Or would systematic differences be present? If so, this might be important information to have when confronting the deep (and often unconscious) tendency towards paternalism within the medical profession. Third, the concepts of SID and pSID may have applications in the realm of informed patient preferences and shared decision-making, which are currently problematic areas for physicians who currently lack useful evidence-based patient information materials to share with patients on many important topics. I refer you to the upcoming issue of the Journal of Family Practice in which I present and discuss benefit and harms of mammography screening for breast cancer in women ages 40-49. After reading the Commentary, I ask you (if you are female) to consider your pSID and compare it to the global recommendation made by the United States Preventive Services Task Force (USPSTF). 1. Barrett B, Brian, Harahan, Brown D, Zhang Z, Brown R. Sufficiently Important Difference for common cold: severity reduction. Ann Fam Med 2007; 5:216-223 Competing interests: None declared |
|||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH |
TRACK: