Original Article
A most stubborn bias: no adjustment method fully resolves confounding by indication in observational studies

https://doi.org/10.1016/j.jclinepi.2009.03.001Get rights and content

Abstract

Objective

To evaluate the effectiveness of methods that control for confounding by indication, we compared breast cancer recurrence rates among women receiving adjuvant chemotherapy with those who did not.

Study Design and Setting

In a medical record review-based study of breast cancer treatment in older women (n = 1798) diagnosed between 1990 and 1994, our crude analysis suggested that adjuvant chemotherapy was positively associated with recurrence (hazard ratio [HR] = 2.6; 95% confidence interval [CI] = 1.9, 3.5). We expected a protective effect, so postulated that the crude association was confounded by indications for chemotherapy. We attempted to adjust for this confounding by restriction, multivariable regression, propensity scores (PSs), and instrumental variable (IV) methods.

Results

After restricting to women at high risk for recurrence (n = 946), chemotherapy was not associated with recurrence (HR = 1.1; 95% CI = 0.7, 1.6) using multivariable regression. PS adjustment yielded similar results (HR = 1.3; 95% CI = 0.8, 2.0). The IV-like method yielded a protective estimate (HR = 0.9; 95% CI = 0.2, 4.3); however, imbalances of measured factors across levels of the IV suggested residual confounding.

Conclusion

Conventional methods do not control for unmeasured factors, which often remain important when addressing confounding by indication. PS and IV analysis methods can be useful under specific situations, but neither method adequately controlled confounding by indication in this study.

Introduction

Confounding by indication remains an often intractable threat to validity in observational studies [1]. Although confounding is best controlled by a randomized design, randomization is not always feasible. For example, patients cannot be randomized to receive placebo when an efficacious therapy is available [2]. Furthermore, trials often exclude patients with preexisting conditions [3], particularly older adults [4]. Nonrandomized designs must evaluate the effectiveness of therapies whose efficacy has been established in select groups by clinical trials, but not in broader populations that might react differently to the therapy. For these and other reasons [3], nonrandomized studies of therapy effectiveness will remain important [1]. In addition, generalizing results from clinical trials with select patient populations may actually cause harm in the heterogeneous populations treated in clinical practice [5].

As an example, clinical trials of adjuvant chemotherapy in women aged 40–59 years with early-stage breast cancer demonstrate its efficacy with reductions in 5-year mortality between 20% and 40% [6], but it is uncertain whether these benefits extend to older women, who bear the majority burden of breast cancer occurrence [7]. Nonrandomized studies of older women with early-stage breast cancer suffered from differences in prognosis between women who received adjuvant chemotherapy and women who did not receive adjuvant chemotherapy [8], [9], and thus are potentially biased by confounding by indication.

When the validity of a study is threatened by confounding by indication, it is not straightforward to determine which method of adjustment, if any, is most effective in obtaining a valid and precise estimate of effect. Conventional methods to adjust for confounding, such as restriction and multivariable regression, leave residual confounding because of unmeasured factors. Thus, propensity score (PS) adjustment and the instrumental variable (IV) approach have become increasingly popular [10], [11], [12], [13], with the intent to address this residual confounding by simulating a randomized environment. PS adjustment theoretically increases comparability between the comparison groups by creating pseudorandomization of measured confounders [14]. The goal of the IV approach is to reduce confounding by indication through the use of a variable that is associated with the exposure, unrelated to the confounders, and has no direct association with the outcome other than through the exposure [15]. However, several investigators have cautioned that these alternative methods are not universal solutions to the problem of confounding by indication [10], [11], [13], [16], [17], [18].

Three observational studies used the SEER-Medicare [19] linked data set and found that adjuvant chemotherapy decreased the rate of breast cancer-specific mortality [20] and all-cause mortality [20], [21], [22] in older women, with the greatest benefit seen in women with node-positive, estrogen receptor negative tumors [20], [21]. Based on these results and those of clinical trials among middle-aged women [6], we expect adjuvant chemotherapy to be protective against breast cancer recurrence in older women. With this prior information in mind, we compared methods used to reduce confounding. We implemented restriction, multivariable regression, PS adjustment, and an IV-like method to estimate incidence rates of breast cancer recurrence in women who received adjuvant chemotherapy compared with women who did not, in the Breast Cancer Treatment Effectiveness in Older Women (BOW) cohort [8], [9], [23].

Section snippets

Study population

The BOW cohort study was conducted at six integrated health care systems that are part of the 14-system consortium of the Cancer Research Network (CRN) [24]. The overall goal of the CRN is to increase the effectiveness of preventive, curative, and supportive interventions for major cancers through a program of collaborative research, and to determine the effectiveness of cancer control interventions that span the natural history of major cancers among diverse populations and health systems. The

Results

Frequencies for demographic and tumor characteristics for the unrestricted cohort who received primary therapy (n = 1798), the cohort restricted to women at high risk for recurrence (n = 946), the PS analytic sample (n = 723), and the IV analytic sample (n = 539) are presented in Table 1 by receipt of chemotherapy. For women classified as at high risk, 20% experienced a breast cancer recurrence. In the unrestricted, restricted, PS, and IV samples, a higher proportion of women who received adjuvant

Discussion

The association between receipt of adjuvant chemotherapy and recurrence risk in older women with breast cancer provides a useful example of the manner in which confounding by indication can complicate nonrandomized studies of treatments in general populations. When considering treatment recommendations to reduce breast cancer recurrence, oncologists treating geriatric patients take into account tumor prognostic factors and additional factors, such as life expectancy, physical function, and

Acknowledgment

Source of financial support: Supported by Public Health Service Grant R01 CA093772 (Breast Cancer Treatment Effectiveness in Older Women, Rebecca A. Silliman, PI) from the National Cancer Institute, National Institutes of Health, Department of Health and Human Services.

The authors thank Dr. Alan Brookhart for his thoughtful review and guidance regarding the instrumental variable analysis. We also thank Dr. Terry Field, site-principal investigator at Fallon Clinic/Meyers Primary Care Institute,

References (39)

  • M. Ulcickas Yood et al.

    Mortality impact of less-than-standard therapy in older breast cancer patients

    J Am Coll Surg

    (2008)
  • M. Charlson et al.

    A new method of classifying prognostic comorbidity in longitudinal studies: development and validation

    J Chronic Dis

    (1987)
  • S.H. Giordano et al.

    Limits of observational data in determining outcomes from cancer therapy

    Cancer

    (2008)
  • K.J. Rothman et al.

    The continuing unethical use of placebo controls

    N Engl J Med

    (1994)
  • H.T. Sørensen et al.

    Beyond randomized controlled trials: A critical comparison of trials with nonrandomized studies

    Hepatology

    (2006)
  • J. Avorn

    In defense of pharmacoepidemiology - embracing the yin and yang of drug research

    N Engl J Med

    (2007)
  • C.P. Gross et al.

    Relation between prepublication release of clinical trial results and the practice of carotid endarterectomy

    JAMA

    (2000)
  • Effects of chemotherapy and hormonal therapy for early breast cancer on recurrence and 15-year survival: an overview of the randomised trials

    Lancet

    (2005)
  • T.L. Lash et al.

    Re: prevalence of cancer

    J Natl Cancer Inst

    (1998)
  • A.M. Geiger et al.

    Recurrences and second primary breast cancers in older women with initial early-stage disease

    Cancer

    (2007)
  • T. Sturmer et al.

    A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods

    J Clin Epidemiol

    (2006)
  • R.J. Glynn et al.

    Indications for propensity scores and review of their use in pharmacoepidemiology

    Basic Clin Pharmacol Toxicol

    (2006)
  • M.S. Cepeda et al.

    Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders

    Am J Epidemiol

    (2003)
  • M.A. Brookhart et al.

    Preference-based instrumental variable methods for the estimation of treatment effects: assessing validity and interpreting results

    Int J Biostat

    (2007)
  • P.R. Rosenbaum et al.

    The central role of the propensity score in observational studies for causal effects

    Biometrika

    (1983)
  • S. Greenland

    An introduction to instrumental variables for epidemiologists

    Int J Epidemiol

    (2000)
  • J.V. Terza et al.

    The use of linear instrumental variables methods in health services research and health economics: a cautionary note

    Health Serv Res

    (2008)
  • M. Hernan et al.

    Instruments for causal inference

    Epidemiology

    (2006)
  • E. Martens et al.

    Instrumental variables: application and limitations

    Epidemiology

    (2006)
  • Cited by (298)

    View all citing articles on Scopus
    View full text