Study Designs for Effectiveness and Translation Research

Background

Practitioners and policymakers need credible evidence of effectiveness to justify allocating resources to complex, expensive health programs. Investigators, however, face challenges in designing sound effectiveness and translation research with relevance for “real-world” settings.

Methods

Research experts and federal and foundation funders (n=∼120) prepared for and participated in a symposium, held May 4–5, 2004, to weigh the strengths, limitations, and trade-offs between alternate designs for studying the effectiveness and translation of complex, multilevel health interventions.

Results

Symposium attendees acknowledged that research phases (hypothesis generating, efficacy, effectiveness, translation) are iterative and cyclical, not linear, since research in advanced phases may reveal unanswered questions in earlier phases. Research questions thus always need to drive the choice of study design. When randomization and experimental control are feasible, participants noted that the randomized controlled trial with individual random assignment remains the gold standard for safeguarding internal validity. Attendees highlighted trade-offs of randomized controlled trial variants, quasi-experimental designs, and natural experiments for use when randomization or experimental control or both are impossible or inadequately address external validity. Participants discussed enhancements to all designs to increase confidence in causal inference while accommodating greater external validity. Since no single study can establish causality, participants encouraged replication of studies and triangulation using different study designs. Participants also recommended participatory research approaches for building population relevance, acceptability, and usefulness.

Conclusions

Consideration of the study design choices, trade-offs, and enhancements discussed here can guide the design, funding, completion, and publication of appropriate policy- and practice-oriented effectiveness and translational research for complex, multilevel health interventions.

Introduction

The demand for public health and healthcare practice and policy to be based on scientific evidence continues to grow, affecting programs, services, and research.1, 2, 3, 4, 5, 6 Attempts to describe characteristics of valid, high-quality research and evaluation that policymakers and practitioners should value are multiplying.7, 8, 9, 10, 11, 12, 13, 14 The United States Preventive Services Task Force,¹⁵ Task Force on Community Preventive Services,16, 17 Cochrane Collaboration,¹⁸ Campbell Collaboration,¹⁹ and the United Kingdom’s National Institute for Health and Clinical Excellence (NICE)²⁰ put a premium on rigorous design as they assess the literature and develop recommendations for practice and policy. The U.S. Institute of Medicine and the National Quality Forum have recommended focusing on the scientific soundness of quality improvement activities.13, 21 U.S. federal agencies are subjecting applications for intervention research funding to more rigorous external peer review than in previous years, and the U.S. Department of Education is placing higher priority on evaluation projects that use rigorous research methods to assess intervention effectiveness.²²

Many quality-of-evidence ratings emphasize internal validity (Does this intervention work under the conditions set forth in the study?) without also giving consideration to external validity (Will it work in other settings and contexts?).²³ Assessing internal validity is of paramount importance, for without it, one cannot be sure whether the intervention works at all. Yet growing recognition of gaps between research and practice has led researchers, policymakers, practitioners, and government officials to call for more research that has relevance for practice and policy across a wide range of real-world settings and situations.17, 23, 24, 25, 26, 27, 28, 29, 30 Some practitioners and policymakers question the effectiveness for their particular situations of interventions deemed efficacious in studies using populations or circumstances different from their own. Others wonder whether interventions whose effectiveness has been established within some practices or communities can be generalized or transferred to a broad range of settings. Studies that consider external as well as internal validity are important for informing real-world decision making in such situations.5, 23, 27, 31

While efficacy research assesses whether an intervention works under ideal conditions, effectiveness research examines whether it works under real-world conditions.³² Translation research, while defined differently within and across disciplines, involves exploring how to translate (or transfer) scientific discoveries into practical applications to improve health.25, 27, 30

Challenges arise when researchers attempt to design effectiveness and translation research to evaluate complex, multilevel health interventions in real-world settings. It may be challenging, for example, to devise an appropriate control group in studies assessing multicomponent interventions when enough is known about the individual components to raise ethical objections if any of them were to be withheld, and when receiving nothing or a placebo would be unacceptable. Or it may be difficult to determine how to evaluate long-term follow-up when significant subject attrition is expected over time.

Other design challenges may arise when the intervention can take many forms or requires program- or population-specific adaptations, when investigators cannot control how the intervention is implemented by different practitioners, when the quality with which the intervention is delivered can vary widely, or when the individuals who would volunteer or agree to participate are different from the target population as a whole. Financial or logistic complications may occur when trying to secure adequate sample size for studies in which large organizational units, entire communities, or nations are the unit of analysis. Finally, the intervention of interest may not be able to be randomly assigned to individuals or groups because they will not agree to be randomized or because all potential participants are exposed to the intervention (e.g., a law).

These design challenges suggest that it might be worthwhile to consider what valuable information can be gained from employing a variety of study designs.³³ One can also ask whether the weight of evidence from nonrandomized study designs can offset the strength of evidence lost when randomization is impossible or inappropriate. The purpose of this project was therefore to explore the strengths, limitations, and trade-offs among a variety of designs applicable to effectiveness and translation research. Of particular interest were designs that would provide evidence not only for whether these interventions would work in the setting in which they were first studied, but also whether they could produce findings generalizable to other settings and contexts. The intent was to identify directions that could be taken to strengthen the evidence base for real-world decision making.