Abstract
Research often fails to find its way into practice or policy in a timely way, if at all. Given the current pressure and pace of health care change, many authors have recommended different approaches to make health care research more relevant and rapid. An emerging standard for research, the “5 R’s” is a synthesis of recommendations for care delivery research that (1) is relevant to stakeholders; (2) is rapid and recursive in application; (3) redefines rigor; (4) reports on resources required; and (5) is replicable. Relevance flows from substantive ongoing participation by stakeholders. Rapidity and recursiveness occur through accelerated design and peer reviews followed by short learning/implementation cycles through which questions and answers evolve over time. Rigor is the disciplined conduct of shared learning within the specific changing situations in diverse settings. Resource reporting includes costs of interventions. Replicability involves designing for the factors that may affect subsequent implementation of an intervention or program in different contexts. These R’s of the research process are mutually reinforcing and can be supported by training that fosters collaborative and reciprocal relationships among researchers, implementers, and other stakeholders. In sum, a standard is emerging for research that is both rigorous and relevant. Consistent and bold application will increase the value, timeliness, and applicability of the research enterprise.
THE NEED FOR RELEVANT RESEARCH IN A RAPIDLY CHANGING HEALTH CARE WORLD
Accelerated pressure for change in health care creates an exploding need for relevant and rapidly generated new information. A growing volume of care delivery experiments around the country pose questions that research can help answer: Which interventions or system changes improve care, access, safety, or quality—and for which populations, under what conditions? Which system changes reduce underuse, overuse, or misuse? Which approaches are implementable and engaging to clinicians and patients—and can be done at reasonable cost?1,2
Evolving clinical, organizational, and business models for health care, such as patient-centered medical homes3 and accountable care organizations4 need rapidly generated research evidence in real-world experiments for multiple stakeholders: implementers who want to improve their practices; purchasers who want to pay for value; health plans that administer benefits and take risks for care provided; policy makers who are being asked to change “the rules of the game” to support new approaches; patients who wish to know their care is effective, safe, and worth their effort and money; and public health, community groups, and agencies who wish to see improved health at a societal level.5,6
The current research approach is not up to this challenge. Most recent research is slow to influence practice, does so only in pockets, or does not address practical needs for decision making.7–9 Innovative ideas to remedy this situation have been proposed and some implemented. Yet the overall problem remains. It is time to pull together and implement changes in research paradigms and habits to better meet the research needs of changing health care delivery.10–14
The threads of a fresh research paradigm are already apparent, having been suggested separately in many publications,15–17 but need to be woven together to form a picture—the whole cloth with which to tailor research to answer the important stakeholder questions. This article weaves those threads together in the form of an integrated set of “5 R’s” to guide research.
THE 5 R’S OF HEALTH CARE DELIVERY RESEARCH
Below we articulate the issues and how they can be addressed through the 5 R’s to generate health care research that (1) is relevant to stakeholders; (2) is rapid and recursive in application; (3) redefines rigor; (4) reports on resources required; and (5) is replicable. Relevance flows from substantive ongoing participation by stakeholders. Rapidity and recursiveness occur through accelerated design and grant reviews followed by short learning/implementation cycles through which questions and answers evolve over time. Rigor is the disciplined conduct of shared learning within the specific changing situations in diverse settings. Resource reporting includes costs of interventions and likely cost of replication in other settings. Replicability involves designing for the factors that may affect subsequent implementation of an intervention or program in different contexts.
Relevant to Stakeholders
What Is the Issue?
Perceived lack of relevance is cited as the primary reason practitioners do not use research.18–21 Research must generate setting-based evidence designed to flow into practice realities and meet stakeholder needs; as has been stated, “If we want more evidence-based practice, we need more practice-based evidence.”22
How Can It Be Addressed?
Involve end users meaningfully and continuously from the outset in forming research questions and selecting outcomes.2,5,15,16,18,21–24
Research agendas should have origins in need on the ground16—with stakeholders being not only customers for research, but also producers of meaningful questions. Questions come from implementers, policy makers, health plans, purchasers, and patients—with researchers who listen and translate different user concerns into researchable questions.5,24 Such participatory, practice- or community-based “partnership research”15 extends to all phases of research: question generation, designs and measures, implementation, interpretation, presentation, and application of results.
Build an ultimate use perspective into all stages of the research process
If research waits too long before considering sustainable real-world implementation, investment in the preceding research may prematurely “freeze” the intervention in ways not compatible with later use. Engaging stakeholders in how to implement at earlier phases may help avoid retooling and reduce the time to real-world application.
Seek continuation, not only translation
Health care practitioners are arguably more likely to apply and sustain what is learned from research in their own practices.25 The question becomes not, “how do we translate this to our practice?” but, “how do we continue, adapt, and spread what we just learned in our practice?” If done widely, this continuation could make research relevant and make knowledge generation part of the fabric of practice.
Rapid and Recursive in Application
What Is the Issue?
It is not acceptable that it takes 17 years on average for a 14% uptake of funded research into practice.7 In a rapidly changing environment, we need to find ways to accelerate the research enterprise.
How Can It Be Addressed?
Engage stakeholders in rapid-learning research systems
In “rapid-learning health care,” “routinely collected real-time clinical data drive the process of scientific discovery, which becomes a natural outgrowth of patient care.”26 Components may include databases or registries organized by populations, electronic health records, guidelines and clinical decision support, patient engagement, and multiple sponsors or research networks.26,27 In “rapid-learning research systems”21 researchers, funders, implementers, health systems, and community partners are brought together to develop questions, answer them, and then ask new questions of practical importance.
Streamline review processes
The health care delivery world moves on with new partners, questions, and technology, whereas traditional grant application and review often takes a long time.21 Rapid review processes26 that shorten the time from conception of a study to its approval, funding, and start can help keep studies timely and relevant.
Pose research questions to multiple networked practices
Practice-based research networks (PBRNs) are practices that work together to answer health care questions and translate findings into practice. PBRNs can generate relevant questions from stakeholders, design research, and collect data that result in rapid answers from large data sets, including deidentified data from clinical and financial records stored in electronic health records from natural experiments happening in real time, such as data for complex patients treated under real-world conditions by real-world clinicians. Some of these practice settings, as well as public health system research networks,28 are becoming true “learning organizations”27 where quality improvement research is included with, not separated from, more experimental findings.
Allow discoveries within a study to influence the study
Discoveries, sometimes unexpected, can modify subsequent data collection and measurement. Data collection is no longer only at fixed points, using static measures. Implementation or study processes are continuously improved along the way. This is a recursive and rapid learning situation.27 When discoveries in a study begin to appear, they may reshape stakeholder questions or begin to answer others. The next set of questions may begin to emerge, along with energy for answering those reshaped or more insightful questions. As discoveries roll in, stakeholders, in partnership with researchers, guide these iterations.
Redefines Rigor
What Is the Issue?
Scientific rigor is essential, but common conceptions of rigor may limit the range of real-world situations that can be studied—and methods, settings, and populations with which to do so. The hallmark of rigor has been the “gold standard” efficacy randomized controlled trial (RCT) emphasizing internal validity.
How Can It Be Addressed?
We suggest a modified version of rigor suited to broader questions meaningful to multiple stakeholders and answered in heterogeneous populations and settings with attention to transportability and sustainable implementation.
Regard rigor as a property of a series of decisions, observations, and relationships rather than of techniques.29
Rigorous research (1) is systematic and organized about concepts, tools, data collection, measures, procedures, and analyses; (2) checks for superfluous connections and confounding variables; (3) has controls and conclusions justified by standards of evidence; and (4) uses transparent descriptions of what was done. Rigor is not defined as a list of certain techniques and exclusion of others. Particular experimental designs, data collection, or analysis techniques are not always considered more rigorous than others or that any one is the optimal design for all questions and all situations. Others have commented, “If techniques are tools in a researcher’s toolbox, then this is like saying that ‘A saw is better than a hammer because it is sharper.’”29 Stated as a principle, “Research agenda determines the research methods rather than methods determine the research agenda.”16
Reporting on Resources
What Is the Issue?
Use of health care resources is a major concern when a priority is to bend the cost curve. Stakeholders are making decisions among alternative care approaches based on the cost of interventions as well as on clinical effectiveness. Information on resources used to conduct or replicate interventions can be helpful in larger economic analysis, but is seldom well reported.31
How Can It Be Addressed?
Use a consistent vocabulary for reporting
For example, measuring cost includes money, but also clinician and staff time and energy, plus intervention systems, infrastructure, or training costs. There are start-up costs, ongoing costs, and opportunity costs. There are costs of doing a study intervention, and likely costs of recreating it in another setting. Costs incurred in one place or to one stakeholder may save costs in some other place or for another stakeholder.
Report on as many of the relevant costs of different interventions as possible and do so in a standard manner.14,32,33
For example, what did the program in question cost to promote and implement, and what are estimates of what it would cost to replicate a similar program under different conditions or settings? Such estimates do not require researchers to do complex economic or cost-benefit analyses, but reporting on resources is important to those who pay attention to value in health care. Although value may mean many things to many people, it is being defined and becoming part of efforts to be rigorous, transparent, and relevant to stakeholders,34 including patients who have their own perspective on value.35
Replicability
What Is the Issue?
Research design can help address questions regarding how an innovation will perform in a new system with different contextual influences.36 Replication of findings is increasingly recognized37 as a major challenge across the translation spectrum from basic discovery (eg, genomewide association studies) to community interventions. The conditions under which a finding can be replicated are central to understanding robust effects that can be reasonably expected under various conditions. Hence, researchers must design for replicability and report results needed for reproducibility—either under the same or different conditions in which the findings are likely to be applied.
How Can It Be Addressed?
Design for sustainable implementation from the start
This approach is arguably more efficient and effective if it saves rework for real-world application. We recommend asking 3 questions: (1) is the study designed to inform implementation—and re-invention in different settings? (2) is the “how” reported as well as the “what” of interventions, and to what extent are procedures replicable in similar or diverse settings? and (3) are contextual factors reported that are important to understanding what happened and why—for example, relevant policies, and inclusion and exclusion criteria for settings and staff as well as patients?
These strategies can go a long way toward making findings replicable, realizing that not every study can be transported to other settings. But such data will allow others to make reasonable judgments about what aspects to retain and what to change for replication or reinvention in a different time and place, using relevant domains for contextual factors.38
EXAMPLES
Some of the 5 R’s have found their way into research studies, networks, and tools. Table 1 describes a few examples. Readers can likely cite other examples or see other R’s in these. The 5 R’s are emerging not as a response to a completed record of implementation in full scope, but as a logical (and promising) challenge to package in application the separate elements already abundantly demonstrated in the literature, but not yet combined to full effect in more than a few examples. What is “bold” is the proposal to routinely apply the 5 R’s as a package and to take on the substantial challenges of practical implementation and evolution. For inspiration, consider McDonnell Douglas’ bold integration of 5 existing technologies for the first time in 1 airplane (the 1935 DC-3), the innovation that swept away the competition and opened the era of commercial air travel, when a plane with 4 of those technologies failed commercially the previous year.51
CONCLUSIONS
Health care transformation needs the full benefit of research to inform decision making and discover new options. The research community owes it to its “customers” and the public to evolve its standards and methods for health care research. The 5 R’s are offered as a next step in the developmental trajectory of an evolving field—a framework for a much needed discussion and adjustment of criteria for what is considered high-quality research.
Routinely Apply the Complete Package of 5 R’s
As is the case for other models—for example, the Reach Effectiveness–Adoption Implementation Maintenance (RE-AIM) evaluation model,52,53 the Chronic Care Model,54 and the Institute of Medicine 6 quality aims55—the effect of the 5 R’s model comes not from doing separate R’s or even 2 or 3 of them, but from doing them all in an integrated fashion whereby each reinforces the others. The 5 R’s are proposed to work together across stages of the research and dissemination process. Table 2 shows a research “preflight” checklist.
Overcome the Practical Challenges of Implementing the R’s
There is little doubt that implementing the 5 R’s on a meaningful scale will require continued changes in thinking and infrastructure pointed out in literature on the separate R’s. Table 3 summarizes such changes.
Although many of these changes are under way in different places in different ways, considerable challenges remain. We believe that emerging stakeholder interests align well with the 5 R’s and will drive such change. For example, the 2014 Academy Health report on improving the evidence base for Medicare policy making56 interviewed leaders in health policy and care delivery for their most pressing health services research needs over the next 3 to 5 years; it was research that (1) aims at understanding the performance of new organizational forms such as accountable care organizations and Medicare Advantage plans; (2) uses comparable data sets for performance of physician practices and new organizational arrangements; (3) engages with the promises and pitfalls of electronic data, rapid cycle research, and comparative effectiveness research; (4) understands how the politics of evidence and policy affect research relevance and usefulness; and (5) builds relationships between researchers and policy makers, with study findings available at the time decisions were made—even if “best available” rather than “best” evidence. Although this study was focused on Medicare, we believe its lessons can be much more broadly taken.
In addition, we solicited feedback from a convenience sample of 8 stakeholders on the importance of research for practical decision making and on the 5 R’s. Participants were balanced across practitioners, other implementers, and administrators. Responses indicated that relevant was the most important R, with rapid a close second, followed by other R’s—none of which were considered unimportant. The most important role for research in practical decisions was testing viability of approaches in their own settings and available resources. Suggested reasons why research is often not useful were lack of relevance, rapidity, or good relationships with researchers. The 2 facets identified as making research more helpful were “faster turnaround” and building better relationships between researchers and clinicians; as one clinician put it, “Relationship is so important, you should put a 6th R in there!”
Recognize the Wide Range of Application for the 5 R’s
Admittedly aspirational, we do not expect every study to comprehensively address all 5 R’s. We do not expect, for example, all epidemiologic research or basic mechanism studies to address all of them. Studies having as their long-term goal achieving translation to real-world settings or making a population impact, however, would benefit from considering each R, reporting on those most relevant, and discussing implications for the others. Examining the implications of all 5 R’s should be useful in the vast majority of research studies, from efficacy to effectiveness to implementation and dissemination, not just for a few community translational “T4” studies, which are investigations of practice intervention effects on population health. This strategy would help align the pipeline of potential interventions with real-world pragmatic requirements.
Build Better Relationships Using the 5 R’s
Practitioners often experience research as interfering with practical procedures or believe that researchers just want study participants to address their own questions and further their careers. On the other hand, researchers often experience clinicians as not interested in research, resistant to research protocols, or not being ready to implement evidence-based findings. This is not a perceived relationship between researchers and practitioners or other stakeholders that will carry us into a successful future. The 5 R’s proposed embody the terms of a new and more transparent win-win partnership between researchers and stakeholders with important questions that research can help answer.
Teach to This New Standard
This new standard (and its implicit partnership between stakeholders and researchers) is especially important for students and those early in their careers, whether clinicians, researchers, policy makers, or others wishing to develop or use research evidence. The 5 R’s are offered as teaching tools as well as research tools—helping all stakeholders wear constructive “hats” with each other when addressing important questions. Over time, this approach may lead to an improved relationship between the research and health care enterprises—on behalf of the public they both serve.
Footnotes
-
Conflicts of interest: authors report none.
-
Funding support: This project has been funded in whole or in part with federal funds from the National Cancer Institute, National Institutes of Health, under contract No. HHSN261200800001E. Dr Stange’s time is supported in part by a Clinical Research Professorship from the American Cancer Society.
-
Disclaimer: The content of this article does not necessarily reflect the views or policies of the National Cancer Institute or the Department of Health and Human Services, nor does mention of trade names, commercial products, or organizations imply endorsement by the US government.
- Received for publication July 17, 2013.
- Revision received May 13, 2014.
- Accepted for publication June 2, 2014.
- © 2014 Annals of Family Medicine, Inc.