The 5 R’s: An Emerging Bold Standard for Conducting Relevant Research in a Changing World
===========================================================================================

* C. J. Peek
* Russell E. Glasgow
* Kurt C. Stange
* Lisa M. Klesges
* E. Peyton Purcell
* Rodger S. Kessler

## Abstract

Research often fails to find its way into practice or policy in a timely way, if at all. Given the current pressure and pace of health care change, many authors have recommended different approaches to make health care research more relevant and rapid. An emerging standard for research, the “5 R’s” is a synthesis of recommendations for care delivery research that (1) is relevant to stakeholders; (2) is rapid and recursive in application; (3) redefines rigor; (4) reports on resources required; and (5) is replicable. *Relevance* flows from substantive ongoing participation by stakeholders. *Rapidity and recursiveness* occur through accelerated design and peer reviews followed by short learning/implementation cycles through which questions and answers evolve over time. *Rigor* is the disciplined conduct of shared learning within the specific changing situations in diverse settings. *Resource reporting* includes costs of interventions. *Replicability* involves designing for the factors that may affect subsequent implementation of an intervention or program in different contexts. These R’s of the research process are mutually reinforcing and can be supported by training that fosters collaborative and reciprocal relationships among researchers, implementers, and other stakeholders. In sum, a standard is emerging for research that is both rigorous and relevant. Consistent and bold application will increase the value, timeliness, and applicability of the research enterprise.

*   research design
*   healthcare research
*   evaluation research
*   partnership research
*   rapid research

## THE NEED FOR RELEVANT RESEARCH IN A RAPIDLY CHANGING HEALTH CARE WORLD

Accelerated pressure for change in health care creates an exploding need for relevant and rapidly generated new information. A growing volume of care delivery experiments around the country pose questions that research can help answer: Which interventions or system changes improve care, access, safety, or quality—and for which populations, under what conditions? Which system changes reduce underuse, overuse, or misuse? Which approaches are implementable and engaging to clinicians and patients—and can be done at reasonable cost?1,2

Evolving clinical, organizational, and business models for health care, such as patient-centered medical homes3 and accountable care organizations4 need rapidly generated research evidence in real-world experiments for multiple stakeholders: implementers who want to improve their practices; purchasers who want to pay for value; health plans that administer benefits and take risks for care provided; policy makers who are being asked to change “the rules of the game” to support new approaches; patients who wish to know their care is effective, safe, and worth their effort and money; and public health, community groups, and agencies who wish to see improved health at a societal level.5,6

The current research approach is not up to this challenge. Most recent research is slow to influence practice, does so only in pockets, or does not address practical needs for decision making.7–9 Innovative ideas to remedy this situation have been proposed and some implemented. Yet the overall problem remains. It is time to pull together and implement changes in research paradigms and habits to better meet the research needs of changing health care delivery.10–14

The threads of a fresh research paradigm are already apparent, having been suggested separately in many publications,15–17 but need to be woven together to form a picture—the whole cloth with which to tailor research to answer the important stakeholder questions. This article weaves those threads together in the form of an integrated set of “5 R’s” to guide research.

## THE 5 R’S OF HEALTH CARE DELIVERY RESEARCH

Below we articulate the issues and how they can be addressed through the 5 R’s to generate health care research that (1) is relevant to stakeholders; (2) is rapid and recursive in application; (3) redefines rigor; (4) reports on resources required; and (5) is replicable. *Relevance* flows from substantive ongoing participation by stakeholders. *Rapidity and recursiveness* occur through accelerated design and grant reviews followed by short learning/implementation cycles through which questions and answers evolve over time. *Rigor* is the disciplined conduct of shared learning within the specific changing situations in diverse settings. *Resource reporting* includes costs of interventions and likely cost of replication in other settings. *Replicability* involves designing for the factors that may affect subsequent implementation of an intervention or program in different contexts.

### Relevant to Stakeholders

#### What Is the Issue?

Perceived lack of relevance is cited as the primary reason practitioners do not use research.18–21 Research must generate setting-based evidence designed to flow into practice realities and meet stakeholder needs; as has been stated, “If we want more evidence-based practice, we need more practice-based evidence.”22

### How Can It Be Addressed?

#### Involve end users meaningfully and continuously from the outset in forming research questions and selecting outcomes.2,5,15,16,18,21–24

Research agendas should have origins in need on the ground16—with stakeholders being not only customers for research, but also producers of meaningful questions. Questions come from implementers, policy makers, health plans, purchasers, and patients—with researchers who listen and translate different user concerns into researchable questions.5,24 Such participatory, practice- or community-based “partnership research”15 extends to all phases of research: question generation, designs and measures, implementation, interpretation, presentation, and application of results.

#### Build an ultimate use perspective into all stages of the research process

If research waits too long before considering sustainable real-world implementation, investment in the preceding research may prematurely “freeze” the intervention in ways not compatible with later use. Engaging stakeholders in how to implement at earlier phases may help avoid retooling and reduce the time to real-world application.

#### Seek continuation, not only translation

Health care practitioners are arguably more likely to apply and sustain what is learned from research in their own practices.25 The question becomes not, “how do we translate this to our practice?” but, “how do we continue, adapt, and spread what we just learned in our practice?” If done widely, this continuation could make research relevant and make knowledge generation part of the fabric of practice.

### Rapid and Recursive in Application

#### What Is the Issue?

It is not acceptable that it takes 17 years on average for a 14% uptake of funded research into practice.7 In a rapidly changing environment, we need to find ways to accelerate the research enterprise.

#### How Can It Be Addressed?

##### Engage stakeholders in rapid-learning research systems

In “rapid-learning health care,” “routinely collected real-time clinical data drive the process of scientific discovery, which becomes a natural outgrowth of patient care.”26 Components may include databases or registries organized by populations, electronic health records, guidelines and clinical decision support, patient engagement, and multiple sponsors or research networks.26,27 In “rapid-learning research systems”21 researchers, funders, implementers, health systems, and community partners are brought together to develop questions, answer them, and then ask new questions of practical importance.

##### Streamline review processes

The health care delivery world moves on with new partners, questions, and technology, whereas traditional grant application and review often takes a long time.21 Rapid review processes26 that shorten the time from conception of a study to its approval, funding, and start can help keep studies timely and relevant.

##### Pose research questions to multiple networked practices

Practice-based research networks (PBRNs) are practices that work together to answer health care questions and translate findings into practice. PBRNs can generate relevant questions from stakeholders, design research, and collect data that result in rapid answers from large data sets, including deidentified data from clinical and financial records stored in electronic health records from natural experiments happening in real time, such as data for complex patients treated under real-world conditions by real-world clinicians. Some of these practice settings, as well as public health system research networks,28 are becoming true “learning organizations”27 where quality improvement research is included with, not separated from, more experimental findings.

##### Allow discoveries within a study to influence the study

Discoveries, sometimes unexpected, can modify subsequent data collection and measurement. Data collection is no longer only at fixed points, using static measures. Implementation or study processes are continuously improved along the way. This is a recursive and rapid learning situation.27 When discoveries in a study begin to appear, they may reshape stakeholder questions or begin to answer others. The next set of questions may begin to emerge, along with energy for answering those reshaped or more insightful questions. As discoveries roll in, stakeholders, in partnership with researchers, guide these iterations.

### Redefines Rigor

#### What Is the Issue?

Scientific rigor is essential, but common conceptions of rigor may limit the range of real-world situations that can be studied—and methods, settings, and populations with which to do so. The hallmark of rigor has been the “gold standard” efficacy randomized controlled trial (RCT) emphasizing internal validity.

#### How Can It Be Addressed?

We suggest a modified version of rigor suited to broader questions meaningful to multiple stakeholders and answered in heterogeneous populations and settings with attention to transportability and sustainable implementation.

##### Regard rigor as a property of a series of decisions, observations, and relationships rather than of techniques.29

Rigorous research (1) is systematic and organized about concepts, tools, data collection, measures, procedures, and analyses; (2) checks for superfluous connections and confounding variables; (3) has controls and conclusions justified by standards of evidence; and (4) uses transparent descriptions of what was done. Rigor is not defined as a list of certain techniques and exclusion of others. Particular experimental designs, data collection, or analysis techniques are not always considered more rigorous than others or that any one is the optimal design for all questions and all situations. Others have commented, “If techniques are tools in a researcher’s toolbox, then this is like saying that ‘A saw is better than a hammer because it is sharper.’”29 Stated as a principle, “Research agenda determines the research methods rather than methods determine the research agenda.”16

##### Give attention to both external and internal validity

Most methods developed to assess research quality focus predominantly or exclusively on internal validity. Rigor also implies attention to transparent reporting on issues related to generalizability.16,30

### Reporting on Resources

#### What Is the Issue?

Use of health care resources is a major concern when a priority is to bend the cost curve. Stakeholders are making decisions among alternative care approaches based on the cost of interventions as well as on clinical effectiveness. Information on resources used to conduct or replicate interventions can be helpful in larger economic analysis, but is seldom well reported.31

#### How Can It Be Addressed?

##### Use a consistent vocabulary for reporting

For example, measuring cost includes money, but also clinician and staff time and energy, plus intervention systems, infrastructure, or training costs. There are start-up costs, ongoing costs, and opportunity costs. There are costs of doing a study intervention, and likely costs of recreating it in another setting. Costs incurred in one place or to one stakeholder may save costs in some other place or for another stakeholder.

##### Report on as many of the relevant costs of different interventions as possible and do so in a standard manner.14,32,33

For example, what did the program in question cost to promote and implement, and what are estimates of what it would cost to replicate a similar program under different conditions or settings? Such estimates do not require researchers to do complex economic or cost-benefit analyses, but reporting on resources is important to those who pay attention to *value* in health care. Although value may mean many things to many people, it is being defined and becoming part of efforts to be rigorous, transparent, and relevant to stakeholders,34 including patients who have their own perspective on value.35

### Replicability

#### What Is the Issue?

Research design can help address questions regarding how an innovation will perform in a new system with different contextual influences.36 Replication of findings is increasingly recognized37 as a major challenge across the translation spectrum from basic discovery (eg, genomewide association studies) to community interventions. The conditions under which a finding can be replicated are central to understanding robust effects that can be reasonably expected under various conditions. Hence, researchers must design for replicability and report results needed for reproducibility—either under the same or different conditions in which the findings are likely to be applied.

#### How Can It Be Addressed?

##### Design for sustainable implementation from the start

This approach is arguably more efficient and effective if it saves rework for real-world application. We recommend asking 3 questions: (1) is the study designed to inform implementation—and re-invention in different settings? (2) is the “how” reported as well as the “what” of interventions, and to what extent are procedures replicable in similar or diverse settings? and (3) are contextual factors reported that are important to understanding what happened and why—for example, relevant policies, and inclusion and exclusion criteria for settings and staff as well as patients?

These strategies can go a long way toward making findings replicable, realizing that not every study can be transported to other settings. But such data will allow others to make reasonable judgments about what aspects to retain and what to change for replication or reinvention in a different time and place, using relevant domains for contextual factors.38

## EXAMPLES

Some of the 5 R’s have found their way into research studies, networks, and tools. Table 1 describes a few examples. Readers can likely cite other examples or see other R’s in these. The 5 R’s are emerging not as a response to a *completed* record of implementation in full scope, but as a logical (and promising) *challenge* to package in application the separate elements already abundantly demonstrated in the literature, but not yet combined to full effect in more than a few examples. What is “bold” is the proposal to routinely apply the 5 R’s as a package and to take on the substantial challenges of practical implementation and evolution. For inspiration, consider McDonnell Douglas’ bold integration of 5 existing technologies for the first time in 1 airplane (the 1935 DC-3), the innovation that swept away the competition and opened the era of commercial air travel, when a plane with 4 of those technologies failed commercially the previous year.51

View this table:
[Table 1](http://www.annfammed.org/content/12/5/447/T1)

Table 1 
Examples of Projects Illustrating the R’s

## CONCLUSIONS

Health care transformation needs the full benefit of research to inform decision making and discover new options. The research community owes it to its “customers” and the public to evolve its standards and methods for health care research. The 5 R’s are offered as a next step in the developmental trajectory of an evolving field—a framework for a much needed discussion and adjustment of criteria for what is considered high-quality research.

### Routinely Apply the Complete Package of 5 R’s

As is the case for other models—for example, the Reach Effectiveness–Adoption Implementation Maintenance (RE-AIM) evaluation model,52,53 the Chronic Care Model,54 and the Institute of Medicine 6 quality aims55—the effect of the 5 R’s model comes not from doing separate R’s or even 2 or 3 of them, but from doing them all in an integrated fashion whereby each reinforces the others. The 5 R’s are proposed to work together across stages of the research and dissemination process. Table 2 shows a research “preflight” checklist.

View this table:
[Table 2](http://www.annfammed.org/content/12/5/447/T2)

Table 2 
Questions to Apply the 5 R’s at Each Stage of the Research and Dissemination Process

### Overcome the Practical Challenges of Implementing the R’s

There is little doubt that implementing the 5 R’s on a meaningful scale will require continued changes in thinking and infrastructure pointed out in literature on the separate R’s. Table 3 summarizes such changes.

View this table:
[Table 3](http://www.annfammed.org/content/12/5/447/T3)

Table 3 
Challenges and Changes for Routine Implementation of 5 R’s

Although many of these changes are under way in different places in different ways, considerable challenges remain. We believe that emerging stakeholder interests align well with the 5 R’s and will drive such change. For example, the 2014 Academy Health report on improving the evidence base for Medicare policy making56 interviewed leaders in health policy and care delivery for their most pressing health services research needs over the next 3 to 5 years; it was research that (1) aims at understanding the performance of new organizational forms such as accountable care organizations and Medicare Advantage plans; (2) uses comparable data sets for performance of physician practices and new organizational arrangements; (3) engages with the promises and pitfalls of electronic data, rapid cycle research, and comparative effectiveness research; (4) understands how the politics of evidence and policy affect research relevance and usefulness; and (5) builds relationships between researchers and policy makers, with study findings available at the time decisions were made—even if “best available” rather than “best” evidence. Although this study was focused on Medicare, we believe its lessons can be much more broadly taken.

In addition, we solicited feedback from a convenience sample of 8 stakeholders on the importance of research for practical decision making and on the 5 R’s. Participants were balanced across practitioners, other implementers, and administrators. Responses indicated that *relevant* was the most important R, with *rapid* a close second, followed by other R’s—none of which were considered unimportant. The most important role for research in practical decisions was testing viability of approaches in their own settings and available resources. Suggested reasons why research is often not useful were lack of relevance, rapidity, or good relationships with researchers. The 2 facets identified as making research more helpful were “faster turnaround” and building better relationships between researchers and clinicians; as one clinician put it, “Relationship is so important, you should put a 6th R in there!”

### Recognize the Wide Range of Application for the 5 R’s

Admittedly aspirational, we do not expect every study to comprehensively address all 5 R’s. We do not expect, for example, all epidemiologic research or basic mechanism studies to address all of them. Studies having as their long-term goal achieving translation to real-world settings or making a population impact, however, would benefit from considering each R, reporting on those most relevant, and discussing implications for the others. Examining the implications of all 5 R’s should be useful in the vast majority of research studies, from efficacy to effectiveness to implementation and dissemination, not just for a few community translational “T4” studies, which are investigations of practice intervention effects on population health. This strategy would help align the pipeline of potential interventions with real-world pragmatic requirements.

### Build Better Relationships Using the 5 R’s

Practitioners often experience research as interfering with practical procedures or believe that researchers just want study participants to address their own questions and further their careers. On the other hand, researchers often experience clinicians as not interested in research, resistant to research protocols, or not being ready to implement evidence-based findings. This is not a perceived relationship between researchers and practitioners or other stakeholders that will carry us into a successful future. The 5 R’s proposed embody the terms of a new and more transparent win-win partnership between researchers and stakeholders with important questions that research can help answer.

### Teach to This New Standard

This new standard (and its implicit partnership between stakeholders and researchers) is especially important for students and those early in their careers, whether clinicians, researchers, policy makers, or others wishing to develop or use research evidence. The 5 R’s are offered as teaching tools as well as research tools—helping all stakeholders wear constructive “hats” with each other when addressing important questions. Over time, this approach may lead to an improved relationship between the research and health care enterprises—on behalf of the public they both serve.

## Footnotes

*   Conflicts of interest: authors report none.

*   **Funding support:** This project has been funded in whole or in part with federal funds from the National Cancer Institute, National Institutes of Health, under contract No. HHSN261200800001E. Dr Stange’s time is supported in part by a Clinical Research Professorship from the American Cancer Society.

*   **Disclaimer:** The content of this article does not necessarily reflect the views or policies of the National Cancer Institute or the Department of Health and Human Services, nor does mention of trade names, commercial products, or organizations imply endorsement by the US government.

*   Received for publication July 17, 2013.
*   Revision received May 13, 2014.
*   Accepted for publication June 2, 2014.


*   © 2014 Annals of Family Medicine, Inc.

## References

1.  Smith JM, Topol E. A call to action: lowering the cost of health care. Am J Prev Med. 2013;44(1)(Suppl 1):S54–S57.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.amepre.2012.09.015&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=23195170&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

2.  Patient-Centered Outcomes Research Institute (PCORI). Funding announcement: Improving healthcare systems. May 22, 2012. [http://www.pcori.org/assets/PFA-Improving-Healthcare-Systems-05222012.pdf](http://www.pcori.org/assets/PFA-Improving-Healthcare-Systems-05222012.pdf). Accessed Jul 15, 2013.
    
    

3.  Nutting PA, Crabtree BF, Miller WL, Stange KC, Stewart E, Jaén C. Transforming physician practices to patient-centered medical homes: lessons from the national demonstration project. Health Aff (Millwood). 2011;30(3):439–445.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiaGVhbHRoYWZmIjtzOjU6InJlc2lkIjtzOjg6IjMwLzMvNDM5IjtzOjQ6ImF0b20iO3M6MjM6Ii9hbm5hbHNmbS8xMi81LzQ0Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

4.  Fisher ES, Shortell SM. Accountable care organizations: accountable for what, to whom, and how. JAMA. 2010;304(15):1715–1716.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.2010.1513&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=20959584&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000283129700028&link_type=ISI) 

5.  Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA. 2012;307(15):1583–1584.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.2012.500&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=22511682&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000302896100016&link_type=ISI) 

6.  Stange KC. Refocusing knowledge generation, application, and education: raising our gaze to promote health across boundaries. Am J Prev Med. 2011;41(4)(Suppl 3):S164–S169.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.amepre.2011.06.022&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=21961659&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

7.  Balas EBS. Managing Clinical Knowledge for Health Care Improvement. Yearbook of Medical Informatics. Stuttgart, Germany: Schattauer; 2000.
    
    

8.  1.  Brownson RC, 
    2.  Colditz GA, 
    3.  Proctor EK
    
    , eds. Dissemination and Implementation Research in Health: Translating Science to Practice. 1st ed. New York, NY: Oxford University Press, Inc; 2012.
    
    

9.  Kessler R, Glasgow RE. A proposal to speed translation of healthcare research into practice: dramatic change is needed. Am J Prev Med. 2011;40(6):637–644.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.amepre.2011.02.023&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=21565657&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000290470400012&link_type=ISI) 

10. Moher D, Hopewell S, Schulz KF, et al; Consolidated Standards of Reporting Trials Group. CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol. 2010;63(8):e1–e37.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2010.03.004&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=20346624&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

11. Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010;340:c332.
    
    

12. Higgins J, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Updated 2011. [http://www.cochrane-handbook.org](http://www.cochrane-handbook.org). Accessed Jul 16, 2013.
    
    

13. Equator Network. Library for Health Research Reporting. EQUATOR Network Resource Center website. Updated 2013. [http://www.equator-network.org/resource-centre/library-of-health-research-reporting](http://www.equator-network.org/resource-centre/library-of-health-research-reporting). Accessed Jul 16, 2013.
    
    

14. Gold M, Siegel J, Russell L, Weinstein M. Cost Effectiveness in Health and Medicine. New York, NY: Oxford University Press; 2003.
    
    

15. Solberg LI, Glasgow RE, Unützer J, et al. Partnership research: a practical trial design for evaluation of a natural experiment to improve depression care. Med Care. 2010;48(7):576–582.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1097/MLR.0b013e3181dbea62&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=20508531&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000279428200003&link_type=ISI) 

16. Kottke TE, Solberg LI, Nelson AF, et al. Optimizing practice through research: a new perspective to solve an old problem. Ann Fam Med. 2008;6(5):459–462.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6NzoiNi81LzQ1OSI7czo0OiJhdG9tIjtzOjIzOiIvYW5uYWxzZm0vMTIvNS80NDcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

17. Berwick DM. Broadening the view of evidence-based medicine. Qual Saf Health Care. 2005;14(5):315–316.
    
    [FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjE0LzUvMzE1IjtzOjQ6ImF0b20iO3M6MjM6Ii9hbm5hbHNmbS8xMi81LzQ0Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

18. Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?” Lancet. 2005;365(9453):82–93.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/S0140-6736(04)17670-8&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=15639683&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000226213100032&link_type=ISI) 

19. Glasgow RE, Chambers D. Developing robust, sustainable, implementation systems using rigorous, rapid and relevant science. Clin Transl Sci. 2012;5(1):48–55.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1111/j.1752-8062.2011.00383.x&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=22376257&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

20. Kessler RS. The patient centered medical home: a great opportunity to move beyond brilliant and irrelevant. Transl Behav Med. 2012;2(3):311–312.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1007/s13142-012-0151-6&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=24073130&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

21. Riley WT, Glasgow RE, Etheredge L, Abernethy AP. Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clin Transl Med. 2013;2(1):10.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1186/2001-1326-2-10&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=23663660&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

22. Green LW. Making research relevant: if it is an evidence-based practice, where’s the practice-based evidence? Fam Pract. 2008;25(Suppl 1):i20–i24.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiZmFtcHJhY3QiO3M6NToicmVzaWQiO3M6MTQ6IjI1L3N1cHBsXzEvaTIwIjtzOjQ6ImF0b20iO3M6MjM6Ii9hbm5hbHNmbS8xMi81LzQ0Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

23. Krist AH, Glenn BA, Glasgow RE, et al. Designing a valid randomized pragmatic primary care implementation trial: the My Own Health Report (MOHR) project. Implement Sci. 2013;8:73.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1186/1748-5908-8-73&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=23799943&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

24. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003;290(12):1624–1632.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.290.12.1624&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=14506122&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000185461300028&link_type=ISI) 

25. Solberg LI, Crain AL, Jaeckels N, et al. The DIAMOND initiative: implementing collaborative care for depression in 75 primary care clinics. Implement Sci. 2013;8(1):135.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1186/1748-5908-8-135&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=24238225&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

26. Abernethy AP, Etheredge LM, Ganz PA, et al. Rapid-learning system for cancer care. J Clin Oncol. 2010;28(27):4268–4274.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamNvIjtzOjU6InJlc2lkIjtzOjEwOiIyOC8yNy80MjY4IjtzOjQ6ImF0b20iO3M6MjM6Ii9hbm5hbHNmbS8xMi81LzQ0Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

27. Etheredge LM. A rapid-learning health system. Health Aff (Millwood). 2007;26(2):w107–w118.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiaGVhbHRoYWZmIjtzOjU6InJlc2lkIjtzOjk6IjI2LzIvdzEwNyI7czo0OiJhdG9tIjtzOjIzOiIvYW5uYWxzZm0vMTIvNS80NDcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

28. Mays GP, Hogg RA. Expanding delivery system research in public health settings: lessons from practice-based research networks. J Public Health Manag Pract. 2012;18(6):485–498.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1097/PHH.0b013e31825f75c9&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=23023272&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

29. 1.  Denzin N, 
    2.  Lincoln Y
    
    Ryan B. Data management and analysis methods. In: Denzin N, Lincoln Y, eds. Handbook of Qualitative Research. 2nd ed. Thousand Oaks, CA: Sage Publications; 2000:769–802.
    
    

30. Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof. 2006;29(1):126–153.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3BlaHAiO3M6NToicmVzaWQiO3M6ODoiMjkvMS8xMjYiO3M6NDoiYXRvbSI7czoyMzoiL2FubmFsc2ZtLzEyLzUvNDQ3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

31. Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks P. The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med. 2004;27(1):3–12.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1207/s15324796abm2701_2&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=14979858&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000188245300002&link_type=ISI) 

32. Ritzwoller DP, Sukhanova A, Gaglio B, Glasgow RE. Costing behavioral interventions: a practical guide to enhance translation. Ann Behav Med. 2009;37(2):218–227.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1007/s12160-009-9088-5&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=19291342&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000267029200012&link_type=ISI) 

33. Ritzwoller DP, Sukhanova AS, Glasgow RE, et al. Intervention costs and cost-effectiveness for a multiple-risk-factor diabetes self-management trial for Latinas: economic analysis of Viva bien! Transl Behav Med. 2011;1(3):427–435.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1007/s13142-011-0037-z&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=22081776&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

34. Porter ME. What is value in health care? N Engl J Med. 2010;363(26):2477–2481.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1056/NEJMp1011024&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=21142528&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000285555600001&link_type=ISI) 

35. deBronkart D. How the e-patient community helped save my life: an essay by Dave deBronkart. BMJ. 2013;346:f1990.
    
    [FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDYvYXByMDJfMS9mMTk5MCI7czo0OiJhdG9tIjtzOjIzOiIvYW5uYWxzZm0vMTIvNS80NDcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

36. Cohn S, Clinch M, Bunn C, Stronge P. Entangled complexity: why complex interventions are just not complicated enough. J Health Serv Res Policy. 2013;18(1):40–43.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3Boc3IiO3M6NToicmVzaWQiO3M6NzoiMTgvMS80MCI7czo0OiJhdG9tIjtzOjIzOiIvYW5uYWxzZm0vMTIvNS80NDcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

37. Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA. 1998;279(4):281–286.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.279.4.281&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=9450711&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000071607000034&link_type=ISI) 

38. Tomoaia-Cotisel A, Scammon DL, Waitzman NJ, et al. Context matters: the experience of 14 research teams in systematically reporting contextual factors important for practice change. Ann Fam Med. 2013;11(Suppl 1):S115–S123.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6MTU6IjExL1N1cHBsXzEvUzExNSI7czo0OiJhdG9tIjtzOjIzOiIvYW5uYWxzZm0vMTIvNS80NDcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

39. Crain AL, Solberg LI, Unützer J, et al. Designing and implementing research on a statewide quality improvement initiative: the DIAMOND study and initiative. Med Care. 2013;51(9):e58–e66.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1097/MLR.0b013e318249d8a4&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=22322100&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000323411600001&link_type=ISI) 

40. Glasgow RE, Magid DJ, Beck A, Ritzwoller D, Estabrooks PA. Practical clinical trials for translating research to practice: design and measurement recommendations. Med Care. 2005;43(6):551–557.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1097/01.mlr.0000163645.41407.09&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=15908849&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000229379200005&link_type=ISI) 

41. Cohen D, McDaniel RR Jr., Crabtree BF, et al. A practice change model for quality improvement in primary care practice. J Healthc Manag. 2004;49(3):155–168, discussion 169–170.
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=15190858&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000221496600005&link_type=ISI) 

42. 1.  Brownson R, 
    2.  Colditz G, 
    3.  Proctor E
    
    Allen J, Linnan L, Emmons K. Fidelity and its relationship to implementation effectiveness, adaptation and dissemination. In: Brownson R, Colditz G, Proctor E, eds. Dissemination and Implementation Research in Health: Translating Science to Practice. 1st ed. New York, NY: Oxford University Press; 2012:281–303.
    
    

43. Cohen DJ, Tallia AF, Crabtree BF, Young DM. Implementing health behavior change in primary care: lessons from prescription for health. Ann Fam Med. 2005;3(Suppl 2):S12–S19.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6MTM6IjMvc3VwcGxfMS9zMTIiO3M6NDoiYXRvbSI7czoyMzoiL2FubmFsc2ZtLzEyLzUvNDQ3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

44. Cohen DJ, Crabtree BF, Etz RS, et al. Fidelity versus flexibility: translating evidence-based research into practice. Am J Prev Med. 2008;35(5)(Suppl):S381–S389.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.amepre.2008.08.005&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=18929985&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000260396800006&link_type=ISI) 

45. Cohen D, Leviton L, Isaacson N, Tallia A, Crabtree B. Online diaries for qualitative evaluation: gaining real-time insights. Am J Eval. 2006;27(2):1–22.
    
    

46. Etz RS, Cohen DJ, Woolf SH, et al. Bridging primary care practices and communities to promote healthy behaviors. Am J Prev Med. 2008;35(5)(Suppl):S390–S397.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.amepre.2008.08.008&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=18929986&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

47. Balasubramanian BA, Cohen DJ, Clark EC, et al. Practice-level approaches for behavioral counseling and patient health behaviors. Am J Prev Med. 2008;35(5)(Suppl):S407–S413.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.amepre.2008.08.004&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=18929988&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

48. Cohen DJ, Balasubramanian BA, Isaacson NF, Clark EC, Etz RS, Crabtree BF. Coordination of health behavior counseling in primary care. Ann Fam Med. 2011;9(5):406–415.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6NzoiOS81LzQwNiI7czo0OiJhdG9tIjtzOjIzOiIvYW5uYWxzZm0vMTIvNS80NDcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

49. Glasgow R, Kessler R, Ory M, et al. Conducting rapid, relevant research: lessons learned from the My Own Health Report (MOHR) project. Am J Prev Med. 2014;47(2):212–219.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.amepre.2014.03.007&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=24953520&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

50. Thorpe KE, Zwarenstein M, Oxman AD, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62(5):464–475.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2008.12.011&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=19348971&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000265424400002&link_type=ISI) 

51. Senge PM. Fifth Discipline: Mastering the Five Practices of the Learning Organization. New York, NY: Doubleday & Co; 1990.
    
    

52. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health. 2013;103(6):e38–e46.
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=23597377&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

53. Kessler RS, Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ. What does it mean to “employ” the RE-AIM model? Eval Health Prof. 2013;36(1):44–66.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3BlaHAiO3M6NToicmVzaWQiO3M6NzoiMzYvMS80NCI7czo0OiJhdG9tIjtzOjIzOiIvYW5uYWxzZm0vMTIvNS80NDcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

54. Wagner EH. Academia, chronic care, and the future of primary care. J Gen Intern Med. 2010;25(Suppl 4):S636–S638.
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=20737241&link_type=MED&atom=%2Fannalsfm%2F12%2F5%2F447.atom) 

55. Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.
    
    

56. Gluck ME, Radomski L. The AcademyHealth Listening Project Report: Improving the Evidence Base for Medicare Policymaking. Academy-Health. Updated February 2014. [http://academyhealth.org/files/publications/listeningprojectmedicare.pdf](http://academyhealth.org/files/publications/listeningprojectmedicare.pdf). Accessed Apr 4, 2014.