The 5 R’s: An Emerging Bold Standard for Conducting Relevant Research in a Changing World

C. J. Peek; Russell E. Glasgow; Kurt C. Stange; Lisa M. Klesges; E. Peyton Purcell; Rodger S. Kessler

doi:10.1370/afm.1688

Article Figures & Data

Tables

Additional Files

View popup

Table 1

Examples of Projects Illustrating the R’s

Example: Study/Project/Resource Title and Relevant R’s	Study/Project/Resource Details
Particular studies and programs
DIAMOND (Depression Improvement Across Minnesota–Offering a New Direction) 1. Relevant to stakeholders 2. Redefines rigor 3. Replicability	The DIAMOND initiative for depression in primary care was a statewide collaborative of practices and health plans accompanied by a separately funded NIMH research study using a stepped wedge/phased intervention design.^15,25,39,40 Practices launched the DIAMOND care process in cohorts, 6 months apart, with baseline data collected for all. Outcomes that mattered to different stakeholders were compared before and after launch across the many practices launching at different times. Outcomes were tracked using both quantitative and qualitative measures, including clinical outcomes, health plan claims data, patient surveys, and practice leader surveys regarding implementation. An explicit balance of fidelity and adaptation to local situations—specifics that practices had to tailor for themselves (eg, choice of discipline for care managers, specific workflow for PHQ-9, type of data tracking system)—helped practices implement the DIAMOND intervention.^41,42
P4H (Prescription for Health) 1. Relevant to stakeholders 2. Recursive 3. Redefines rigor 4. Reports on resources	P4H was an initiative of The Robert Wood Johnson Foundation (RWJF) with the Agency for Healthcare Research and Quality (AHRQ) to fund a collaboration of 17 PBRNs that developed and evaluated strategies to improve health behavior changes for multiple behaviors through linkage to community resources.43 Practices worked with researchers, and teams of researchers, and PBRN leaders worked with each other and with a cross-cutting research group to share evolving learning, and develop common measures and an evolving research agenda.⁴⁴ Using mixed quantitative and qualitative methods (including researcher diary data and interviews)⁴⁵ and cost analyses, P4H showed that primary care practices have the ability to develop their linkages to connect patients with community resources⁴⁶ to improve practice processes,⁴⁷ health behavior counseling, and patient behavior change.⁴⁸
¡Viva Bien! 1. Relevant to stakeholders 2. Reports on resources 3. Replicability	¡Viva Bien!^32,33 was a randomized trial that provided a clear description of methods, implementation costs for a diabetes self-management program, and estimates of costs to replicate the program under different conditions, calculating incremental costs per behavioral, biologic, and quality-of-life change. It discussed how to separate the costs of development and research from implementation, and how to conduct relatively straight-forward sensitivity analyses to estimate costs of replicating a program or policy under different conditions.
MOHR (My Own Health Report) 1. Relevant to stakeholders 2. Rapid and recursive 3. Redefines rigor 4. Reports on resources 5. Replicability	MOHR^23,49 is a pragmatic participatory trial in which diverse primary care practices implement the collection of patient-reported information and provide patients advice, goal setting, and counseling in response—with deliberate diversity of settings and populations to ensure greater generalizability of results. Practices, patients, funding agencies, and content experts were engaged throughout the study to take into account local resources and characteristics in design, implementation, evaluation, and dissemination. Core elements of the study protocol were identified, with local tailoring to ensure implementation was relevant to local culture and practice on issues such as workflows, eligible patients, when and where assessment would be completed, whether electronic or paper, and how clinicians would receive the feedback. The trial used mixed methods, including cost analyses.
Research networks across studies
PRC (Prevention Research Centers) of the Centers for Disease Control and Prevention 1. Relevant to stakeholders 2. Recursive 3. Replicability	PRC directs a national network of 37 academic research centers at public health or medical schools with a preventive medicine residency program, translating research results into policy and public health practice. Centers have capacity for community-based, participatory prevention research needed to drive community changes to prevent and control chronic disease. Research involves collaboration among partners bringing different expertise to the table, identifies research needs of partners, conducts research that builds on previous evidence for promising interventions, and recommends how interventions can be packaged for replication and adoption (http://www.cdc.gov/prc/index.htm).
QUERI (Quality Enhancement Research Initiative) 1. Relevant to stakeholders 2. Rapid and recursive 3. Redefines rigor 4. Replicability	QUERI is a Veterans Affairs initiative that brings together operations with research staff to address key gaps in quality and outcomes. It has contributed to remarkable and rapid improvements in the quality of care received by veterans across 10 conditions deemed high-risk or highly prevalent. This initiative uses a 6-step process to spot gaps in performance and to identify and implement interventions. QUERI studies and facilitates adoption of new treatments, tests, and models of care into routine clinical practice—feasibility, implementation, adoption, and impact (http://www.queri.research.va.gov/default.cfm).
Research application tools and resources
RTIPs (Research Tested Intervention Programs) 1. Relevant to stakeholders 2. Reports on resources 3. Replicability	RTIPs is a resource of the National Cancer Institute that provides information on the specific conditions under which each of their tested interventions has been evaluated and tools for addressing issues about applicability (http://rtips.cancer.gov/rtips). New features related to external validity using the RE-AIM framework are included to help users better determine the likely public health impact of a given program if replicated in their setting. RTIPs also reports on the resources required to implement these programs.
PRECIS (Pragmatic Explanatory Continuum Indicator Summary) 1. Relevant to stakeholders 2. Redefines rigor 3. Replicability	PRECIS50 is a graphic representation of the extent to which a study is pragmatic (testing effect in usual conditions) vs explanatory (testing effect in ideal conditions) on 10 key dimensions. If used consistently, this tool could greatly help practitioners decide whether a study is likely to be reproducible in their setting and researchers to investigate the dimensions along which similarity is more vs less critical for replication.

NIMH = National Institute of Mental Health; PBRN = practice-based research network; PHQ-9 = 9-item Patient Health Questionnaire; RE-AIM = Reach Effectiveness–Adoption Implementation Maintenance.

View popup

Table 2

Questions to Apply the 5 R’s at Each Stage of the Research and Dissemination Process

Stage of Research	Bold Standard 5 R’s
Stage of Research	Relevant to Stakeholders	Rapid and Recursive	Redefines Rigor	Reports on Resources	Replicability
Design	End users of this research identified? Stakeholders who need to be involved identified? Plan in place to engage their perspectives? Plan in place to gather stakeholder questions and what is important to them?	Rapid cycle measurement and assessment built into the design? How? Approach in place to allow early discoveries to shape the study?	How is study systematic and pragmatic about concepts, measures, data collection procedures, and analysis plan? Multiple methods used? How? Internal and external validity balanced? How?	Intervention costs (monetary and other) measured? How? A standard vocabulary for reporting on resources in place? What?	Study designed to inform implementation and reinvention in different settings? How? Likely relevant settings for this research identified?
Implementation	Stakeholders involved in ongoing refinement? How? Changes they suggested along the way recorded? Changes suggested implemented? Which ones?	Short-cycle learning taking place to refine design and measurement? Is learning influencing the study? How?	Systematic approach being followed to concepts, tools, data collection, measures, procedures, analyses? Checks for bias and superfluous connections in place? Clear description of what is being done recorded?	Cost data gathered on an ongoing basis? Using a consistent vocabulary for different kinds of costs?	Contextual factors documented that are important to understanding what happened (and why) in the study setting?
Reporting	Diverse stakeholders involved in interpreting and reporting findings? Their different interpretations reported?	Emergent findings shared on an ongoing basis throughout the study? Have adaptations made been reported?	Study methods reported transparently and thoroughly? Reported how study checked for potential biases and superfluous connections? Reported how conclusions are justified by standards of evidence?	Study reports useful cost data using a defined vocabulary for different kinds of costs? Estimates made for costs under different conditions?	Contextual factors relevant to reinvention in new settings reported, including variation across settings or within settings?
Dissemination	Target audiences, stakeholders, or likely users involved in next steps? Findings expressed in language and context that mean something to different stakeholders?	Guidelines provided for adaptation and customization/tailoring for future use?	Description included for how internal and external validity findings support wider use?	Intervention cost data discussed as a factor in dissemination?	Data-supported suggestions included about the contexts for which program or intervention is relevant or reproducible?

View popup

Table 3

Challenges and Changes for Routine Implementation of 5 R’s

Challenges	Changes to Address Challenges
Accelerating the pace and iterative nature of the research enterprise Decision-maker needs outpace current speed of review cycles: grant review; funding decision; IRB approval and modification processes Study implementation time frames Publication cycles not amenable to “just in time” decisions; slow review and release of findings (see more below on dissemination) Low priority assigned to designs that can speed research	Harness stakeholder interest in timeliness to drive a cultural shift to shorten what is considered “rapid” or “timely” compared with present custom Implement a variety of technical changes to research processes already suggested in literature^{19,21,26,27,54} Use rapid-cycle testing of hypotheses, allowing ineffective ideas to “fail fast” and successful innovations to spread quickly Link social media with traditional communications vehicles
Expanding limited concepts of rigor (eg, preference for, confidence in, or insistence on certain designs such as RCTs) by: Funding agencies offering calls for proposals Grant application reviewers Researchers “Customers” of research (stakeholders who use the findings)	Among all parties, build awareness of and comfort with a broader “palette” of research designs, so that research design is driven by the questions, rather than research questions driven by designs Use professional meetings/training events to more clearly articulate features, pros/cons of different designs—their appropriate or promising scope of application
Ensuring a blend of research team skills and interests Skill and interest in stakeholder involvement in generating questions, articulating ultimate use of study findings, study design, implementation, reporting, and dissemination Awareness of and respect for political as well as scientific concerns of stakeholders such as policy makers Skill and comfort in building relationships with clinicians and clinics—consultative, cooperative, problem solving Experience and confidence with the broader “palette” of research designs, including rapid learning in real-world experiments	Propose an enhanced “job description” for research teams—a checklist of skills, interests, and relationships required for specific studies Beyond essential methodologic, data-gathering, and analytic skills, include “softer” skills and methods such as shown in left column Build up those skills through examples, conferences, and training among both existing and new researchers
Increasing clinician familiarity with being active research partners Negative experiences or preconceptions about feasibility or practical value of doing research in the practice Few or no current relationships with researchers Unfamiliarity of working with researchers to turn practice concerns and curiosity into researchable questions Unfamiliarity with building research data gathering into routine clinic systems rather than being an effortful “add on” Not connecting research with more familiar quality improvement, rapid-cycle learning	Provide examples and assistance through professional venues and practice facilitation or technical assistance that help clinicians and researchers adjust mindset, methods, and interactions to create practical research partnerships along the lines described in the literature^15,16,25
Raising priority on collection and reporting on context and resources Limited researcher and reviewer expectation that data on resource use of interventions or on context information relevant to transportability or reinvention in new settings be gathered systematically or reported Space limitations and/or customary priorities in journals that reduce additional context and resource data reporting	Adjust research announcements and grant review guidelines to ask for greater reporting on context and resources required; accompany by explanation of why For publication in limited space, consider other methods such as web supplements to access detailed context and resource use data if not in standard published article
More powerfully bringing publication and dissemination to practical decision making Limited readiness to publish replications of key findings in original or new contexts or to publish negative results of replication Reaching those stakeholders who want to make research-based decisions at the time and place decisions are made Limited dissemination in publications or forms in which stakeholders are already engaged, knowing that different forms of publication/dissemination reach different stakeholders	Publish replications (successful or not) in places where stakeholders will find them Reward researchers via funding and career paths for key replications, not only for new positive results Create a stakeholder map—which stakeholders need what information from the study, in what form, and where it is most likely to be read Create stakeholder-specific versions of core journal publications to increase reach of the information

IRB = institutional review board; RCT = randomized controlled trial.

Additional Files

Tables

In Brief

The 5 R's: An Emerging Bold Standard for Conducting Relevant Research in a Changing World

C.J. Peek , and colleagues

Background The accelerated pressure for change in health care has created an exploding need for relevant and rapidly generated new information. The current slow and fragmented approach to research, however, often fails to address practical needs for decision making. This paper synthesizes several existing approaches and develops a new "5 R's" standard to guide research and help meet the changing needs of health care delivery.

What This Study Found The 5Rs are intended to generate research that 1) is relevant to stakeholders, 2) is rapid and recursive in application, 3) redefines rigor, 4) reports on resources required, and 5) is replicable. The R's of the research process are mutually reinforcing and can be supported by training that fosters collaborative and reciprocal relationships among researchers, implementers, and other stakeholders.

Implications

The 5R's can serve as a framework for discussion and adjustment of criteria for what is considered high-quality research. Consistent and bold application of this standard, the authors conclude, will increase the value, timeliness and applicability of the research enterprise.