Abstract
PURPOSE We aimed to advance the internal and external validity of research by sharing our empirical experience and recommendations for systematically reporting contextual factors.
METHODS Fourteen teams conducting research on primary care practice transformation retrospectively considered contextual factors important to interpreting their findings (internal validity) and transporting or reinventing their findings in other settings/situations (external validity). Each team provided a table or list of important contextual factors and interpretive text included as appendices to the articles in this supplement. Team members identified the most important contextual factors for their studies. We grouped the findings thematically and developed recommendations for reporting context.
RESULTS The most important contextual factors sorted into 5 domains: (1) the practice setting, (2) the larger organization, (3) the external environment, (4) implementation pathway, and (5) the motivation for implementation. To understand context, investigators recommend (1) engaging diverse perspectives and data sources, (2) considering multiple levels, (3) evaluating history and evolution over time, (4) looking at formal and informal systems and culture, and (5) assessing the (often nonlinear) interactions between contextual factors and both the process and outcome of studies. We include a template with tabular and interpretive elements to help study teams engage research participants in reporting relevant context.
CONCLUSIONS These findings demonstrate the feasibility and potential utility of identifying and reporting contextual factors. Involving diverse stakeholders in assessing context at multiple stages of the research process, examining their association with outcomes, and consistently reporting critical contextual factors are important challenges for a field interested in improving the internal and external validity and impact of health care research.
- context
- methods
- research design
- validity
- generalizability
- patient-centered medical home
- primary care
- change
- organizational
- practice-based research
- transformation
INTRODUCTION
Context is important for most phenomena of health care and health.1–9 But contextual factors rarely are recorded, analyzed, or included in research reports.7 As a result, attempts to replicate research often fail, because context important to interpreting and applying findings remains unknown. Efforts to translate research into practice often fail because contextual factors important for understanding and knowledgably synthesizing findings across studies in meta-analyses and evidence-based guidelines are not known.10 As a recent analysis noted, “Investigation of how intervention effects are modified by context is a new methodological frontier in community intervention trial research.”11
Although prior research has attempted to identify contextual factors based on theory,8,11,12 technical expert panels,6,12 reviews of (often decontextualized) published research,2,7,9,10,12,13 or surveys of individuals participating in quality improvement projects,14 we had a unique opportunity to develop and empirically apply an approach to identifying and reporting contextual factors across 14 studies of primary care practice change funded by the Agency for Healthcare Research and Quality (AHRQ). We attempted to consider and prioritize the most important contextual elements from a wide range of factors that could affect the interpretation and transportability of findings.
Specifics of contextual factors identified in these studies are reported in appendixes to each article in this supplement.15 Here, we summarize major contextual factors identified across these studies and make recommendations for reporting context affecting the interpretation, implementation, and dissemination of research.
METHODS
In the summer of 2010, AHRQ awarded 14 grants to evaluate successful “natural experiments” with potential to inform the process by which primary care practices transform into patient-centered medical homes (PCMHs). McNellis and colleagues16 present an overview of each project. The projects included a wide range of community, health care, and payer systems; practice types; and patient populations; they also had considerable diversity in the natural experiments they evaluated.
Grantees agreed to apply a template for context reporting, recently proposed by Stange and Glasgow,17 as their projects were nearing completion. They involved as many stakeholders as feasible in identifying contextual factors most important to understanding what happened and why in their studies; what others would want to know when transporting study findings to different settings, times, and situations; domains of potentially important contextual factors; and up to 500 words of explanatory text.
For this article, a subset of authors (A.T-C., D.L.S., N.W.) reviewed the 14 articles and completed templates, and identified themes by iteratively grouping similar reported factors. Subsequently, each research team determined 3 or 4 groups of contextual factors they judged to be most important for understanding or transporting study findings. The same subset of authors reviewed these contextual factors and refined emergent themes. Themes were verified and further refined through member checks with at least 1 representative from participating teams (11 of 14 teams participated in the verification). The subset of authors identified quotations to illustrate each theme and depicted themes in a contextual framework. In addition, each team submitted a list of tips for reporting contextual factors that were thematically organized by 2 authors (P.F.C., J.R.H.). This process informed the development of a refined template for reporting context.
RESULTS
Framework of Contextual Factors
The framework that emerged from our analysis classified contextual factors into 5 domains: 3 pertaining to specific levels and 2 themes that cut across levels, as depicted in Table 1 and Figure 1.
Themes and Examples of Contextual Factors
Understanding an intervention’s context: emergent contextual framework.
The 3 levels of context highlight the interrelated and sometimes hierarchical nature of contextual factors. Level 1, the practice level, includes practice characteristics and factors directly related to experiences of patients and clinicians. Facets such as clinician and employee mix, staff demographics and training, patient panel size and characteristics, and structural capabilities define this level. Level 2, the larger organizational level, captures factors such as leadership structure and character, degree and nature of formal integration of care processes, and contractual arrangements, including payment models, that help shape the practice environment. Level 3, the external environment, denotes factors outside of the practice/organization, such as political authority, market structure, external sources of financing, and coordination with the community.
The 2 cross-cutting themes highlight important contextual processes not readily discernible in these levels. One theme, implementation pathways, captures locally relevant elements of an intervention, including operational changes (eg, addition of new employees, redefined roles, team communication strategies, feedback loops) as well as objectives of the intervention and outcomes (eg, health status of targeted populations or populations, patient satisfaction, and financial stability). The other theme, motivation for implementation, allows researchers to identify key drivers of an intervention including an advocacy-driven mission or other goals such as enhanced patient experience, quality and cost of care, and incentives that motivate implementation of the intervention.
To illustrate the framework, we provide examples and representative quotations below.
Levels of Context
At the practice level, training of practice members, characteristics affecting how they work together, and leadership are often relevant contextual factors. One team noted that their “practice included primary and mental health care providers…the colocation of services made it easier to integrate care.” Another team highlighted the importance of strong “visionary leaders among their practice leads (usually a lead provider)” and “the presence of boundary spanners at all levels within the practice helped sustain the work of transformation and diffuse information.”
At the organizational level, membership in a larger network often has implications for initiatives to bring about practice change. For example, the fact that one study was conducted in a network of 17 Federally Qualified Health Centers (FQHCs) was important because the “FQHC care model incorporates many elements of PCMH, so the concepts were not new to the staff.”
At the external environment level, factors pertaining to the political milieu, market forces, and community resources all may come into play. For example, for some projects, the availability of external financial support was viewed as critical. As one team noted: “The PCMH transformation initiative was supported by the 3 largest commercial insurers and the 3 Medicaid managed care organizations in the region. Practices received quarterly lump-sum payments based on the size of their practices.” For other projects, the built environment was important. According to another team: “The different geographic settings presented different needs (eg, transportation is an issue for patients in rural areas).”
Cross-Cutting Themes
Operational changes, and their sequence of implementation during the project, were important. In one study:
New members of the care team were hired (eg, diabetes educators, expansion of social work staff) and tasks were assigned to others (eg, depression screening and monofilament testing conducted by nurses, outreach team contacting patients about concerning clinical measures).
Another team observed: “A stepwise implementation process (over time), built on an existing [electronic health record]; started with appropriate access, then added the care team, and then the planned care components.”
The motivation for implementing the intervention also was seen as critical. “Many of the practices wanted to ‘do what’s best for their patients’; improving clinical care was a main driver of the initiative and a source of motivation for all levels of stakeholders.”
Brief Examples
The examples below, and the appendixes of each article in this supplement,15 illustrate the importance of contextual factors identified by our method. Typically, these factors are only hinted at in research reporting.
Example 1
A study of 120 certified medical homes in Minnesota found extensive variation among them in performance measures (SD = 30%) and presence of PCMH practice systems (SD=20%).18 To understand the external and internal validity of this finding, it is important to know that 75% of these clinics were part of large medical groups (having ≥21 clinics). The practices would therefore have more resources for change and be under pressure to be similar in function and performance within each large group. In addition, to become certified, clinics had to meet standards for access, registry use, care coordination, care plans, and continuous improvement, which were included in the systems. Finally, the standards and certification process were rigorous enough that only 20% of primary care practices in the state applied for certification. These contextual factors should have produced much smaller variation, so the study findings suggest it would be a mistake to expect that all PCMHs are meaningfully similar or will have similar results.
Example 2
The North Carolina Improving Performance in Practice initiative19 described an external quality improvement environment that included key stakeholders from Medicaid case management, an Area Health Education Center–led practice support program with medical society backing, and a major private payer reimbursing for higher-quality care. As part of the larger organization driving change, coaches helped practices implement key improvement elements. At the practice level, top leaders facilitated changes by providing vision and then delegating operational authority to a middle manager to carry out activities. Cross-cutting motivating factors included an economic recession period that motivated practice members to look for alternative revenue streams. When trying to transport this intervention, it would be important to align practice change with available external incentives, such as Maintenance of Certification for medical boards, meaningful use, and insurer support. It also would be important to identify practices that value population health and consider the availability of practice coaching to keep practices on track when making office system changes.
Example 3
The Group Health Cooperative project,20 set in a system that integrates insurance and care delivery for a defined population, has many baseline organizational features that may not be present in other settings. At Group Health, physicians have historically been paid by salary rather than according to fee for service or capitation. Patients are empaneled to primary care physicians who work in teams of physician assistants, registered nurses, clinical pharmacists, licensed practical nurses, and medical assistants. Primary care teams are supported by a “medical neighborhood” that includes a 24-hour consulting nurse service, case management programs, and 10 specialty and behavioral health units, 6 urgent care centers, and 7 acute care hospitals (6 contracted and 1 maternity hospital). Although clinics are part of an integrated system clinic, systemwide and local leadership still had considerable impact on the way key elements were rolled out and overall willingness to change in each clinic.
During the study period, several important changes affected the transformation process, including substantial (19%) growth of the enrolled patient population between 2007 and 2011, and concomitant growth in the number of employees, creating unanticipated and uneven challenges for training, team development, and patient paneling. Further, in 2007, Group Health adopted the Lean Production System as its central method for quality improvement, change management, and reduction of waste. Unlike other PCMH demonstration projects that rely on practice facilitation or coaching, Lean methods emphasize use of front-line team members in process improvement and problem solving. The study period also featured other care innovations and changes in Group Health’s integrated system that may have influenced study results, including initiatives on standardized opioid prescribing, high-end imaging, hospital transitions, and urgent care expansion.
Several of these changes intersect with contextual factors categories, such as the growth of enrollment, which started with changes in the environment (local markets) but had subsequent implications at both the organizational and practice levels. The use of Lean was an implementation pathway, but also had important organization- and practice-level impacts.
Example 4
In a state-led medical home initiative involving 152 primary care practices in 7 regions across Pennsylvania, an analysis of 25 diverse adult medicine practices focused initially on improving diabetes care.21 Important contextual factors included (1) the study’s PCMH transformation collaborative developed from an existing statewide collaborative exploring care of patients with chronic conditions (adults with diabetes and children with asthma) and overlapped with other health care reform initiatives; (2) research network practices participated in a learning collaborative with practice facilitator support; and (3) the initiative had multipayer support, which was unique to this region. Building on an existing initiative (the Chronic Care Initiative) provided a sense of partnership and common purpose, and a common condition (diabetes) around which to focus PCMH development; however, sites had difficulty separating the 2 initiatives, with the least engaged stakeholders (medical assistants and front-office staff) often conflating them.
The overlapping, but at times conflicting, goals and objectives of ongoing system initiatives complicated program implementation. The use of a learning collaborative to support regional PCMH transformation provided a common structure and expectations for the PCMH initiative. Each practice site received financial support for a care manager and had a process focus of identifying high-risk patients for outreach and better service engagement. Early on, the collaborative provided on-site practice coaches with in-person meetings for participating sites. Later, practice coaches were replaced with telephonic practice support. Practices provided monthly performance reports to the collaborative, which created a common set of expectations. Finally, the PCMH transformation initiative was supported by the 3 largest commercial insurers and 3 Medicaid managed care organizations in the region. Practices received quarterly lump-sum payments that were based on the size of their practices (clinician full-time equivalents) and largely on attainment of 2008 NCQA PCMH recognition. Without a comprehensive incentive model, it is unlikely that a breadth of practices would have participated.
Tips and a Template for Assessing and Reporting Context
Table 2 shows tips for reporting contextual factors in research based on our collective experience. Figure 2 is the refined worksheet or template for assessing contextual factors. The instructions provide information on whom to engage in the process and when and how to collect and analyze the information. The template output consists of a table of bullet points and accompanying explanatory text that can be used as a source document for reporting contextual factors in a variety of ways appropriate to the audience.
Tips for Assessing and Reporting Contextual Factors
Context Matters worksheet.
DISCUSSION
Divorcing a phenomenon from its context often is seen as a way of being objective and scientific; however, the scientific method is about systematic observation and hypothesis testing. For complex phenomena in health care and health, this systematic scientific way of generating new knowledge is enriched and made whole by considering context.
The contextual factors reported in the articles in this supplement, and the overarching lessons drawn here, show that it is feasible to identify and convey relevant information that goes well beyond the usual decontextualized way of reporting research. This is a systematic way to consider the facets of a study’s context that are essential to understanding what happened during implementation and to making reasonable judgments in transporting the findings to a new context. Reporting rich contextual information is a way of understanding contextual factors not as threats to external validity, but as opportunities to maximize learning that enrich the usefulness of systematic reviews and provide information needed to foster learning organizations22–25 and health care systems.26
This robust work was grounded in the individual and shared experiences of research teams working on a common area of investigation in diverse settings. The studies used varied methods, and many of them used qualitative methods specifically designed to incorporate relevant context from multiple perspectives.
A limitation of our analysis is that we may not have captured all domains of relevant context. The largely retrospective nature of our reporting, which undoubtedly led to underreporting of some factors, and the commonality in the research question funded by AHRQ through a single request for applications mean that other kinds of research may find different domains of context to be important. Grantees’ use of qualitative methods may underrepresent the challenges of paying attention to and reporting context in other quantitative study designs that are intended to filter out contextual factors. Importantly, this work did not prospectively evaluate the association between identified contextual factors and important process and outcome measures. Formally linking context with process and outcomes remains an important area for further methods development. These limitations, as well as the study strengths, helped to inform our recommendations for the field.
Other empirical research and theoretical models inform the approach reported here. Kaplan and colleagues12 developed a Model for Understanding Success in Quality (MUSIQ) that identifies 25 contextual factors likely to influence quality improvement success. In an exploratory survey of individuals participating in 74 quality improvement projects, they found an association of these contextual factors with at least 1 quality improvement outcome.14 Others have focused on the importance of contextual factors in implementing evidence-based guidelines,27,28 understanding complex interventions,11,29,30 and considering health and health care as complex systems.31–36
In addition, a number of health and health care models provide important clues to domains of contextual factors. For example, the chronic care model37–39 and the expanded chronic care model40 show specific community and health care system domains that may be important influences on productive clinician-patient interactions. The consolidated framework for advancing implementation science41 identifies intervention characteristics, outer and inner setting factors, characteristics of stakeholders, and processes that may be important for intervention implementation. The Conceptual Model for Considering the Determinants of Diffusion, Dissemination, and Implementation of Innovations in Health Service Delivery and Organizations provides a framework for considering the interaction of micro and macro contextual factors.42
Glasgow et al,43,44 Green et al,45,46 and others13,47 have suggested that the research community develop standards and expand the emphasis on external validity, to balance the current strong focus on internal validity. This process includes identifying and reporting relevant contextual factors needed to generalize, or at least knowledgeably reinvent research findings in other settings, situations, and times.
The richness of contextual information is not readily amenable to a checklist approach to researching and reporting, and contextual factors are likely to be interconnected.48 The approach reported here,16 however, shows that context reporting can begin with considering domains of potentially relevant contextual factors prompted by a multilevel grid/list, and then engaging multiple perspectives in deciding which specific factors are likely to be most important. Using both qualitative and quantitative methods to systematically assess these factors and how they change during the course of a study, linking them to outcomes, and then reporting them has great potential to more rapidly advance knowledge generation and inform implementation and health improvement compared with current decontextualized approaches to research and dissemination. We encourage investigators to consider potentially important contextual factors when first focusing a research question and designing a study, and to continue to assess these factors as they evolve throughout a study. We believe that funders should give extra consideration to research applications that pay attention to context over those that focus solely on internal validity, and that journals should encourage reporting of relevant context and make needed publication space available.
Individual research reports could be made more useful by reporting key contextual factors in structured abstracts and describing them in sufficient detail in the body of the article so that the reader can interpret their effect on internal and external validity of findings. Different contextual factors could be included in different sections of an article. For example, factors known at the study outset may best fit in the background or in an enhanced methods section, in the description of the setting or intervention. Contextual factors identified during the course of a study or that change over time may best fit in the results section. The most important contextual factors deserve interpretation in the discussion section.
Reporting relevant context would make meta-analyses and other forms of systematic reviews more useful by helping to make sense out of the heterogeneity of effects. This practice would lead to more useful interpretive syntheses that go beyond the common conclusion in reports that “the evidence is insufficient” and “more research is needed.”
For now, we encourage others to use, adapt, and expand on the context reporting template presented in this article, and to enhance the value, validity, and transportability of research by placing it into context.
Acknowledgments
The authors are grateful to the Agency for Healthcare Research and Quality for supporting both the research projects on which this analysis is based and the supplement in which it is being published. We also appreciate the contributions of these additional members of our research teams: Diane Rittenhouse, MD, MPH; Neil S. Calman, MD; Alison Wise Kristin Reiter, PhD; Shoou-Yih Lee, PhD; Madeline Mitchell, MURP; Warren Newton, MD, MPH; Darren DeWalt, MD, MPH; Beat Steiner, MD; Greg Randolph, MD, MPH; Ann Lefebvre, MSW; Samuel Cykert, MD; Julia J. Smith, MA; Jennifer L. Shaw, MA; Denise A. Dillard, PhD; Quenna Szafran; Kathleen Reilly, MPH; Robert J. Reid, MD, PhD; Michael K. Magill, MD; Julie Day, MD; Jaewhan Kim, PhD; Carolyn Berry, PhD; Tod Mijanovich, PhD; Margret Paul, MPH; and Russell E. Glasgow, PhD.
Footnotes
-
Conflicts of interest: authors report none.
-
Funding support: Dr Stange’s time was supported in part by a Clinical Research Professorship from the American Cancer Society.
- Received for publication March 30, 2013.
- Revision received April 9, 2013.
- Accepted for publication April 9, 2013.
- © 2013 Annals of Family Medicine, Inc.
References
In this issue
Jump to section
Related Articles
Cited By...
- The PUPPY Study - Protocol for a Longitudinal Mixed Methods Study Exploring Problems Coordinating and Accessing Primary Care for Attached and Unattached Patients Exacerbated During the COVID-19 Pandemic Year
- Regional variation in primary care improvement strategies and policy: case studies that consider qualitative contextual data for performance measurement in three Canadian provinces
- A Longitudinal Study of Trends in Burnout During Primary Care Transformation
- Use of a Clinical Care Algorithm to Improve Care for Children With Hematogenous Osteomyelitis
- In This Issue: Community Health, Clinical Care, and Clinician Calling
- An OMERACT Initiative Toward Consensus to Identify and Characterize Candidate Contextual Factors: Report from the Contextual Factors Working Group
- Factors Related to Implementation and Reach of a Pragmatic Multisite Trial: The My Own Health Report (MOHR) Study
- Synthesis Of Research On Patient-Centered Medical Homes Brings Systematic Differences Into Relief
- Research Reporting Guidelines and the New Annals Instructions for Authors
- Achieving Coordinated Care for Patients With Complex Cases of Cancer: A Multiteam System Approach
- Effects of Patient-centered Medical Home Transformation on Child Patient Experience
- Surgical Site Infection Reduction by the Solutions for Patient Safety Hospital Engagement Network
- Capturing the experiences of patients across multiple complex interventions: a meta-qualitative approach
- In This Issue: Generalist Care Around the World
- Recommendations for a Mixed Methods Approach to Evaluating the Patient-Centered Medical Home
- Perspectives in Primary Care: Implementing Patient-Centered Care Coordination for Individuals with Multiple Chronic Medical Conditions
- The 5 R's: An Emerging Bold Standard for Conducting Relevant Research in a Changing World
- Tips for Readers, Reviewers, and Authors Advancing Health and Primary Health Care
- Understanding the Context of Health for Persons With Multiple Chronic Conditions: Moving From What Is the Matter to What Matters
- Organizational Culture Associated With Provider Satisfaction
- Lessons Learned From the Study of Primary Care Transformation
- In This Issue: Practice Change--Context Matters