Effects of Facilitated Team Meetings and Learning Collaboratives on Colorectal Cancer Screening Rates in Primary Care Practices: A Cluster Randomized Trial

Shaw et al published a provocative study in the May/June 2013 issue of Annals of Family Medicine, which demonstrates the challenge of rigorously evaluating primary care practice changes.[1] They evaluate the efficacy of practice-based facilitation to improve colorectal cancer screening rates within one year. Practice-based facilitation aims to improve patient outcomes by having someone take responsibility for working with providers to analyze and change how they deliver care.[2] The published results show no effect of facilitation (a negative study) on colorectal cancer screening. In addition to its contemporary intervention and design, what is interesting about this study is that it was published.

Studies that do not demonstrate an intervention effect are less likely to be published and they pose a challenge that people reading medical research need to understand. A recent review of 23 studies of practice facilitation revealed 3 that had no effect while 20 had some modest influence (positive studies).[3] The authors also acknowledge that the bias against publishing negative studies may have influenced their conclusions that facilitation had a relatively robust effect on practice guideline adoption.[3] While it is possible that facilitation may have some modest effects, it is also plausible, based on Shaw's study, that facilitation has limited effects on colorectal cancer screening. Furthermore, it is clear that facilitation requires substantial resources that are challenging to provide in many organizations. [4] Those organizations, however, are seeking solutions to delivering the care that addresses the well-recognized quality chasm; the gap between what we know and what we do in medical practice. [5] It is therefore important to understand whether the Shaw study shows no effect of facilitation. We suggest it in fact demonstrates several of the limitations that may hide any true effect: an intervention of modest intensity, non-representative measurement of the effect, an effect measure that has limited potential to change during the period of observation, and mounting an intervention isolated from its organizational context. We will address each limitation subsequently.

Shaw et al. deliver an intervention of modest intensity, which consisted of a series of facilitated team meetings over 6 months. The theoretical underpinnings are not stated and the intervention was only fully delivered in 7 of 12 practices and not delivered at all in 2 practices. Although it is largely understood that screening is a health behavior, the intervention did not directly address a specific behavior aimed at the desired change. It is clear that physicians and medical staff need to recommend screening more often and employ more effective approaches to this communication. [6] Important questions and considerations that could drive the conceptual basis for the intervention include: Why do primary care physicians fail to recommend screening for colorectal cancer screening? How can facilitated team meetings change this behavior, and when can results be expected? Should screening rates increase immediately, or is there a lag before changes in physician practice behaviors influence screening rates? Importantly, the role of the patient cannot be ignored. Even an optimal outcome on practice behavior will not increase screening rates if patients do not schedule an appointment where the recommendation can be made. Likewise, if patients fail to keep the appointments or fail to adhere to recommendations, intervention outcomes cannot be realized. Given the diffuse nature of the intervention across a limited number of intervention clinics, and the absence of any patient-oriented intervention components, the likelihood of significant effect from this intervention was quite small.

However, the lack of positive findings may be due to the limitations of the study design and procedures rather than the intervention. It is highly doubtful that 30 consecutive patients visiting the office on a single day are representative of the patients seen at the practice. The small samples from each clinic and their lack of representativeness produced a wide range of baseline screening rates (14-93%). These are more likely the result of chance than a reasonable representation of the true screening rates at these clinics. A subsequent comparison of these baseline screening rates to the rates of an entirely different 30 consecutive patients visiting the office one year later is extremely unlikely to have produced a significant effect between intervention and control clinics. Assuming that the reported clinic screening rates were normally distributed around the mean of 49% for the 12 clinics, those practices with higher rates than the mean screening rate at baseline would be more likely to have the rate decrease in the follow-up sample, and vice versa. The odds are similar to rolling a 7 after rolling a 2 or a 12 in a game of dice. Although independent events, a higher probability event is more likely to occur after a lower probability event simply because it is a higher probability event. Had the authors evaluated another sample of 30 patients in the same practice on the same day, rather than one year later, they would have probably found similar "changes" in screening rates from one sample to the next. Clearly, the lack of adequate representative samples from the clinics involved in this study severely limits the conclusions that can be drawn.

Even with adequate, representative, and repeated samples, it is unclear if an effect could be found at one year based on the outcome criteria chosen because it was unlikely to have changed substantially in one year. If the same patients were evaluated at pre- and post- intervention, or the sample was in fact representative, then patients meeting the criteria at pre-intervention (fecal occult blood test in the last year, sigmoidoscopy in the last 5 years or colonoscopy in the last 10 years) were also likely to meet the criteria at post-intervention. Therefore, across clinics, only a little more than half of the sample (51%) would be expected to benefit from the intervention, producing an intervention ceiling effect. A practice that had 93% of their 30 patients appropriately screened at baseline could hardly expect improvement at follow-up. Clinical interventions and implementing practice change requires careful consideration of the time it will take to show an effect;[7] as well as consideration of ceiling effects, especially from practices that already have high screening rates.

The cluster randomized design is more rigorous than quasi- experimental designs,such as stepped wedge or interrupted time series,[8] but a quasi-experimental evaluation of this intervention would have allowed the investigators to study fewer clinics more intensively. These designs would have allowed the investigators to distribute resources more efficiently to ensure better treatment fidelity and evaluate the effects in a larger and more representative sample of patients at each clinic.

The problematic intervention and design flaws suggest an unreasonable expectation of detecting an intervention effect. This paper does not show the futility of practice facilitation, but instead demonstrates how factors other than the intervention can produce null findings. This is not to say that practice facilitation was inappropriately judged ineffective in this study, but that it is impossible to know the true effect of practice facilitation on colorectal cancer screening based on this study design and its procedures.

Clinical practice is changing rapidly, and expectations are high that practice changes should be "transformative." This study clarifies the challenges in detecting and measuring changes from practice-based interventions. The work of making practice changes has been demonstrated by some of these same investigators when evaluating patient centered medical homes.[4] Furthermore, these same investigators grasp the importance of the system in which the intervention is implemented.[4, 9] The investigators revealed important context within the study through qualitative examination of leadership and psychological safety. As also noted in the May/June 2013 Annals of Family Medicine, leadership has responsibility to create the environment in which teams can be successful.[10] Consideration of the system into which the intervention is introduced is a critical factor. However, 3 practices in this study only partially delivered the intervention; and 2 did not deliver it at all. Whether the organizations should also have had an explicit intervention, or all practices needed some baseline evaluations of their readiness for change, was not discussed.

Studies such as the one published by Shaw et al. help us carefully consider scientific methodology and practice change. The inclusion of a conceptual foundation for the desired effect should be considered in subsequent work. In addition, a dependable study design with adequate power is more likely to detect an effect if present. Finally, investigators who publish null results and avail their work to critical review, as well as the journals that publish such reports, are to be commended for making the investment in scientific exploration with the ultimate goal of improved patient outcomes. Committed investigators and journals are both needed to find the changes that are worth our time to implement.

1. Shaw, E.K., et al., Effects of facilitated team meetings and learning collaboratives on colorectal cancer screening rates in primary care practices: a cluster randomized trial. Ann Fam Med, 2013. 11(3): p. 220-8, S1-8.

2. Knox L, et al., Developing and Running a Primary Care Practice Facilitation Program: A How-to Guide (Prepared by Mathematica Policy Research under Contract No. HHSA290200900019I TO 5.) AHRQ Publication No. 12-0011. Rockville, MD: Agency for Healthcare Research and Quality. . December 2011.

3. Baskerville, N.B., C. Liddy, and W. Hogg, Systematic review and meta-analysis of practice facilitation within primary care settings. Ann Fam Med, 2012. 10(1): p. 63-74.

4. Crabtree, B.F., et al., Summary of the National Demonstration Project and recommendations for the patient-centered medical home. Ann Fam Med, 2010. 8 Suppl 1(8): p. S80-90; S92.

5. Institute of, M., Crossing the quality chasm: a new health system for the 21st century. 2001, Washington, DC: National Academy Press. 337.

6. Zapka, J.G. and S.C. Lemon, Interventions for patients, providers, and health care organizations. Cancer, 2004. 101(5 Suppl): p. 1165-1187.

7. Alexander, J., I. Prabhu Das, and T.P. Johnson, Time issues in multilevel interventions for cancer treatment and prevention. J Natl Cancer Inst Monogr, 2012. 2012(44): p. 42-8.

8. Woertman, W., et al., Stepped wedge designs could reduce the required sample size in cluster randomized trials. J Clin Epidemiol, 2013. 66(7): p. 752-8. 9. Stange, K.C., et al., State-of-the-art and future directions in multilevel interventions across the cancer control continuum. J.Natl.Cancer Inst.Monogr., 2012. 2012(44): p. 20-31.

10. Taplin, S.H., M.K. Foster, and S.M. Shortell, Organizational leadership for building effective health care teams. Ann Fam Med, 2013. 11(3): p. 279-81.

Competing interests: ?? None declared