If You’ve Seen One Quality Improvement Collaborative …

• Quality assurance, health care
• breakthrough series

In this issue, Schonlau et al describe the results of a RAND study of a national quality improvement collaborative (QIC) formed to improve asthma care using the Breakthrough Series (BTS) approach of the Institute for Healthcare Improvement.¹ Although the study was well designed, and they found greater improvements in some chart audit-based process measures of care, enthusiasts for this approach to quality improvement can hardly take solace from this study. The only process measures with greater improvement in the intervention arm were those with initially extraordinarily low performance rates (0% to 7%); the small difference in performance change compared with the control group (mean of 16%) could be accounted for by awareness of the intervention intent and better documentation, and no measure has a strong evidence-based link to outcomes. Patient surveys were conducted, but no preintervention baseline measurement was possible, so the higher rates in the intervention arm of 1 of 5 process measures and of 1 of 5 “outcome” measures (satisfaction with clinician and lay educator communication) cannot be interpreted and are of limited value.

Thus, overall the study perpetuates the dilemma that Mittman described in his editorial last year about the need for an evidence base for QICs.² Since then, at least 1 much larger randomized controlled trial of a national QIC among neonatal intensive care units appeared to show better results in 2 process of care measures, but outcomes were not improved.³ A careful reading of that trial probably does not change the continuing validity of Mittman’s conclusion that “its [the QIC method] overall effectiveness remains highly uncertain but is probably modest,” but only for that particular type of QIC, which is national, selective, and focused on a single topic.²

Despite the above comments and my own negative trials of the QIC approach, I continue to believe this approach has greater potential once we can clarify what types of QICs are useful and what they are useful for.^4,5 Although both Mittman and Ovretveit et al (in an earlier commentary about QICs) allude to various types, they both focus nearly entirely on the BTS-type approach to a QIC, as do the few published studies.^2,6,7 There are other types of collaboration for quality improvement, and it may be even more important that we understand and evaluate them. Although the methods used in the BTS may be good, this type of QIC is hampered by focusing on specific time-limited process improvement projects, with participants drawn from all over the nation, and at a cost feasible only for large organizations or government agencies. These features limit any potential for a QIC effect on the larger local environments, any benefit from ongoing collaboration, or much chance to get local leadership seriously involved. Without involved leadership, the organizational changes likely to be more important and long-lasting than a process-focused improvement effort will not occur, so all that happens are some targeted efforts led by enthusiastic champions without much authority.

There are other approaches to collaborative quality improvement. The one with which I am most familiar is that of the Institute for Clinical Systems Improvement (ICSI).⁸ ICSI began in 1993 sponsored by 1 health plan and a business group, but it became a regional institution in 2001 with support from all 6 health plans in the area. It has always been run collaboratively among medical group members (who have dominated the board), and it currently counts 75% of the physicians and most medical groups and hospitals in the area among its members. Although ICSI does run specific project-type quality improvement groups using variations on the BTS approach, these groups are a part of an overall collaborative improvement effort that includes the following:

1. Building guidelines that are now widely accepted as the standard of care in the state

2. Providing training and materials in quality improvement and organizational change methods and topics

3. Requiring ongoing leadership involvement and commitment from its member organizations

4. Facilitating organizational culture changes

5. Providing a conceptual framework called “Cultivating Quality,” which includes an understanding of the synergistic importance of commitment, collaboration, climate, and culture

6. Initiating quality improvement efforts that involve most of the region’s stakeholders projects

7. Advocating and facilitating environmental changes that support data-based public accountability, incentives, and collaboration among health plans, clinicians, purchasers, and the public

The apparent result has been heightened interest in and support for quality improvement in the entire state, measurable widespread improvements in quality measures, a new community-wide standardized comparative measurement system that went public in 2004, and a reorientation toward evidence-based medicine and systems thinking.

Other examples of QICs are broadly based regionally. The Rochester Health Commission has been operating since 1995 to “help stakeholders reach consensus on actions that are needed to continuously improve the Rochester health care system.”⁸ Its focus has been to spin off a variety of improvement collaborative initiatives that report back to the community. Since 1999 the Pittsburgh Regional Healthcare Initiative (PRHI) brought together a variety of stakeholders to work on improvement efforts in selected areas, publicizing them and their results.⁸ The Wisconsin Collaborative for Healthcare Quality has recently developed as a partnership between medical groups, hospitals, health plans, and business to provide a public performance reporting system.⁹ The Veteran’s Affairs System developed its own intraorganizational QICs for specific topic areas that bring together researchers, quality improvement leaders, and organizational leaders to work on learning and applying the best science in change management and in medical care. There are other variations as well.

Each example has in common collaboration among organizations or parts of a national organization to improve the overall quality of medical care by using varying combinations of comparative performance measurements and targeted improvement initiatives. They differ from the BTS approach in that they are in it for the long term, their efforts are led and supported by the leaders of individual member organizations, and they have a variety of ongoing communication and meeting efforts that go beyond particular quality initiative. As such, I believe that they have much greater potential to change the attitudes, priorities, and support for quality in an entire region and to lead to care transformation. Efforts to evaluate such QICs must therefore include those more subjective but important influences on care quality, not just particular changes in a few measures.

To understand the value of collaborative efforts at quality improvement, we need to study different types of QICs and a variety of evaluation methods that go well beyond whether a particular initiative resulted in short-term improvements in care process or outcomes. For example, it is hard to interpret negative results of a trial without knowing whether the intervention tested was used and to what extent. Similarly, it is important to understand what happened after a particular initiative ended. Were improvements sustained and spread? Did the experience change clinician organizations in ways that led to delayed benefits or losses? What was the role of performance reporting or other stimuli? We also need qualitative case studies of collaborative initiatives and change efforts within individual clinician groups that are affected by a QIC. Such studies can provide important new insights and hypotheses. Finally, we need to study the effects of a QIC on the local environment for health care.

We are clearly only beginning to learn how to transform health care, and how to study transformation efforts. It is too early to say whether collaborative improvement efforts will be important and which variations on those collaborations might be most effective. We do need more evidence and more creative studies, but those goals will not be facilitated by thinking of QICs as homogeneous. At this stage in their development, it may be more useful to realize that when you’ve seen one QIC, you’ve seen one QIC.

• Received for publication March 3, 2005.
• Accepted for publication March 7, 2005.

• © 2005 Annals of Family Medicine, Inc.