Abstract
PURPOSE Scale-up and spread of evidence-based practices is one of the most important challenges facing health care. We tested whether a statewide initiative, Depression Improvement Across Minnesota–Offering a New Direction (DIAMOND), to implement the collaborative care model for depression in 75 primary care clinics resulted in patient outcome improvements corresponding to those reported in randomized controlled trials.
METHODS Health plans provided a new monthly payment to participating clinics after a 6-month intensive training program with ongoing data submission, networking, and consultation. Implementation was staggered, with 5 sequences of 10 to 40 clinics every 6 months. Payers provided weekly contact information for members from participating clinics who were filling antidepressant prescriptions, and we conducted baseline and 6-month surveys of 1,578 patients about their care and outcomes.
RESULTS There were 466 patients in DIAMOND clinics who received usual care before implementation (UCB), 559 who received usual care in DIAMOND clinics after implementation (UCA), 245 who received DIAMOND care after implementation (DCA), and 308 who received usual care in comparison clinics (UC). Patients who received DIAMOND care after implementation reported more collaborative care depression services than the 3 comparison groups (10.9 vs 6.4–6.7, on a scale of 0 of 14, where higher numbers indicate more services; P <.001) and more satisfaction with their care (4.0 vs 3.4 on a scale 1 to 5, in which higher scores indicate higher satisfaction; P ≤.001). Depression remission rates, however, were not significantly different among the 4 groups (36.4% DCA vs 35.8% UCB, 35.0% UCA, 33.9% UC; P = .94).
CONCLUSIONS Despite the incentive of a supporting payment change and intensive training and support for clinics volunteering to participate, no difference in depression outcomes was documented. Specific unmeasured actions present in trials but not present in these clinics may be critical for successful outcome improvement.
INTRODUCTION
Despite 25 years of extensive efforts to improve care quality and extensive research efforts to identify the best methods for widespread implementation of evidence-based practices, we still have limited knowledge about how to facilitate such spread. One of the most striking examples of a gap between evidence and practice in need of such spread is in primary care for depression. At least 79 randomized controlled trials of the collaborative care model for depression have found improved patient outcomes, and there is evidence that the model is cost-effective and even cost-saving.1–8 As a result, one might expect widespread implementation of collaborative care, but even care systems and clinics that participated in randomized trials have not usually continued the approach.9,10
A key barrier to implementation of collaborative care is lack of reimbursement for the model’s components. The model also requires major changes in traditional primary care and mental health practice, because it is built on a multidisciplinary team approach with a primary care physician, care manager, and consulting psychiatrist. When a multistakeholder quality improvement collaborative in Minnesota, the Institute for Clinical Systems Improvement (ICSI), developed a statewide initiative to overcome these implementation barriers by providing a new payment approach and the training, tools, and facilitation of practice change, it provided an important opportunity to learn about the potential for large scale uptake and spread.11 The initiative was called DIAMOND (Depression Improvement Across Minnesota–Offering a New Direction).
This study was developed to answer the questions of local and national stakeholders, especially regarding initiative effects on patient outcomes. A staggered implementation (stepped-wedge) design served the needs of both the initiative for manageable implementation and the evaluation for a comparison group.
METHODS
Intervention
The DIAMOND initiative steering committee adapted the version of the collaborative care model tested in the largest randomized trial—the IMPACT (Improving Mood: Promoting Access to Collaborative Treatment) study.6,12 This model included 7 components: (1) consistent use of the 9-item Patient Health Questionnaire (PHQ-9) depression scale for monitoring depression severity13,14; (2) systematic patient follow-up tracking and monitoring; (3) treatment intensification for patients not improving; (4) relapse prevention planning for patients achieving remission; (5) on-site care manager for educating, monitoring, and coordinating care; (6) scheduled weekly caseload review with a consulting psychiatrist; and (7) monthly descriptive data submissions.
Primary care clinics were recruited by ICSI staff from the medical group members of ICSI based on their interest and readiness. Eighty clinics signed up for 1 of 5 sequences of a potential 280 clinics. Additional clinics joined later but others dropped out, so only 75 clinics completed training, implemented the DIAMOND care model, and provided DIAMOND care for at least several years. More than 10,000 patients were enrolled in the initiative from March 2008 through May 2013.
Training was provided to clinic leaders, clinicians, and staff during the 6 months just before implementation, and a version of the care management tracking system used in the IMPACT study was used by 54% of participating clinics.15 The remainder set up their own electronic medical record-based versions. After implementation, ICSI staff conducted networking conference calls and individual site consultation.16,17 Clinics could tailor their approach as long as they provided the model components described above.
The other major component of the DIAMOND intervention was new financial coverage for this care by all large commercial payers in the state for enrolled adult patients (PHQ-9 scores greater than 9, and specific codes for major depression or dysthymia). Payment amounts and arrangements varied by individual payer contracts with participating clinics, usually in a monthly bundled fee. Unfortunately, one-third of enrolled patients were covered by Medicaid or Medicare, neither of which provided payments for DIAMOND care.
Research Design
The study used a stepped-wedge design, with new patients recruited weekly for 30 months at all sites.18–22 Key outcome measures of care received, depression severity, work productivity, health status, and satisfaction were derived from patient surveys at baseline and 6 months later.
Participants
To standardize patient recruitment across clinics while minimizing impact on care, all but 1 commercial payer sent weekly lists of members recently initiating antidepressant medications in a participating clinic. These lists were supplemented with lists of patients billed for DIAMOND care. Patients were then called to assess study eligibility and willingness to participate. The recruitment process and comparison of participants and nonparticipants showed only small differences between patients in the sampling frame and enrolled study participants.18 The study was reviewed, approved, and monitored by the relevant institutional review boards.
Patient Survey
This survey assessed the main patient outcomes at enrollment and again 6 months later, using the PHQ-9 measure of depression severity,23,24 the Work Productivity and Activity Impairment (WPAI),25 the World Health Organization question about health status,26 a modified version of the Patient Assessment of Chronic Illness Care (PACIC),27 and a question about satisfaction.
PHQ-9 scores (range, 0 to 27, with higher scores indicating greater depression severity) measured baseline and 6-month depression severity. Patients were classified as improved if the 6-month PHQ-9 score was 50% lower than baseline, and as in remission if the 6-month PHQ-9 score was less than 5.
The WPAI is a self-report measure of absence from work because of health problems, as well as of productivity impairment while at work. It was administered to patients who reported working for pay at least part-time. The proportion of expected work time that was affected by health problems was calculated as work time missed plus time at work multiplied by impairment. Functional health status and satisfaction with care were each assessed by single items (range of 1 to 5, with higher scores indicating higher status and greater satisfaction).
The PACIC was revised with its developer so questions reflected collaborative care of depression.27–29 The revision reduced the original 20 questions to 14 reporting on the number of care elements received in the previous 6 months (scores range from 0 to 14).
Predictors of Patient Outcomes
Patients were placed into 1 of 4 treatment groups characterized by their clinic’s involvement, as well as their own enrollment in DIAMOND care. Payer utilization data showing DIAMOND care charges and patient self-report identified those who had received DIAMOND care. The baseline survey date was used to determine whether patients were participating in the study before or after clinic implementation of the DIAMOND model. The resulting 4 treatment groups consisted of patients who received (1) usual care in DIAMOND care clinics before implementation (UCB), (2) usual care in DIAMOND clinics after implementation (UCA), (3) DIAMOND care after implementation (DCA), and (4) usual care in clinics that planned to but never implemented DIAMOND care (UC).
The effectiveness of DIAMOND care was gauged by comparing patient outcomes among DCA patients relative to the 3 comparison groups.
The timing of each patient’s outcomes was calculated as the number of months that had elapsed between the date the first sequence of clinics implemented DIAMOND (March 2008) and the patient’s baseline survey date. This secular trend covariate was included in the patient outcome analyses to quantify community-wide changes in the outcomes of depression treatment that were independent of those attributable to the DIAMOND model.
Analyses
Patient-Reported Care Processes and Outcomes
The likelihoods of depression response and remission at 6 months were predicted in generalized linear mixed models (binomial distribution, logit link) from treatment group, secular trend, and, if significant, the treatment group by secular trend interaction. Treatment group and secular trend were fixed effects, patients were nested within primary care clinics, a random clinic intercept was estimated, fixed-effects standard errors were calculated using empirical sandwich estimation, and each observation was weighted by the multiplicative product of inverse probability weights to correct for the likelihoods of 6-month survey response and of receiving DIAMOND care. Model-estimated likelihoods are presented. Simple effects tests that compared model-predicted, 6-month outcomes of each comparison group with DCA guided interpretation of significant treatment effects.
Similarly, structured analyses assessed whether baseline to 6-month changes in depression symptoms, care processes, productivity loss, functional health status, and satisfaction with care (normal distribution, identity link) differed by treatment. Repeated outcomes were predicted from fixed effects of treatment group, outcome timing (baseline, 6 months), treatment by timing, secular trend, and, if significant, secular trend by timing and by treatment. The model structure differed from the 6-month outcome models in that repeated outcomes were nested within patients, and a random patient intercept was estimated. Simple effects tests by treatment group and by timing aided interpretation.
Patient Survey Retention and Treatment Selection
We assessed response bias in key survey outcomes by comparing patient characteristics of 6-month survey respondents (n = 1,578) with those of nonrespondents (n = 770). Characteristics that were associated with response status at P <.20 were considered for inclusion in a non–parsimonious response propensity model. The logistic model nested patients within clinics, predicted 6-month response from baseline characteristics, estimated a random clinic intercept, and used empirical sandwich estimation to calculate fixed-effects standard errors. Characteristics that predicted response at P <.10, did not diminish goodness of fit, and were not collinear with other characteristics were retained. Final model response propensities corresponded to a response likelihood M = 0.68 (Mrespondent = 0.70, Mnonrespondent = 0.63). Higher response propensities were associated with higher likelihoods of depression response (P <.005) and remission (P <.002), lower PHQ-9 scores (P < .001) and WPAI scores (P = .02), and higher functional health status (P <.001), all at 6 months. Because response propensity was related to survey response and to key study outcomes, we attempted to ameliorate potential response bias by calculating stabilized inverse probability weights (M = 1.04; Mrespondent = 1.00, Mnonrespondent = 1.14) for the patient outcome analyses.
Although UCB or UC patients could not receive DIAMOND care, patients treated after DIAMOND was implemented in their clinics could be in the UCA or DCA group. We also used a propensity approach to assess DIAMOND care treatment selection. Characteristics associated with DCA group membership were included in a non–parsimonious propensity model to calculate stabilized inverse probability weights for UCA and DCA patients. The resulting weights were MDCA = 2.31 and MUCA = 0.94, and set to 1.00 for UCB and UC patients.
All analyses were conducted using SAS 9.2 software (SAS Institute) with P values <.05 deemed to be statistically significant.
RESULTS
Participation
Of the 24,065 names received from payers during the 3 years, 5,080 had incorrect contact information and 4,509 could not be reached within the 21-day window required to ensure baseline assessment before major effects from treatment. The participation rate of those contacted and potentially eligible was 36.4%. (Figure 1 displays further details about recruitment.) Sixty-seven percent of patients completed the 6-month survey with similar retention rates across the study groups: UCB = 67%; UCA = 65%; DCA = 72%; UC = 68%. The characteristics of enrolled patients by treatment group are displayed in Table 1. The UCA and DCA patients differed on baseline characteristics, suggesting that patients with more severe depression may have been referred to DIAMOND care. Additional information about the frequency of various comorbidities among the patients in each study group is contained in the Supplemental Table 1 (available at http://www.annfammed.org/content/13/5/412/suppl/DC1), although none of these differed significantly among the 4 groups.
Study patient flow diagram.
DIAMOND = Depression Improvement Across Minnesota-Offering a New Direction; UCB = usual care before implementation; UCA = usual care in DIAMOND clinic after implementation; DCA = DIAMOND care after implementation; UC = usual care in comparison clinics.
Baseline Survey Patient Characteristics by Treatment Group
Clinic characteristics were obtained from all 75 clinics (Table 2). One-half of the clinics were from outside the metropolitan Twin Cities region, 39% had fewer than 6 primary care physicians, and 95% were part of medical groups containing at least 3 clinics.
Clinic Characteristics (n = 75)
Patient-Reported Depression Care Process and Outcomes
Table 3 displays patient reports of DIAMOND processes and response and remission rates for the 4 patient groups. Although patients in the DCA group reported receiving 67% more of the care processes that were part of the DIAMOND initiative than patients in any comparison group (P <.001), their 6-month response and remission rates were not statistically different. Baseline PHQ-9 scores were higher among DCA patients than among comparison groups (P <.002 vs UCB; P <.01 vs UC; P <.0001 vs UCA), but the slightly larger drop in PHQ-9 scores among DCA patients did not reach statistical significance (P = .63). Only 4 medical groups had enough patients across all of their clinics to permit statistical assessment of their outcomes by medical group, and none attained statistical significance. Informal nonquantitative review did not suggest that there were any unintended adverse outcomes. Additional information about the frequency by study group with which specific care processes were reported is available in Supplemental Table 2 (http://www.annfammed.org/content/13/5/412/suppl/DC1), and the relationship between care processes and outcomes can be found in Supplemental Table 3 (http://www.annfammed.org/content/13/5/412/suppl/DC1).
Model-Predicted Depression Care Process and Outcome Measures, Adjusted for Secular Trend and Weighted by Survey Response and Treatment Selection Likelihoods
Patient-Reported Work Productivity, Health Status, and Satisfaction
Among the 59% of patients who reported working outside the home at the 6-month survey, the mean productivity loss of 27.6% (combined absenteeism and presenteeism) was less than at baseline (37.7%, P <.001), but there was no difference in rates of decline across treatment groups (Table 4). There was also no difference in health status at either time among the 4 treatment groups, although among each, health status improved from baseline (M = 2.99) to 6 months (M = 3.13) by about 5% (P = <.001). Finally, care satisfaction ratings by DCA patients were significantly higher than the comparison groups at both baseline and at 6 months (P <.001), and satisfaction ratings increased more among DIAMOND care patients than comparison groups (P <.05).
Model-Predicted Secondary Outcomes, Adjusted for Secular Trend and Weighted by Survey Response and Treatment Selection Likelihoods
DISCUSSION
This unique large-scale initiative to spread the collaborative care DIAMOND model for depression appears to have improved patient satisfaction, but it had little impact on other patient outcomes. Despite good evidence of implementation of practice systems important to collaborative care, and despite enrolled patients reporting receiving more desired care processes, patients receiving DIAMOND care had neither better depression outcomes nor better improvement in work productivity or health status.
These results were surprising and disappointing to the participants in the DIAMOND initiative, especially because measurements by the initiative from clinic data submissions had shown 6-month response and remission rates of 40% and 30% among all patients enrolled, and 66% and 48% among those 61% that could be measured again at 6 months, rates that are almost as good as those seen in intervention groups in the scientific trials. In the absence of any type of comparison group, however, initiative leaders had no way of knowing that usual care among these clinics was as good as it was.30
Despite our measures of system implementation and care provision, we are unable to definitively identify the reason the groups did not differ. One likely factor is the high quality of usual depression care in Minnesota, making it harder to further improve. Health Employer Data Information Set scores for all 3 of the main health plans in Minnesota for depression medication management show each in the top decile for both measures.31 Moreover, the DIAMOND clinics were all members of ICSI and had gained experience in quality improvement methods. In addition, the up-to-4-week delay in obtaining study baseline PHQ-9 scores may have reduced baseline scores and therefore the potential for improvement.
As previously reported, our measures of implementation suggest that most practice systems needed to provide collaborative care for depression were in place in these clinics at 1 year and were sustained at 2 years (although there was considerable variation).32 It is possible that some particularly crucial components of the intervention were either inadequately measured by our measures or were diluted by other less important components. We suspect that a particularly important component is the extent to which treatment-to-target and treatment intensification by changing medications or adding psychotherapy occurs for patients who are not improving. Treatment optimization is dependent on how consistently such cases are followed, monitored, and highlighted during systematic case reviews with the consulting psychiatrist, how often those reviews result in recommendations for treatment changes, and how often these recommendations are then fully carried out. Our measurement systems were not specific enough to provide such information.
One large medical group had few patients in our study sample because its main insurer declined participation. This group separately analyzed their data and found both more medication changes and better depression outcomes among DIAMOND care patients than in nonrecipients.33,34 This group was one of the few with internal psychiatric leaders very actively involved in the initiative committees, so they may have ensured more active treatment intensification. It is also possible that other unadjusted biases between patient groups confounded differences, but the consistency in results across the 3 different comparison groups makes such bias less likely.
At least 10 systematic reviews and/or meta-analyses of the 79 randomized controlled trials of collaborative care for depression have been published.1,3,4,8,35–40 They have almost all concluded that the model produces better outcomes, but the following reports have tried to identify the individual components of the model that were most important: staff assistance with case management and mental health specialist involvement3; revision of professional roles and provision of a care manager who delivers psychotherapy (not part of this initiative)37; patient education and self-management, symptom monitoring, decision support for treatment adherence, patient registries, and mental health supervision of care managers8; professional background and method of supervision of care managers1; systematic follow-up and service restructuring35; and ensuring adequate doses of antidepressants.33 Although most of these strategies were recommended to participating clinics, we lack detailed measures of the extent to which they were implemented.
This study shows the difficulties of widespread implementation of evidence-based practices that require major changes in roles and extensive financial and leadership support. In the clinical trials that provided the evidence for collaborative care, support for these major changes was usually provided by the research team, so those attempting to replicate trial results may have to replicate those supports. That this initiative was unable to produce results similar to clinical trials with volunteer clinics, despite the guidance and support of a skilled and experienced quality improvement collaborative and payment changes, highlights how difficult it really is to implement evidence-based care effectively.
Footnotes
Conflicts of interest: authors report none.
Supplementary materials: Available at http://www.AnnFamMed.org/content/13/5/412/suppl/DC1.
- Received for publication March 19, 2015.
- Revision received June 30, 2015.
- Accepted for publication July 7, 2015.
- © 2015 Annals of Family Medicine, Inc.
References
In this issue
Jump to section
Related Articles
Cited By...
- Evaluating sustained reach and effectiveness of collaborative care models: A Cross-sectional study of the New York State Collaborative Care Medicaid Program
- Challenges in conducting research on collaborative mental health care: a qualitative study
- Large-Scale Implementation of Collaborative Care Management for Depression and Diabetes and/or Cardiovascular Disease
- In This Issue: Generalist Care Around the World