Immunization Coverage and Medicaid Managed Care in New Mexico: A Multimethod Assessment

Schillaci et al.¹ studied the temporal association between the initiation of Medicaid managed care (MMC) and changing immunization coverage in New Mexico, a relatively poor, and multiethnic state. Their research included an analysis of trends over time in vaccination coverage rates, with public used data files from the National Immunization Survey (NIS). The authors use these data to support their hypothesis that MMC in New Mexico has caused a decline in vaccination coverage rates. In their analyses, children were “4:3:1 up-to-date” if the vaccination providers of sampled children reported administering 4 or more doses of diphtheria-tetanus-acellular pertussis vaccine, 3 or more doses of polio vaccine, 1 or more doses of measles-mumps-rubella vaccine.

Valid analyses of data from the NIS require the use of the NIS survey weights and special statistical software that allows the complex design of the survey to be taken into account in determining standard errors and p-values for testing hypotheses. However, the authors did not use the NIS survey weights to estimate vaccination coverage rates and used statistical methods that did not account for the complex design of the survey. Specifically, the authors used Fisher’s exact test² and the Cochran-Armitage-trend test^3,4 to evaluate the estimated trend in vaccination coverage rates. Although these latter methods are appropriate for unweighted data obtained by simple random sampling, they were inappropriate for the analysis of data from the NIS. As a result, the conclusion the authors draw from their analysis of NIS data may be misleading. Zell et al.⁵ and Smith et al.^6,7,8 describe the design and weighting procedures for the NIS. Korn and Graubard⁹ give a recent description of statistical methods that are appropriate for the analysis of data from complex probability sample surveys, like the NIS.

The following table lists the 4:3:1 vaccination coverage estimates for New Mexico published by Schillaci et al. and the official weighted estimates published by the Centers for Disease Control and Prevention (CDC).

To evaluate whether there was a statistically significant decreasing trend in the official 4:3:1 vaccination coverage rate between 1996 and 2002, we applied a parametric bootstrap. In this context, the slope of the regression of the estimated vaccination coverage rates on the NIS survey year measures the trend. The 95% percentile confidence interval for the slope is [-3.6 , +0.91 ], and includes zero, suggesting that little support for a statistical significant non-zero trend over time. Although the 4:3:1 coverage estimates decrease over time, negative result from our analysis may be attributed to the large standard errors of the annual estimated vaccination coverage rates. Finally, Schillaci et al. have chosen to conduct an ecologic study to evaluate the correlation of trends in vaccination coverage with the implementation of MMC in New Mexico. This design rules out the possibility that rates may have decreased over time regardless of whether MMC was implemented. Among MMC-eligible children, the effect of MMC could have been more directly evaluated by comparing differences in the coverage rates trends between children receiving doses from MMC and children who did not receive doses from MMC.

References
1 Schillaci MA., Waitzkin H, Carson EA, Lopez CM, Boehm DA, Lopez LA, Mahoney, S.F. Immunization coverage and Medicaid managed care in New Mexico: a multimethod assessment. Annals of Family Medicine, Vol.2, No. 1, January/February 2004, 13-21.
2 Fisher RA. The Design of Experiments. Edinburgh: Oliver and Boyd: 1935. 3 Cochran WB. Some methods of strengthening the common X2 tests. Biometrics. 1954;10:417-451.
4 Armitage P. Tests for linear trends in proportions and frequencies. Biometrics. 1955;11:375-386.
5 Zell ER, Ezzati-Rice TM, Battaglia MP, Wrigh, RA. The National Immunization Survey: the methodology of a vaccination surveillance system. Pub Health Rep. 2000;115(1):65-77.
6 Smith PJ, Battaglia MP, Huggins VJ, et al. Overview of the sampling design and statistical methods used in the National Immunization Survey. Am J of Prev Med, 2001;20(4S):17-24.
7 Smith PJ, Rao JNK, Battaglia MP, Ezzati-Rice TM, Daniels D, Khare M. Compensating for nonresponse bias in the National Immunization Survey using response propensities. NCHS Series 2 Report. Hyattsville, MD. National Center for Health Statistics.
8 Smith PJ, Hoaglin DC, Battaglia MP, Barker LE, Khare M. Statistical Methodology of the National Immunization Survey: 1994-2002. NCHS Series 2 Report. Hyattsville, MD. National Center for Health Statistics. 2004. To appear.
9 Korn EL, Graubard BI. Analysis of Health Surveys. (1999) John Wiley and Sons: New York.

Competing interests:   None declared

In theory, one would think it would be quite straight-forward to measure the impact of Medicaid Managed Care (MMC) on childhood immunization rates. In practice, MMC represents but one part of the health delivery system, and is dependent on the local, regional, and national healthcare environments in which it exists. Thus, in inner-city Los Angeles, children in Medicaid enrolled in managed care were less likely to be up-to-date with immunizations than those in public health clinics.(1) In New York City, however, children enrolled in MMC were more likely to be up-to-date than those in non-managed care Medicaid programs.(2) In Philadelphia, children with MMC were no more or less likely to be up-to-date than their non-managed care Medicaid conterparts.(3) In sum, the evidence regarding the relationship of MMC to childhood immunization rates is decidedly mixed.(4)

In the latest issue of the Annals of Family Medicine, using an explanatory mixed methods design, Schillaci et. al.(5) examined this relationship in New Mexico, which in 1997 mandated managed care for Medicaid recipients. The authors took the novel and informative approach of applying ethnographic methods to the study of childhood immunization delivery. Through these techniques, Schillaci et. al. documented several important and concerning changes in the immunization delivery system: 1) public health clinics reported a reduction in funding which consequently limited their ability to provide timely immunizations; 2) some facilities described a increase the use of informal referrals, whereby patients who could not get prompt appointments with their MMC primary care provider were sent to public health clinics for immunization; and 3) these informal referrals increased the workload and administrative burden placed on public health clinics (who were providing immunizations but not serving as the MMC provider).

Although the ethnographic component of the Schillaci paper provides several important insights into the challenges and frustrations that can result with large system changes, the quantitative component of the paper must be interpreted with more caution. The authors’ attempt to link changes in MMC with statewide immunization rates is best described as an ecologic study. As such, it is subject to a problem termed the ecologic fallacy:(6) the qualities of a group (in this case, the statewide immunization rate) can be falsely ascribed to specific individuals within that group (in this case, MMC patients). Can changes in the statewide immunization rate be fairly linked to changes in MMC, when MMC covered only 39% of New Mexico children, and when the immunization rate of MMC compared with non-MMC patients was unknown? Additionally, there are a number of limitations which make the authors’ claim that immunization rates decreased during the study period difficult to accept:

1. Their data show immunization rates of 73% in 1994, and 73% in 2001, with fluctuations in between. The authors choose to begin their statistical analysis in 1996, a year in which New Mexico had rates of 80%. Although MMC was instituted in New Mexico in 1997, the first year in which all Medicaid children in the National Immunization Survey (NIS, the data source used in this study) were covered by MMC was 1999. In 1999, immunization rates in New Mexico were 76%, having decreased to 73% in 1998, a year in which a proportion of the immunizations in the NIS were given prior to MMC. Their own data would seem to suggest that childhood immunization rates had peaked even before the implementation of MMC. An alternative approach would be to delete those years from the analysis when the NIS contained data on both children with MMC and with traditional Medicaid.

2. The statistical evidence for a decrease in statewide immunization rates is not convincing. As mentioned above, the authors choose a year when rates were highest to begin their analysis, and when one might expect some regression to the mean. Furthermore they use a one-sided significance level, which is equivalent to a 2-sided p-value of 0.10. The 95% confidence limits around their up-to-date rates are large, ranging from 9% to 15%, and in only one year (1996) do the confidence limits not include 73%.

3. The main outcome is up-to-date status with the 4:3:1 vaccination series. However, it could be argued that the 4:3:1:3 series (which includes 3 Haemophilus influenza vaccinations) or the 4:3:1:3:3 (which includes hepatitis B vaccine) provide a more complete picture of up-to- date status, and therefore are more appropriate outcome measures to utilize. The authors do not provide data on these more complete vaccination series.

The analytic and methodologic problems described above distract from the important findings of this study. MMC was introduced in New Mexico at the same time that “the state government reduced funding and personnel for immunization activities in the New Mexico Department of Health.”(5) Prior to 1997, the year in which these changes were made, New Mexico consistently ranked between 25th and 30th in the nation in childhood immunization rates. After 1997, the state fell to 51st, and remained at 50th in 2001. Thus, although childhood immunization rates in New Mexico may not have been declining during this time, they certainly were not rising at a time when the rest of the nation was. As a result, New Mexico fell behind the rest of the country.

Despite an underlying assumption “that MMC would improve the immunization rates”(5) for New Mexican children on Medicaid, rates remained static in that state while the rest of the country surged ahead. The authors correctly point out the complexity of assessing the impact of broad policy changes in complex insurance systems. One might add that the complexity is only increased when multiple changes are made at one time. However, it seems unfounded, based on the data presented in this paper, to blame childhood under-immunization in New Mexico on MMC, when other changes in health care delivery, most notably funding cuts to public immunization programs, were instituted at the same time.

References: 1. Wood D, Halfon N, Sherbourne C, Grabowsky M. Access to infant immunizations for poor, inner-city families: what is the impact of managed care? J Health Care Poor Underserved. 1994;5:112-123. 2. Hanson KL, Fairbrother G, Kory P, Butts GC, Friedman S. The transition from Medicaid fee-for-service to managed care among private practitioners in New York City: effect on immunization and screening rates. Matern Child Health J. 1998;2:5-14. 3. Alessandrini EA, Shaw KN, Bilker WB, Schwarz DF, Bell LM. Effects of Medicaid managed care on quality: childhood immunizations. Pediatrics. 2001;107:1335-1342. 4. Szilagyi PG. Medicaid managed care and childhood immunization delivery. J Public Health Manag Pract. 1998;4:67-72. 5. Schillaci MA, Waitzkin H, Carson EA, Lopez CM, Boehm DA, Lopez LA, Mahoney SF. Immunization coverage and Medicaid managed care in New Mexico: a multimethod assessment. Annal Family Medicine. 2004;2:13-21. 6. Gordis L. Epidemiology. Philadelphia, Pennsylvania: W. B. Saunders Company; 1996: p. 169.

Competing interests:   None declared