PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of hsresearchLink to Publisher's site
 
Health Serv Res. Aug 2008; 43(4): 1325–1347.
PMCID: PMC2517275
Multimethod Evaluation of Health Policy Change: An Application to Medicaid Managed Care in a Rural State
Howard Waitzkin, Michael Schillaci, and Cathleen E Willging
Address correspondence to Howard Waitzkin, M.D., Ph.D., Distinguished Professor, Departments of Sociology, Family & Community Medicine, and Internal Medicine, University of New Mexico, MSC 05 3080, 1070 Social Sciences Building, 1915 Roma NE, Room 1103, Albuquerque, NM 87131-0001; e-mail: waitzkin/at/unm.edu. Michael Schillaci, Ph.D., is Assistant Professor with the Department of Social Sciences, University of Toronto Scarborough, Toronto, Ontario, Canada. Cathleen E. Willging, Ph.D., is the Research Scientist and Program Director with the Behavioral Health Research Center of the Southwest, Pacific Institute for Research and Evaluation, Albuquerque, NM.
Objective
To answer questions about the impacts of Medicaid managed care (MMC) at the individual, organizational/community, and population levels of analysis.
Data Sources/Study Setting
Multimethod approach to study MMC in New Mexico, a rural state with challenging access barriers.
Study Design
Individual level: surveys to assess barriers to care, access, utilization, and satisfaction. Organizational/community level: ethnography to determine changes experienced by safety net institutions and local communities. Population level: analysis of secondary databases to examine trends in preventable adverse sentinel events.
Data Collection/Extraction Methods
Survey: multivariate statistical methods, including factor analysis and logistic regression. Ethnography: iterative coding and triangulation to assess documents, field observations, and in-depth interviews. Secondary databases: plots of sentinel events over time.
Principal Findings
The survey component revealed no consistent changes after MMC, relatively favorable experiences for Medicaid patients, and persisting access barriers for the uninsured. In the ethnographic component, safety net institutions experienced increased workload and financial stress; mental health services declined sharply. Immunization rate, as an important sentinel event, deteriorated.
Conclusions
MMC exerted greater effects on safety net providers than on individuals and did not address problems of the uninsured. A multimethod approach can facilitate evaluation of change in health policy.
Keywords: Medicaid managed care, safety net, multimethod research, access, ethnography
In health services research, the problem of how to evaluate the results of policy change has remained challenging (Evaluation Gap Working Group 2006). While the Centers for Medicare and Medicaid Services mandates that state governments evaluate changes in their Medicaid programs, these evaluations have proven limited, typically confirming budget neutrality and finding no adverse effects (Chang et al. 2003). More comprehensive evaluations of policy changes in Medicaid, which address in more depth issues such as utilization, costs, quality, access, or satisfaction, have occurred rarely (e.g., Freund et al. 1989).
We report here a multimethod assessment of Medicaid reform within a rural state. The study addressed research questions at three levels of analysis. (a) Sequential surveys answered questions about the impacts of Medicaid managed care (MMC) for individuals. (b) Ethnographic methods assessed what effects occurred for safety net institutions and local communities. (c) Analysis of publicly available secondary databases determined how sentinel events changed as population-level indicators.
By the end of the 1990s, most states required some or all Medicaid recipients to enroll in managed care. Regarding our interest in questions about individual-level impacts, some MMC programs led to improved or unchanged access for Medicaid recipients, while other programs manifested enduring access barriers (Baker and Afendulis 2005; Berman, Armon, and Todd 2005; Fairbrother et al. 2005; Sommers 2005; Seligman et al. 2007); MMC exerted no consistent effects on access for the uninsured (Haberer, Garrett, and Baker 2005). Utilization under MMC proved inconsistent, even involving services targeted for reduction such as emergency department visits (Alessandrini et al. 2001b; Garrett, Davidoff, and Yemane 2003; Davidoff et al. 2007).
Research has generated contradictory findings about other individual-level outcomes (Waitzkin 2001). Some MMC programs generated wide satisfaction among users (Roohan et al. 2003). Others showed dissatisfaction, especially for minorities and/or those with language barriers (Weech-Maldonado et al. 2003; Valet et al. 2004). Regarding quality of care, national data revealed shortcomings under MMC provided by commercial health plans (Thompson et al. 2003), while studies of specific MMC medical and/or mental health programs produced either favorable (Bouchery and Harwood 2003), adverse (Busch, Frank, and Lehman 2004), or neutral (Ridgely and Maglione 2006) findings. Although most studies showed moderating effects of MMC on costs (Shireman et al. 2002; Long, Coughlin, and King 2005), others revealed little change in costs for specific subgroups (Domino and Salkever 2003).
Regarding our concern about research questions at the organizational/community level, while some safety net institutions survived and even flourished under MMC, others encountered major crises (Sparer et al. 2002; Bindman, Yoon, and Grumbach 2003; Felland et al. 2003; Jacobson et al. 2005; Zuvekas 2005). Some but not all states included rural areas in MMC (Slifkin et al. 1998). In rural areas, the theoretical advantages of market competition did not necessarily apply (Kronick et al. 1993).
Surveillance of population-level sentinel events under MMC has occurred rarely. Some prior studies have assessed interventions to improve immunizations under Medicaid (Vivier et al. 2000; Alessandrini et al. 2001a; Kirschke et al 2004). Other sentinel events pertinent to population-level research questions have received little attention.
New Mexico manifests high levels of poverty, uninsurance, minority ethnicity, and limited physician capacity (U.S. Census Bureau 2004, 2005; Morgan 2005; U.S. Health Resources and Services Administration 2006). The state recently ranked 47th among the 50 states in personal income per capita ($24,291), third in persons below the poverty level (18.4 percent), second in lack of health insurance (22.1. percent), and first to 11th in unemployment. Drug- and alcohol-induced death rates per capita ranked first and second highest, respectively; suicide death rate ranked fifth; and homicide death rate ranked sixth. In a population of 1.8 million, approximately 21 percent held Medicaid coverage. Hispanics and American Indians comprised 51.6 percent of the state's population. Of 33 counties, 32 contained federally designated Health Professional Shortage Areas and/or Medically Underserved Areas.
Until 1997, New Mexico maintained a Medicaid system based on fee-for-service. Public and private providers received specified fees for Medicaid patients. The Medicaid program did not require selection of a primary care practitioner (PCP). Safety net providers, especially community health centers (CHCs) and public hospitals, could use Medicaid revenues to subsidize services for uninsured patients. The state government provided preventive services such as immunizations through its Department of Health.
During July 1997, New Mexico initiated mandatory MMC (named “Salud!”) for nearly all Medicaid recipients. At 3-month intervals, the state government implemented MMC first in urban counties, next in suburban counties, then in rural counties, and finally in “frontier” counties. The state approved three private, for-profit managed care organizations (MCOs) to provide MMC services. These MCOs either offered services directly or subcontracted with other providers, including CHCs, public hospitals, and private practitioners' offices. The new system required a PCP for each Medicaid patient. Because MCOs were to provide preventive services, the state reduced services previously provided by the Department of Health (for instance, immunizations). To deliver mental health services, the MCOs introduced new administrative layers: MCOs contracted with behavioral health organizations (BHOs), which subcontracted with regional care coordinators (RCCs), which in turn subcontracted with service providers.
Before New Mexico's MMC program began, leaders of safety net organizations asked our research team to study the reform. We received transitional funding from the state government and the state's largest public hospital to support the work. Later, we obtained federal funding to expand the research throughout the state. By planning the study so that we could detect favorable, neutral, and unfavorable impacts of MMC, we hoped our evaluation would provide information that program managers could use to improve the MMC program.
Conceptually, we considered specific research questions about the effects of policy change at three levels of analysis. At the individual level, our research question was: What were the effects of MMC on barriers to care, access, utilization, and satisfaction? As a result of gate-keeping and administrative review, we expected that individual utilization of services might change. Because of the bureaucratic procedures, we expected that MMC might adversely affect patients' satisfaction; alternatively, satisfaction might improve with an assigned PCP who could serve as a “medical home.” Because enrollment in Medicaid shifted frequently (for instance, as people became employed or unemployed), we decided to study not only the population covered by Medicaid but also those with other types of insurance and the uninsured.
At the organizational/community level, we asked: What changes took place in workload, processes of supervision, visions of service ethics, financial stress, and provider accountability at safety net institutions and in the communities that the institutions served ? As control over Medicaid procedures reverted to a small number of MCOs, safety net institutions might experience reduced revenues and difficulties in maintaining services for the uninsured. Alternatively, systematic scrutiny by MCOs might improve accountability and quality.
At the level of the population, we asked: What trends occurred in patterns of sentinel health events, linked to MMC reform. For instance, changing access might affect rates of prenatal care, hospitalizations sensitive to outpatient care, cancers subject to early detection, immunizations, and diseases preventable by vaccines.
We implemented three methods to address the research questions. Figure 1 summarizes the study's components. Several approaches in each component reduced the likelihood of bias: highly structured, validated instruments, and standardized, quantitative data analysis (survey); a variety of data sources, data collection and analysis involving multiple observers, and an iterative coding and interpretation process (ethnography); and application of standardized statistical methodology to databases whose public accessibility permitted reassessment by other analysts (sentinel events).
Figure 1
Figure 1
Conceptual and Methodological Model of the Effects of Medicaid Managed Care Reform.
Survey Component
Survey Waves, Respondents, Sample Size, and Response Rates
We conducted telephone surveys of randomly sampled low-income households. The surveys were sequential and cross-sectional; each wave involved a separate sampling of the target population (we did not follow a cohort of subjects mainly due to frequent changes in insurance status). Although the state government did not initiate a baseline study before MMC and we could not accomplish such a study due to resource limitations, we conducted surveys shortly after implementation (at 9 months in two representative counties and at 18 and 27 months statewide), when we asked respondents to rate their experiences before and after MMC. To obtain information about children (if any), the interviewee answered questions about one designated child with the most recent birthday.
With α of 0.05 and 80 percent power, the sample size to detect a 5 percent difference from a baseline prevalence of 0.40 was 1,612. For lower baseline prevalences, or larger differences, the sample sizes were smaller. Six hundred fifty-seven persons participated in the smaller, 9-month survey, while 1,819 completed interviews in the 18-month statewide survey. Due to larger than expected differences in the 18-month survey and unexpectedly higher survey costs, the 27-month survey included 1,216 respondents. In the three waves, respondents gave information, respectively, about 267, 926, and 603 children in their households. The three waves showed response rates of 68.0, 82.8, and 76.4 percent.
Survey Items
We employed items from commonly used surveys, such as the Behavioral Risk Factor Surveillance System, the Consumer Assessment of Health Plans Study, and SF-36 measure of health status. Spanish-speaking team members translated the instrument into Spanish, different team members back translated into English, and the team resolved discrepancies through discussion.
Data Analysis
Bivariate analyses compared respondents by insurance category, demographic characteristics, and experiences before and after MMC. In each wave, pertinent dependent variables were entered into principal components factor analysis. The item loading highest on each factor was used as the dependent variable in logistic regression analysis. We conducted most analyses with SAS software (Cary, NC). SUDAAN software (Research Triangle Park, NC) provided weights to adjust findings as appropriate to population characteristics.
Ethnographic Component
Field Observations and Interviews
Ethnography is a qualitative method, usually involving intensive field observations and in-depth interviews. In health services research, ethnography can help document intended and unintended effects of government policies on institutions and communities (Rylko-Bauer and Farmer 2002; Singer and Castro 2004).
Graduate students conducted interviews at Income Support Division (ISD)—“welfare”—offices, CHCs, emergency departments, private PCPs' offices, and mental health institutions. Observations also took place at community meetings and public forums. At each of 11 health care sites, ethnographers conducted field observations, performed structured interviews, and collected pertinent documents, at baseline and again at 9, 18, and 27 months after MMC implementation. Postdoctoral fellows monitored numbers of interviews, interview response forms, and field notes to enhance quality control. The anthropology professor who directed this component met with the team every 2 weeks to oversee data collection and assure quality. Ethnographers completed 15–20 interviews at each site during each phase with approximately equal numbers of administrators, professionals, non-professional workers, and clients. Over the course of the 4-year study, the ethnographers conducted approximately 900 interviews.
Data Analysis
Ethnographic data were analyzed through iterative codings, which included terms such as PCP, referrals, workload, and financial impact. Through joint discussions, the ethnographic team then developed a coded index of topics (Emerson, Fretz, and Shaw 1996). A software package, ATLAS.ti, facilitated data analysis. Triangulation (i.e., comparing observations, interviews, and documents) helped clarify themes, concerns, and meanings that emerged repeatedly. Separate articles provides a more complete description of our ethnographic data collection, analysis, and interpretation process (Horton 2004; Lamphere 2005; Willging, Waitzkin, and Wagner 2005).
Sentinel Events Component
Data Sources
We employed data sets available at no or minimal cost in the public sphere. Adverse sentinel events included rates of hospitalization for conditions preventable by outpatient care (Bindman et al. 2005; Basu, Friedman, and Burstin 2006), mortality from cancers subject to early detection, prenatal care, immunization coverage levels, and incidence of vaccine-preventable diseases. We used several databases: (1) birth and death certificates, (2) hospital inpatient discharge data, (3) the New Mexico Tumor Registry, (4) the National Immunization Survey, and (5) data on reportable diseases from the State Department of Health.
Data Analysis
We generated bivariate plots of sentinel events over time. Formal statistical comparisons of data before and after MMC used Fisher's Exact Test for differences or ratios of rates and the Cochran–Armitage test for trend. We examined, when feasible, differences among geographical areas with larger versus smaller proportions of low-income households; we anticipated that MMC might impact these areas differently. Post hoc analyses of statistical power for nonsignificant findings determined that lack of significance did not reflect insufficient sample sizes. Partly through ethnographic observations, we examined causality to assess whether concurrent trends due to causes other than MMC might have contributed to findings.
Survey Component
This component addressed the first research question, concerning effects on barriers to care, access, utilization, and satisfaction.
9-Month Survey.1
Most respondents reported little change after MMC, which did not significantly affect barriers to access. Uninsured respondents experienced major access barriers. A substantial proportion of Medicaid adults reported transportation as a barrier to care. Adult Medicaid and Medicare recipients enjoyed the most favorable access. Medicaid recipients were more likely to utilize an emergency department than other groups. No differences emerged by insurance category in satisfaction. In logistic regression analyses, uninsured adults and children showed greater barriers and less access than those in other insurance categories; no significant changes occurred after MMC.
18-Month Survey
Both Medicaid and uninsured patients reported cost as barriers to physician services and medications (despite Medicaid's coverage for both). Medicaid recipients remained more likely to report transportation barriers and emergency department visits. All groups except Medicare reported cost as a barrier to dental services. The uninsured were significantly less likely and Medicaid recipients were as likely as other respondents to report a regular PCP or to have visited a physician (Table 1). For children, respondents' answers showed the same overall patterns.
Table 1
Table 1
Barriers to Care, Access, Utilization, and Satisfaction for Low-Income Adults and Children, by Insurance Category, 18-Month Survey
In logistic regression analyses (Table 2A), the uninsured remained significantly more likely to report persisting barriers to care, as well as reduced access and utilization. Medicaid children experienced favorable access and utilization. Comparing their situation with the time before MMC, uninsured adults reported the quality of their care as significantly worse, while adult Medicaid recipients and Hispanics (the latter with borderline statistical significance) reported a favorable change in quality (Table 2B).
Table 2A
Table 2A
Relationship of Respondent Characteristics to Key Measures of Health Services, 18-Month Survey (Logistic Regression Model)
Table 2B
Table 2B
Perception of Change in Health Care Quality since MMC Began (Logistic Regression Model), 18-Month Survey
We examined changes in insurance over time. For adults with Medicaid coverage before MMC, 12.7 percent obtained private insurance by the time of the 18-month survey, while 25.4 percent became uninsured (Table 2C). Among these same respondents, 38 percent obtained full- or part-time employment. Only 22.2 percent of the latter employed respondents gained private insurance, while 37.4 percent reported lack of insurance.
Table 2C
Table 2C
Comparison of Insurance Categories of Respondents before Implementation of MMC, and at the Time of the 18-Month Survey
27-Month Survey
In this survey, we performed more detailed analyses of respondents' current cost and transportation barriers.2 About 25 percent of respondents with Medicaid reported a cost barrier. Forty-nine percent of uninsured adults and 32 percent of uninsured children were unable to see a doctor during the prior year due to cost. About one-quarter of adults and 10 percent of children with Medicaid experienced cost barriers to seeing a physician or dentist and to obtaining a prescription. One-third of Medicaid adults and 15 percent of Medicaid children reported transportation barriers.
We examined current direct medical costs (out-of-pocket payments) and indirect medical costs (waiting and travel times). Most adults and children (57 and 72 percent) on Medicaid reported no out-of-pocket expenditures; by comparison, most adults (80 percent) and children (60 percent) with private insurance incurred such expenditures. On average, respondents with some type of insurance coverage including Medicaid experienced shorter wait times (31–35 minutes) than those without insurance (42–45 minutes). Despite perceived transportation barriers, adults and children with Medicaid reported the shortest travel time to a physician's office, about 20 minutes on average; uninsured adults reported the longest average travel time, 43 minutes.
Ethnographic Component
The ethnographic work aimed to answer the second research question, about changes that took place in workload, processes of supervision, visions of service ethics, financial stress, and provider accountability at safety net institutions and in the communities that the institutions served.
Initial improvements occurred after MMC. According to safety net providers, advantages included the requirement of a PCP, emergency department case managers, and expansion of prenatal case management services. MCOs also sent reminders to parents to schedule well child exams. Disruptive transfers of hospitalized patients at the onset of MMC, experienced in some other states, did not occur. Most Medicaid patients expressed satisfaction with MMC, along the lines of: “without Salud!, we wouldn't be able to afford health care,” or, “I am diabetic and would be on the street if I didn't have Salud!”
Regarding workload, safety net personnel experienced the transition to MMC as stressful and chaotic. Rosters of patients and PCPs were unavailable. Telephone systems to inform patients and staff members about MMC often were busy or inoperable. Workload increased, due to increased administrative procedures and paperwork. MCOs' separate contracts with medical laboratories, BHOs, pharmacies, and transportation agencies led to confusion. Over time, providers' heavier workload persisted, although stress decreased with increased knowledge about MMC procedures.
Medicaid eligibility procedures grew more complex. About one-third of Medicaid applicants did not understand and/or did not complete the process. Clients who went to urban ISD offices to establish eligibility almost uniformly reported an inimical climate. With welfare reform and simultaneous staff cutbacks, ISD eligibility workers—who processed Medicaid applications—saw their caseloads increase from about 300 to as many as 700 clients. While arising from the larger policy environment rather than MMC, difficulties in eligibility determination contributed to access problems that uninsured people reported in the surveys.
Safety net institutions experienced heightened financial stress. Not one had prior experience with MMC, and only a few practice sites had received substantial managed care payments from MCOs for privately insured patients. The institutions encountered greater challenges in collecting Medicaid revenues and in using Medicaid to subsidize care for the uninsured. All CHCs received less income, due to delays and denials of payment. Providers lost capitation payments because MMC randomly assigned patients to new PCPs. Private practitioners experienced higher overhead costs due to MMC.
More adverse effects arose at rural sites than at urban sites. Outside urban areas, MCO penetration rates remained relatively low. Consequently, about 20 percent of rural MMC patients were assigned to PCPs more distant than nearby CHCs. Because of the financial problems, six private PCPs decided to retire, to sell their practices, to move, or to open other practice sites for insured patients.
As a financial strategy, providers affiliated with more than one MCO, and rural providers initiated a coalition to increase their leverage in bargaining. Another strategy involved opening new part-time clinics or offices, offering new services, and reducing clinic hours. The latter cutback contributed to reduced access for uninsured people, as revealed in the survey component.
Regarding service ethics, all safety net providers decided to buffer patients from obstacles imposed by MMC. CHCs and one emergency department stopped requesting preapproval for visits. A CHC for the homeless maintained services for MMC patients but did not seek reimbursement from the MCOs. Indian Health Service facilities accepted American Indian patients with Medicaid, while the MCOs received patients' capitation payments. Providers' advocacy efforts included making phone calls for patients, helping them fill out forms, offering transportation, and assisting them in the enrollment process. Another buffering strategy involved “creative coding” in billing procedures so that MMC covered services that patients needed.
A crisis occurred in mental health services. MCOs initiated strict cost control and accountability measures in collaboration with BHOs, which authorized lower levels of care for acutely ill patients. Safety net institutions sustained financial losses as BHOs delayed or withheld service authorizations and reimbursements. Increased workload contributed to clinicians' stress, burnout, and turnover. Sixty child and adolescent mental health programs closed; agencies that remained reduced services. A net decline of 17 child psychiatrists in the state occurred after MMC. Of 47 licensed child psychiatrists, nine did not accept MMC patients. Eighty-six percent of psychologists accepted Medicaid before MMC; 34 percent did so afterward. In 2000, the state government verified that MMC recipients received lower levels of care than their conditions warranted. The federal government withdrew New Mexico's waiver to include mental health services under MMC. Later, the Bush administration reversed this decision but required extensive changes in the system (Willging et al. 2003; Willging 2005).
Ethnography clarified processes that contributed to declining immunization coverage (as observed in the sentinel events component). Reduced funding for immunizations at public health clinics resulted from the state government's decision to require immunizations by MCOs. PCPs found, however, that immunizations increased costs and reduced efficiency. Some PCPs contracting under MMC informally referred children to CHCs and public health clinics for immunizations, even though these institutions had received reduced funding. Waiting times for immunizations increased.
Sentinel Events Component
This component aimed to answer the third research question, concerning trends in patterns of sentinel health events.
Birth and Death Certificates (Figure 2A)
MMC did not reverse a decreasing trend in the proportion of births with a high level of prenatal care, for both high- (Cochran–Armitage Z=−9.87, p<.001) and low-income areas (Z=−2.67, p=.010). The proportion of births with a low level of prenatal care continued to rise. A previous improving trend in the mean first month of prenatal care did not continue after MMC.
Figure 2
Figure 2
Changes in Key Sentinel Events. (A) Prenatal Care. (B) Immunization Coverage Level Estimates.
Hospital Inpatient Discharge Data
Regarding age-adjusted rates of hospital admissions for conditions sensitive to outpatient care, hospitalizations for asthma decreased significantly after MMC, while those for diabetes and angina pectoris did not. An overall trend in rate or length of hospitalization for these conditions did not emerge.
New Mexico Tumor Registry
During the first 3 years after MMC, the proportion of early stage diagnoses of breast cancer increased significantly; rates of cervical and lung cancer did not change.
National Immunization Survey (Figure 2B)
Immunization coverage levels decreased significantly after MMC implementation. In 1996, for the 4:3:1 vaccination series, New Mexico had achieved a coverage level of 80 percent and ranked 30th in the nation. By 2000, this level decreased to 72 percent, and New Mexico ranked 50th. The proportion of those vaccinated by public sources decreased significantly, while those by private providers, who received capitated payments for children covered by MMC, increased only marginally. (As noted above, ethnographic data suggested that reduced funding for public heath clinics and MCOs' referrals of children with MMC coverage to safety net providers for immunizations contributed to the decrease in coverage levels.)
Data on Reportable Diseases
The incidence of vaccine-preventable diseases fluctuated considerably before and after MMC, but no temporal pattern emerged.
In summary, our project used a long-term, multimethod approach to study the impact of MMC. In the survey component, we found that no consistent changes occurred at the individual level after MMC implementation. Uninsured adults and children experienced worse barriers, less access, lower utilization, and less satisfaction than adults in other insurance categories; experiences of those with Medicaid proved relatively favorable, both before and after MMC. In the ethnographic component, safety net institutions experienced marked increases in workload and financial stress, especially in rural areas. Providers worked to buffer the impact of MMC for patients. Mental health services deteriorated sharply. In the sentinel events component, after MMC, immunization rates declined.
Strengths and Weaknesses of the Multimethod Approach to Assess Policy Change
To our knowledge, our research represents the first multimethod study to use complementary survey, ethnographic, and sentinel event techniques to answer questions at different levels of analysis in assessing a major health policy change. Some strengths and weaknesses of the multimethod approach became evident. Relying on statistical measures of central tendency, surveys best depict major trends in a population's experience. However, such surveys remain relatively insensitive to infrequent experiences affecting small segments of the population. Although ethnographic approaches do not lead to precise estimates about a population's experience, they can identify problems occurring infrequently and provide depth of understanding. The sentinel events component tracked changes in population-level indicators after MMC, but these indicators could not establish definitively the reform's causal impact (although the ethnographic component clarified probable causal processes). Regarding generalizability, multimethod research could assess MMC or similar policy changes elsewhere.
The survey illustrated limitations of cross-sectional, telephone-based research. Our data could have underestimated access barriers by excluding households without telephones. From our own and others' prior research, we expect that the inclusion of people without telephones would accentuate the access barriers deriving from lack of insurance that we found in the telephone survey. In addition, survey items that asked subjects to report on experiences that occurred before and after MMC implementation raised the possibility of recall bias. Asking adults to report the experiences of one child may have led to some difficulty in accurately recalling the index child's experiences for those with more than one child. However, there is no reason to expect that errors in recall occurred differentially among insurance categories.
Arguments for multimethod research must consider costs. Our study's comprehensive design required substantial support, but similar research could proceed with less financing. To conduct a survey of this magnitude would require about $20,000 per wave for a survey research center, plus about $8,500 per year for a trained research assistant. Ethnographic work by two half-time research assistants would cost about $34,000 per year. With a 25 percent research assistant, the sentinel events component would cost about $8,500 annually. Modest expenses for local travel, supplies, and telephone calls would take an additional $10,000 per year. Supervisory functions usually could take place within the current job definitions of faculty members at local universities or professionals based at safety net institutions. Local groups could employ a multimethod approach within existing institutional budgets combined with limited additional fund raising.
Implications for Program Monitoring and Policy Making
By contributing to program monitoring and policy making, our study established a fruitful model for evaluative health policy research. Managers and policy makers responded to our evaluation as well as other input to initiate changes in MMC. For instance, data from the population-based survey revealed that MMC caused no major changes for Medicaid recipients. Lack of insurance, however, remained the single most important predictor of barriers, access, utilization, and satisfaction, both before and after MMC. Our data proved helpful in working toward policies that moved beyond Medicaid to coverage for the uninsured.
The ethnographic component suggested problems not elicited in the surveys. Ethnographic observations helped justify additional funding for safety net institutions (Waitzkin et al. 2002). Organizing efforts based partly on the ethnographic work created such attention that the state initiated a major new mental health initiative (Willging, Waitzkin, and Wagner 2004, 2005).
After the sentinel events component clarified an unexpected deterioration in immunization coverage coinciding with MMC implementation, intense efforts to change policies and practices occurred. When our article appeared in a major primary care journal (Schillaci et al. 2004), the immunization crisis received wide coverage in newspapers and broadcast media. This information assisted in the creation of a statewide immunization task force. Funding and organizational changes led to improved practices. By 2004, New Mexico moved from 50th to 15th among the states in immunization coverage (Institute for Public Health 2006).
MMC affected safety net providers profoundly while leading to little impact on patients, partly because providers worked to buffer the adverse impacts of new policies. Despite MMC, lack of insurance continued to impede access. Future policies should address the effects of MMC on safety net providers and should recognize that such reform does not adequately address the problems of the uninsured.
Acknowledgments
Robert Williams and Megan Perry contributed to the survey component of this project. Louise Lamphere contributed to the ethnographic component. Ann Carson contributed to the sentinel events component. Bert Useem provided helpful suggestions. We also thank the survey respondents, safety net providers, and patients who participated in the study. The editors and two anonymous reviewers provided helpful suggestions that improved the manuscript. Institutional Review Boards of the Health Sciences Center and general campus of the University of New Mexico approved the methods, including those to assure participants' informed consent.
Grant Support: This work was supported in part by grants from the Agency for Healthcare Research and Quality (1R01 HS09703), the National Institute of Mental Health (1R24 MH58404, 1R25 MH60288, 1R03 MH65564, and 1R01 MH076084), the Hospital of the University of New Mexico, and the Dedicated Health Research Funds of the University of New Mexico School of Medicine.
Disclosures: All authors attest that they have no financial interest conflicting with complete and accurate reporting of the study findings. All authors have argued for improved access to health and mental health services under Medicaid managed care and in general.
Disclaimers: The methods, observations, and interpretations in this report do not necessarily represent policies of the funding agencies.
NOTES
1An earlier publication (Waitzkin et al. 2002) presents tables that detail findings from the 9-month survey.
2Tables that present these and other findings from the 27-month survey are available by request.
Supplementary material
The following supplementary material for this article is available online:
Appendix SA1
Author Matrix.
This material is available as part of the online article from http://www.blackwell-synergy.com/doi/abs/10.1111/j.1475-6773.2008.00842.x (this link will take you to the article abstract).
Please note: Blackwell Publishing is not responsible for the content or functionality of any supplementary materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.
  • Alessandrini E A, Shaw K N, Bilker W B, Schwartz D F, Bell L M. Effects of Medicaid Managed Care on Quality: Childhood Immunizations. Pediatrics. 2001a;107:1335–42. [PubMed]
  • Alessandrini E A, Shaw K N, Perry K A, Bilker W B, Baker M D, Schwarz D F. Effects of Medicaid Managed Care on Health Care Use: Infant Emergency Department and Ambulatory Services. Pediatrics. 2001b;108:103–10. [PubMed]
  • Baker L C, Afendulis C. Medicaid Managed Care and Health Care for Children. Health Services Research. 2005;40:1466–88. [PMC free article] [PubMed]
  • Basu J, Friedman B, Burstin H. Preventable Hospitalization and Medicaid Managed Care: Does Race Matter? Journal of Health Care for the Poor and Underserved. 2006;17:101–15. [PubMed]
  • Berman S, Armon C, Todd J. Impact of a Decline in Colorado Medicaid Managed Care Enrollment on Access and Quality of Preventive Primary Care Services. Pediatrics. 2005;116:1474–9. [PubMed]
  • Bindman A B, Chattopadhyay A, Osmond D H, Huen W, Bacchetti P. The Impact of Medicaid Managed Care on Hospitalizations for Ambulatory Care Sensitive Conditions. Health Services Research. 2005;40:19–38. [PMC free article] [PubMed]
  • Bindman A B, Yoon J, Grumbach K. Trends in Physician Participation in Medicaid. The California Experience. Journal of Ambulatory Care Management. 2003;26:334–43. [PubMed]
  • Bouchery E, Harwood H. The Nebraska Medicaid Managed Behavioral Health Care Initiative. Journal of Behavioral Health Services and Research. 2003;30:93–108. [PubMed]
  • Busch A B, Frank R G, Lehman A F. The Effect of a Managed Behavioral Health Carve-Out on Quality of Care for Medicaid Patients Diagnosed as Having Schizophrenia. Archives of General Psychiatry. 2004;61:442–8. [PubMed]
  • Chang D, Burton A, O'Brien J, Hurly R E. Honesty as Good Policy: Evaluating Maryland's Medicaid Managed Care Program. Milbank Quarterly. 2003;81:389–414. [PubMed]
  • Davidoff A, Hill I, Courtot B, Adams E. Effects of Managed Care on Service Use and Access for Publicly Insured Children with Chronic Health Conditions. Pediatrics. 2007;119:956–64. [PubMed]
  • Domino M E, Salkever D S. Price Elasticity and Pharmaceutical Selection: The Influence of Managed Care. Health Economics. 2003;12:565–86. [PubMed]
  • Emerson R M, Fretz R I, Shaw L L. Writing Ethnographic Fieldnotes. Chicago: University of Chicago Press; 1996.
  • Evaluation Gap Working Group. When Will We Ever Learn? Improving Lives through Impact Evaluation. Washington, DC: Center for Global Development; 2006.
  • Fairbrother G, Park H L, Haidery A, Gray B H. Periods of Unmanaged Care in Medicaid Managed Care. Journal of Health Care for the Poor and Underserved. 2005;16:444–52. [PubMed]
  • Felland L E, Lesser C S, Staiti A B, Katz A, Lichiello P. The Resilience of the Health Care Safety Net, 1996–2001. Health Services Research. 2003;38:489–502. [PMC free article] [PubMed]
  • Freund D A, Rossiter L F, Fox P D, Meyer J A, Hurley R E, Carey T S, Paul J E. Evaluation of the Medicaid Competition Demonstrations. Health Care Financing Review. 1989;11:81–97. [PubMed]
  • Garrett B, Davidoff A J, Yemane A. Effects of Medicaid Managed Care Programs on Health Services Access and Use. Health Services Research. 2003;28:575–94. [PMC free article] [PubMed]
  • Haberer J E, Garrett B, Baker L C. Does Medicaid Managed Care Affect Access to Care for the Uninsured? Health Affairs. 2005;24:1095–105. [PubMed]
  • Horton S. Different Subjects: The Health Care System's Participation in the Differential Construction of the Cultural Citizenship of Cuban Refugees and Mexican Immigrants. Medical Anthropology Quarterly. 2004;18:472–89. [PubMed]
  • Institute for Public Health. Childhood Immunizations.” Albuquerque, NM: University of New Mexico School of Medicine [accessed on October 27, 2007]. Available at http://hsc.unm.edu/som/iph/pubHealth101immune.shtml.
  • Jacobson P D, Dalton V K, Berson-Grand J, Weisman C S. Survival Strategies for Michigan's Health Care Safety Net Providers. Health Services Research. 2005;40:923–40. [PMC free article] [PubMed]
  • Kirschke D L, Craig A S, Schaffner W, Daugherty J R, Narramore J, Griffin M R. Childhood Immunization Rates before and after the Implementation of Medicaid Managed Care. Archives of Pediatric and Adolescent Medicine. 2004;158:230–5. [PubMed]
  • Kronick R, Goodman D C, Wennberg J, Wagner E. The Marketplace in Health Care Reform: The Demographic Limitations of Managed Competition. New England Journal of Medicine. 1993;328:148–52. [PubMed]
  • Lamphere L. Providers and Staff Respond to Medicaid Managed Care: The Unintended Consequences of Reform in New Mexico. Medical Anthropology Quarterly. 2005;19:3–25. [PubMed]
  • Long S K, Coughlin T A, King J J. Capitated Medicaid Managed Care in a Rural Area: The Impact of Minnesota's PMAP Program. Journal of Rural Health. 2005;21:12–20. [PubMed]
  • Morgan K O. Health Care State Rankings. Lawrence, KS: Morgan Quitno Press; 2005.
  • Ridgely M S, Maglione M A. Managing Medicaid Behavioral Health Care: Findings of a National Survey in the Year 2000. Psychiatric Services. 2006;57:1000–6. [PubMed]
  • Roohan P J, Franko S J, Anarella J P, Dellehunt L K, Gesten F C. Do Commercial Managed Care Members Rate Their Health Plans Differently Than Medicaid Managed Care Members? Health Services Research. 2003;38:1121–34. [PMC free article] [PubMed]
  • Rylko-Bauer B, Farmer P. Managed Care or Managed Inequality? A Call for Critiques of Market-Based Medicine. Medical Anthropology Quarterly. 2002;16:476–502. [PubMed]
  • Schillaci Michael A, Waitzkin Howard, López C M, Boehm D A, López L A, Mahoney S F. Falling Immunization Coverage and Medicaid Managed Care in New Mexico: A Multi-Method Assessment. Annals of Family Medicine. 2004;2:13–21. [PubMed]
  • Seligman H K, Chattopadhyay A, Vittinghoff E, Bindman A B. Racial and Ethnic Differences in Receipt of Primary Care Services between Medicaid Fee-for-Service and Managed Care Plans. Journal of Ambulatory Care Management. 2007;30:264–73. [PubMed]
  • Shireman T I, Hornung R W, Ho M, Moomaw C J, Jang R. Medicaid Managed Care Prescription Use and Cost Savings. Journal of the American Pharmaceutical Association. 2002;42:587–93. [PubMed]
  • Singer M, Castro A. Introduction. In: Castro A, Singer M, editors. Unhealthy Health Policy: A Critical Anthropological Examination. Walnut Creek, CA: AltaMira Press; 2004. pp. xi–xx.
  • Slifkin R T, Hoag S D, Silverman P, Felt-Lisk S, Popkin B. Medicaid Managed Care Programs in Rural Areas: A Fifty-State Overview. Health Affairs. 1998;17:217–27. [PubMed]
  • Sommers B D. From Medicaid to Uninsured: Drop-Out among Children in Public Insurance Programs. Health Services Research. 2005;40:59–78. [PMC free article] [PubMed]
  • Sparer M S, Brown L D, Gusmano M K, Rowe C, Gray B H. Promising Practices: How Leading Safety-Net Plans Are Managing the Care of Medicaid Clients. Health Affairs (Millwood) 2002;21:284–91. [PubMed]
  • Thompson J W, Ryan K W, Pinidiya S D, Bost J E. Quality of Care for Children in Commercial and Medicaid Managed Care. Journal of the American Medical Association. 2003;290:1486–93. [PubMed]
  • U.S. Census Bureau. “Statistical Abstract of the United States” [accessed on September 15, 2006]. Available at http://www.census.gov/statab.
  • U.S. Health Resources and Services Administration. “Medically Underserved Areas/Medically Underserved Populations” [accessed on September 15, 2006]. Available at http://bphc.hrsa.gov/databases/newmua.
  • Valet R S, Kutny D F, Hickson G B, Cooper W O. Family Reports of Care Denials for Children Enrolled in TennCare. Pediatrics. 2004;114:e37–42. [PubMed]
  • Vivier P M, Alario A J, O'Haire C, Dansereau L M, Jakum E B, Peter G. The Impact of Outreach Efforts in Reaching Underimmunized Children in a Medicaid Managed Care Practice. Archives of Pediatric and Adolescent Medicine. 2000;154:1243–7. [PubMed]
  • Waitzkin H. At the Front Lines of Medicine. Lanham, MD: Rowman and Littlefield; 2001.
  • Waitzkin H, Williams R L, Bock J A, McCloskey J J, Willging C E, Wagner W. Safety-Net Institutions Buffer the Impact of Medicaid Managed Care: A Multi-Method Assessment in a Rural State. American Journal of Public Health. 2002;92:598–610. [PubMed]
  • Weech-Maldonado R, Morales L S, Elliott M, Spritzer K, Marshall G, Hays R D. Race/Ethnicity, Language, and Patients' Assessments of Care in Medicaid Managed Care. Health Services Research. 2003;39:789–808. [PMC free article] [PubMed]
  • Willging C E. Power, Blame, and Accountability: Medicaid Managed Care for Mental Health Services in New Mexico. Medical Anthropology Quarterly. 2005;19:84–102. [PMC free article] [PubMed]
  • Willging C, Semansky R, Waitzkin H. New Mexico's Medicaid Managed Care Waiver: Organizing Input from Mental Health Consumers and Advocates. Psychiatric Services. 2003;54:289–91. [PMC free article] [PubMed]
  • Willging C, Waitzkin H, Wagner W. The Death and Resurrection of Medicaid Managed Care for Mental Health Services in New Mexico. In: Castro A, Singer M, editors. Unhealthy Health Policy: A Critical Anthropological Examination. Walnut Creek, CA: Altamira Press; 2004. pp. 247–56.
  • Willging C, Waitzkin H, Wagner W. Medicaid Managed Care for Mental Health Services in a Rural State. Journal of Health Care for the Poor and Underserved. 2005;16:497–514. [PubMed]
  • Zuvekas A. Health Centers and the Health Care System. Journal of Ambulatory Care Management. 2005;28:331–9. [PubMed]
Articles from Health Services Research are provided here courtesy of
Health Research & Educational Trust