One area of uncertainty is the formula through which performance is translated into additional revenue. Many programs use “tournaments” or fixed achievement thresholds to determine whether providers merit additional pay (Centers for Medicare and Medicaid Services 2005). A theoretical argument exists, and some empirical evidence supports the notion that this model rewards providers who are already performing well while providing little incentive for quality improvement (Rosenthal et al. 2005; Chien et al. 2007a; Hayward 2007; Karve et al. 2008; Friedberg et al. 2010;). Paying providers a “piece-rate” for each patient meeting a performance benchmark (e.g., each diabetic receiving HbA1c testing twice a year) is one way to reward continual improvement. This payment scheme theoretically rewards continuous improvement in performance regardless of the baseline levels. It may also minimize incentives for providers to avoid high-risk or nonadherent patients from their practices or panels and thereby reduce the risk of exacerbating disparities (Hofer et al. 1999).

Piece-rate performance incentives have been studied a handful of times particularly in conjunction with improving childhood immunization or well-child visit rates (Chien, Conti, and Pollack 2007b). Nearly 15 years ago, two randomized studies concluded that piece-rate P4P rewards were effective at improving immunization documentation in urban practices serving low-income patients (Fairbrother et al. 1999, 2001). These studies were performed at a time when the Centers of Disease Control (CDC) measured immunization rates using the 4 : 3 : 1 immunization series (four doses of vaccines against Diphtheria–Tetanus–Pertussis, three doses against Polio, and one dose of protection against Measles–Mumps–Rubella). Between 1999 and 2002, immunization recommendations evolved significantly to incorporate the development of new vaccines and guidelines for “catching up” children who experienced vaccination delays. By 2003, the CDC began measuring progress toward national childhood immunization goals using the 4 : 3 : 1 : 3 : 3 : 1 immunization series (the vaccines in the 4 : 3 : 1 series plus three doses against Haemophilus influenzae Type B, three against Hepatitis B, and one against Varicella Zoster Virus). This rate continues to be a core metric by which national, state, and local health agencies gauge their progress (Center for Disease Control and Prevention 2009). A recent study of piece-rate P4P evaluated the effectiveness of this strategy in raising attendance levels at the well-child visits in which the 4 : 3 : 1 : 3 : 3 : 1 immunization series is delivered. This study found that the two Medicaid-sponsored programs that raised rates more than the Medicaid national mean had combined their incentives with programmatic adjuncts like patient incentives and reminders (Felt-Lisk, Gimm, and Peterson 2007). “Piece-rate” payment strategies have a small but promising empirical base that warrants more rigorous examination.

This study takes advantage of a natural experiment that occurred within the Medicaid health plans in New York State. One health plan introduced a distinctive piece-rate P4P program aimed at improving the childhood immunization rates as measured by the 4 : 3 : 1 : 3 : 3 : 1 series, while others attempted more conventional tactics (e.g., reminders and outreach to providers and parents without incentives) (Jacobsen Van JC 2005). Patient-level data allow us to explore whether the P4P program is associated with changes in immunization rates for specific patient subgroups. Although previous studies have not been able to examine whether children of racial/ethnic minority backgrounds or with chronic health conditions are differentially impacted by P4P programs, it is known that there is a paradoxical racial/ethnic disparity in immunization rates among those who live below the federal poverty line—Hispanics have the highest immunization rates (followed by blacks then whites) (Center for Disease Control and Prevention 2009).

This study allowed us to examine, but did not shed significant light on, the mechanisms by which improvements were achieved. Our exploratory analyses of patient-level claims data for continuous enrollees showed that patients actually experienced no significant change in visit number, number of shots per visit, immunization rates, or preexisting disparities. The lack of a significant finding may be because we could not assess visit rates change for children who are not continuously enrolled. It is possible that gains in immunization rates were concentrated among children who recently obtained Medicaid insurance. The increasing number of combination vaccines may also mask more substantial changes in immunization adherence (e.g., a single shot can bring a child with up to three delayed immunizations into compliance).

Our study demonstrates many strengths relative to the existing literature. We observed a robust piece-rate P4P intervention with substantial administrative supports (e.g., the provision of patient lists to practices) over a 4-year period, and we obtained within-state comparison health plan quality data, thus minimizing artifacts from state-to-state variations in Medicaid eligibility, coverage, and reimbursement.

There are, however, three key study limitations. We only have aggregate data for our comparison group. Although our concerns about immunization rates being affected by variations in state Medicaid laws or practices are mitigated, we are unable to pool the intervention and comparison group data and conduct a patient-level analysis to assess potential changes in the patients or practices that may affect immunization rates within New York over the study period. Secondly, our evaluation starts at the end of the same year that the intervention begins. We were not able to obtain health plan immunization rates before 2003 because plans were being benchmarked according to different immunization guidelines (e.g., the 4 : 3 : 1 : 3 or 4 : 3 : 1 : 3 : 3 series rather than the 4 : 3 : 1 : 3 : 3 : 1 series instituted in 2003). Data from 2003 are the best baseline possible given data constraints and because large-scale interventions like this take time to implement. We were not able to measure, however, a preintervention trend and look for a deviation from prior trends. Lastly, the data used for this study were not amenable to differentiating between better documentation and more complete immunization of the target population as has been noted in previous studies (Fairbrother et al. 1999, 2001). Differentiating between quality documentation and quality of care is an important distinction, but we have reason to believe that the overall quality of immunization documentation is more accurate and reliable than 15 years ago when previous studies noted that documentation and underlying care were dissociated. By 2003, proper immunization documentation was required and enforced to such a degree that under-documentation could be considered tantamount to poorer quality care. A short list of the substantial and sustained state and federal immunization efforts that have been implemented since the early P4P studies include the following: (a) Vaccines for Children (VFC) program; (b) the coordinated effort to update the recommended childhood immunization schedule by three main professional societies (i.e., Advisory Committee on Immunization Practices [ACIP], American Academy of Pediatrics [AAP], and American Academy of Family Physicians [AAFP]); (c) the passing of state laws requiring immunization documentation before school entry in all 50 states; and (d) advances in quality reporting within state departments of health (e.g., New York's QARR program). All of these programs require, verify, or enforce proper immunization documentation and most would agree that clinicians should base their treatment decisions on what is actually documented (i.e., if it was not documented, it was not done).