The success of quality improvement initiatives depends not only on whether they achieve their intended outcomes, but on their unintended consequences. With “pay for performance” schemes there is a risk that rewarding performance of certain clinical activities will divert attention from other, unrewarded activities. The original UK Quality and Outcomes Framework was developed with a wide range of indicators, 76 clinical and 70 non-clinical, in part to prevent doctors focusing on too narrow a range of activities and to encourage a broader systematic approach to improving quality of care, although quality improvement initiatives implemented in response to the incentives tended to focus specifically on the incentivised conditions and activities.29
We used patient-level data from general practices using electronic patient records from 1999 onwards to examine trends in achievement for incentivised and non-incentivised activities.
Strengths and limitations of the study
Our study is subject to limitations. First, any changes in the consistency and accuracy of data recording over time would affect our findings, particularly for incentivised indicators. Such changes are more likely to affect indicators assessed by practices (such as body mass index) than indicators assessed by third parties such as laboratories (for example, serum creatinine concentration). However, we found no consistent relation between changes in achievement rates over time and the agent responsible for measuring indicator parameters.
Second, changes in case mix over time might have affected achievement rates, particularly if there were changes in case finding activity under the incentive scheme. To account for this, we controlled for changes over time in indicator denominators and age and sex profiles. We also found no substantial differences in patterns of achievement for different cohorts of patients (such as diabetic patients diagnosed in 2000-1 compared with patients diagnosed in later years).
Third, although the sampled practices were nationally representative in terms of patient demographics, they might have been atypical in terms of organisation and quality of care. Collectively, they had marginally higher achievement scores on incentivised indicators than the national average in each intervention year,30
which might be attributable to higher baseline performance in the pre-incentive period or a greater response to the incentives. However, with respect to their general pattern of achievement for the incentivised indicators—high in the first year followed by diminishing improvements in the second and third years—the sampled practices followed the national trend.8
Fourth, we were highly conservative in selecting indicators, focusing on processes not subject to other forms of incentive and for which guidelines remained unchanged throughout the study period. This limits the generalisability of our findings, although comparison with trends in achievement rates for non-selected incentivised indicators suggests that the selected indicators were not atypical.
In addition to examining trends in achievement for incentivised and non-incentivised aspects of care, we made direct comparisons between these. To strengthen this comparison we grouped indicators, a priori, by activity type. We also controlled for different baseline achievement rates and trends, and for changes in disease registers over time. Nevertheless, the subgroups remained non-equivalent in important respects, both in terms of patient groups and care processes. For example, in contrast to other subgroups, most non-incentivised prescribing indicators related to “negative” (patient safety) activities, and the general focus of the incentives on promoting recommended processes of care rather than on avoiding proscribed activities may have diverted attention from patient safety issues. Activities that were incorporated into the incentive scheme may also have been accorded greater importance by clinicians even before the scheme was implemented.
Non-equivalence does not invalidate our findings on the effect of the incentives on incentivised and non-incentivised indicators, but it raises the question of whether such non-equivalence can account for the observed post-intervention differences between the groups. To investigate further, we used data from the first two pre-intervention years to predict rates of achievement in the last pre-intervention year (2002-3) and found only small deviations from expectation for all four indicator groups. This implies that uncontrolled sources of non-equivalence did not produce group differences in the immediate pre-intervention period, and that the post-intervention differences we observed developed after that point (see appendix on bmj.com). However, as the financial incentives were introduced simultaneously nationwide, with no control practices, we could not test our assumption that in the absence of the incentives the pre-intervention achievement trends would have continued.
The effect of the incentives
Of the 23 incentivised indicators we analysed, measurement indicators improved at the fastest rate in the pre-intervention period, from a relatively low baseline in 2000-1. After the introduction of incentives, achievement rates increased substantially for all measurement indicators, with increases above projected rates in 2004-5 of up to 38%. Of the six indicators with the greatest increases, five entailed recording smoking status—a technically straightforward activity that had a low baseline (either because practices were not asking most of their patients before the incentives or were not recording that they were doing so), required minimal patient compliance, and was monitored exclusively by the practices. Collectively, prescribing activities had higher baseline achievement rates than measurement activities in 2000-1, slower increases in achievement rates in the pre-intervention period, and smaller increases above projected rates in the first intervention year (2004-5) of between 1.2% and 8.3%. Achievement rates for all incentivised indicators reached a plateau in the second and third years of the scheme, and as a result only 14 of the 23 incentivised indicators had achievement rates significantly higher than rates projected from pre-intervention trends after three years.
For non-incentivised indicators the introduction of financial incentives seemed to have little overall impact in the first year, with quality continuing to improve at around the pre-intervention rate. In the second and third years, although quality continued to improve, the rate of improvement slowed. By 2006-7 quality was significantly worse than projected from pre-incentive trends, most notably for measurement activities, and was also significantly below—relative to projection—the quality for incentivised activities.
The general improvement in quality before the introduction of the financial incentives suggests that the quality initiatives implemented in the UK at that time—including clinical audits, development of information technology, and creation of quality oriented statutory bodies—were having an effect in improving quality of care. The Quality and Outcomes Framework built on this infrastructure and introduced a national set of quality targets (clinical guidelines) supported by computer prompts and feedback and backed by financial and reputational incentives. The scheme was associated with additional increases in performance in its first year across a broad range of incentivised activities, but the rate of improvement was not sustained. There are several possible explanations for this. Practices might have improved their recording procedures in the first year of the scheme in response to the incentives, effectively “correcting” achievement rates, particularly for some measurement indicators. After this, the rate of quality improvement seemed to plateau. Alternatively, practices might have reached their achievement limit by 2006-7, with the incentive scheme hastening their arrival at this ceiling. A third explanation is that practices relaxed their efforts after the first year of the scheme. Most practices attained achievement rates above the maximum achievement thresholds (the level of achievement required to secure maximum remuneration under the scheme) for most incentivised indicators in the first year, and thereafter had no financial incentive for further improvement.
Conclusions and policy implications
Quality of primary care was generally improving in England in the early 2000s. The introduction of an incentive scheme seemed to accelerate this trend for incentivised activities, but quality quickly reached a plateau. Incentives had little apparent impact on non-incentivised activities in the short term, but seem to have had some detrimental effects in the longer term, possibly because of practices focusing on patients for whom rewards applied. Some aspects of the UK Quality and Outcomes Framework and its setting may have limited its impact on non-incentivised activities (such as the lack of financial penalties, the wide range of indicators, and the existence of other quality initiatives) and other aspects may have exacerbated its impact (such as the large size of the incentives and the appearance of incentive related prompts on clinical computing systems during consultations). Findings may be different under schemes with different incentive structures operating in different settings. Nevertheless, these findings show some important limitations of financial incentive schemes in health care, and the importance of monitoring, as far as possible, activities that are not incentivised in addition to those that are when determining the effects of such schemes.
What is already known on this topic
- Carefully constructed financial incentive schemes in healthcare can lead to improvements in quality of care for incentivised activities
- Concerns remain about the potential unintended consequences of financial incentives, in particular that they could lead to neglect of non-incentivised aspects of patient care
What this study adds
- We examined trends in quality of care for 42 activities (23 incentivised under the UK Quality and Outcomes Framework incentive scheme, and 19 not incentivised) selected from 428 identified quality of care indicators
- Quality of care initially improved for incentivised activities but quickly reached a plateau. Incentives had little impact on non-incentivised activities in the short term, but after three years quality of care for some fell significantly below levels predicted from pre-incentive trends
- Improvements associated with financial incentives seem to have been achieved at the expense of small detrimental effects on aspects of care that were not incentivised