We found relatively weak associations between the assessed indicators of quality of HIV care. Of the 28 possible correlations between the eight quality measures, only two (7%) were greater than 0.40, two were between 0.30 and 0.39, and 20 (71%) were less than 0.20. This was particularly surprising because we assessed quality of care for a single chronic medical condition in sites that were specialised HIV clinics or had specialised HIV care teams. Furthermore, there were no more “high performing” organisations than were predicted by chance. Only one site was in the top quarter for seven measures, and no sites were in the top quarter for six or eight measures. We expected that their focus on HIV care would lead some of these sites to develop systems and procedures that would positively affect multiple aspects of care. Moreover, guidelines for HIV care had been widely disseminated when these data were collected.15 16 17
We thought that in HIV care sites the preconditions would exist for high correlations among measures, including focus on a single condition, relatively high proportions of specialisation (97%), and the presence of multidisciplinary HIV care teams (at 80% of sites). Our results suggest that specialisation and focus on a specific condition may not be sufficient to produce high quality in multiple aspects of care. Consistency may require the coordination of multiple processes and procedures. Consider, for the sake of argument, two contrasting models of the clinical processes related to the eight quality measures we assessed. In the first model, a common system connects all eight measures. Elements in this common system include the physical space, clinic staff, a phone and messaging system, medical records, regular group meetings, and shared (specialised) clinical knowledge. In the second model, each quality measure can be thought of as the outcome of an independent chain of linked processes; failure of any single process in the chain causes the desired quality event not to occur. Because each chain of processes is independent of the others, success or failure of one chain has little impact on the success or failure of a simultaneously operating or parallel chain.
For example, starting and maintaining a patient on HAART may require preparatory visits with several different providers (for example, physicians, pharmacists, and case managers) and access to these providers during the initial phases of treatment. Doing tests to screen for tuberculosis requires a provider, usually a nurse, who carries out the test, ensures that it is appropriately read 24-48 hours later, and documents the results. Ensuring regular cervical cancer screening, on the other hand, may require the cooperation of a nearby gynaecology practice. Each of these examples involves largely independent chains of processes. P carinii pneumonia prophylaxis and HAART may have been more highly correlated than most other pairs of measures because the chains of processes that produce these outcomes have several shared elements (that is, both are prescriptions given by physicians, and both are guided by CD4 cell counts).
Adequate coordination among processes is probably more difficult to achieve when quality measures assess care given by different providers at multiple care sites (such as different services in a hospital). One potential reason that studies of quality19 20
and quality improvement efforts13 21 22 23
have yielded less impressive results than many expected may be the difficulty of simultaneously improving and coordinating multiple systems.
During the study period there were no specific incentives in place (financial or otherwise) for the practices we studied to meet specific quality targets, nor were there any centralised or public processes to measure quality. Such measurement processes and incentives may increase correlations among quality measures, even in the absence of effective and coordinated systems and processes.
One interpretation of these data could be that providers recognise that they cannot provide uniformly high quality care and that they therefore prioritise. For example, few would debate that HAART use is clinically the most consequential of the eight measures we assessed, and the median proportion for use was among the highest the eight proportions at 0.81. Furthermore, the correlation between HAART use and P carinii pneumonia prophylaxis (also consequential clinically) was relatively high (0.42). On the other hand, prioritisation would not explain why the median proportion for hepatitis C screening was higher than for HAART use. High proportions of hepatitis C screening may be observed because it involves only ordering a blood test, and because once a positive result is found the test does not need to be repeated. If providers know that they have to trade off some goals of care against others, however, this is further proof that performance on one measure might imply little about performance on others, even in the setting of specialty care for a single disease. While no one advocates a healthcare system in which one measure of good care competes against another, limited resources and difficult choices are a reality in all healthcare settings.
We examined quality measures that could be assessed by reviewing medical records. All but one (non-detectable viral load) were measures of process. Our findings might have differed if we had been able to assess mortality, rates of admission to hospital, appropriate management of opportunistic infections, changes in health status, adherence to medication, or patients' reports about care.24
We think that measurement of other processes or outcomes, however, would yield even lower correlations. Four of the care processes we assessed (screening for tuberculosis, hepatitis C, and cervical cancer, and influenza vaccinations) are simple to implement for anyone with basic clinical training, and the remaining four (HAART therapy, viral load control, P carinii
pneumonia prophylaxis, and frequency of visits) are the subject of detailed guidelines for clinical practice.25 26
Guidelines for tuberculosis skin testing do not suggest yearly testing, but rather that annual repeat testing (after an initial negative test result) should be considered in populations with a “substantial risk” of exposure (such as prison inmates),18
which may have reduced the proportion who received tuberculosis screening. Another limitation of reviewing medical record is that processes may have been completed, but not documented, biasing our estimates downwards.
Finally, we studied patients at clinics receiving specific funding, and our findings may not generalise to other HIV care settings. Because this specific funding goes to rural and urban underserved care sites, our findings may not generalise to sites that care for patients with, for example, higher incomes, more education, and better health insurance. The sites studied, however, receive considerable scrutiny as a condition of participation in the programme, and quality levels there might be higher than at some other HIV care sites. To the extent that our study design excludes sites with consistently low quality scores, the correlations that we report are lower than they would be in a broader sample.
Our findings have implications for efforts to monitor quality and improvement. Current policy initiatives that seek to pay physicians for their performance on a small selected set of indicators or that create tiers of physicians or hospitals may not improve quality across a broad spectrum of care or conditions. Indeed, such programmes could prompt physicians or physician organisations to channel efforts into affecting the indicators being assessed to the detriment of other aspects of quality.27
More empirical studies are needed on the impact of pay-for-performance initiatives and other improvement strategies on overall quality.28 29
Our results suggest that none of the sites we studied had the kinds of administrative, clinical, and human resources systems in place that are necessary to produce consistently high care quality. Continued and concerted efforts to improve healthcare systems may yield such patterns of high performance, but that goal has remained elusive to date. This should stimulate us to redouble our efforts to identify and implement the kinds of system changes that will allow us to cross the “quality chasm.”30
Focusing on the improvement and coordination of multiple systems within organisations may be a useful direction to pursue.
What is already known on this topic
- Selected indicators are often used as measures of overall quality of care
- Few studies have published correlations between indicators of care quality, and none has done so for outpatient specialty care
What this study adds
- There were low correlations among quality indicators for people with HIV disease
- It might be hazardous to generalise from performance on a small number of quality indicators to performance on other indicators that are not measured