This demonstration project addresses several important questions concerning the feasibility and merit of measuring patients' experiences with individual physicians. Perhaps most importantly, the measures showed high physician-level reliability with samples of 45 patients per physician, and at these levels of reliability, the risk of misclassification was low (≤2.5%) given a reporting framework that differentiated 3 levels of performance. However, the results also underscored that high misclassification risk is inherent for scores closest to the performance cutpoint (“benchmark”) and where multiple cutpoints are introduced. Thus, reporting protocols will need to limit the number of performance categories, and determine how to fairly handle cases most proximal to performance cutpoints, where misclassification risk will be high irrespective of measurement reliability. Also importantly, the study showed that individual physicians and practice sites are the principal delivery system influences on patients' primary care experiences. Network organizations and health plans had little apparent influence. These findings accord with previous analyses of practice site, network, and health plan influences on patients' experiences,4,23
but extend beyond previous work by estimating the effects of individual physicians within practices sites.
The results address several persistent concerns about physician-level performance assessment. First, the difficulties of achieving highly reliable performance measures have been noted for condition-specific indicators, given limited sample sizes in most physicians' panels, even for high prevalence conditions like diabetes and hypertension.2,24,25
Because the measures evaluated here apply to all active patients in a physician's panel, sample sizes required for highly reliable measures are easily met. Second, questions about whether measurement variance can be fairly ascribed to physicians themselves versus to the systems in which they work and the patients they care for have been prominent a concern.24–27
In this study, physicians and sites accounted for the vast majority of delivery system variance for all measures, but there was substantial sharing of variance between physicians and sites. The results suggest a shared accountability for patients' care experiences and for improving this aspect of quality.
Importantly, although, a substantial amount of variance on all measures remained unaccounted for. Although the delivery system variance accounted for here is considerably higher than that accounted for in studies of other types of performance indicators,24,26,28
it nonetheless raises a critical question for the quality field: Is it legitimate to focus on the performance of physicians and practices when there are so many other influences at play? This seems analogous to questioning whether clinicians ought to focus on particular known clinical factors, such as blood cholesterol levels, even when these factors only account for a modest share of disease risk. As with health, the influences on quality are multifactorial. Because there is not likely to be a single element that substantially determines any dimension of quality, we must identify factors that show meaningful influence and are within the delivery system's purview. In this study, average performance scores across the physician population spanned more than 20 points out of 100. This suggests that meaningful improvement can be accomplished simply by working to narrow this differential. The well-documented benefits of high quality clinician-patient interactions—including patients' adherence to medical advice,29–33
improved clinical status,34–37
loyalty to a physician's practice,38
and reduced malpractice litigation39–41
—suggest the value of doing so.
As with any area of quality measurement, however, there are costs that must be considered against the value of the information. Data collection costs associated with this study suggest that obtaining comparable information for adult primary care physicians statewide (n
=5,537) would cost approximately $2.5 million- or 50-cents per adult resident. Extrapolating to adult primary care physicians nationally (n
=227,698), “per capita” costs across U.S. adults appear similar. Of course costs of such an initiative are highly sensitive to numerous variables—including the frequency, scope and modes of data collection6
—and thus can only be very roughly gauged from this initiative. But a serious investment would clearly be required to accomplish widespread implementation of such measures, and while several such initiatives are currently underway,42–45
the potential for sustaining and expanding upon these over the long term remains unclear.
There are several relevant study limitations. First, the study included only patients of managed care plans (commercial and Medicaid). For other insurance products, where payers are not explicitly aware of members' primary care arrangements, a different sampling methodology would be required. Second, the initiative was limited to one state. In other geographic areas, individual plan effects could be larger, although previous evidence from national studies suggests these are unlikely to be of a magnitude necessitating plan-specific samples in a physician's practice.4,23
Finally, the measures here do not afford information on technical quality of care. Methodologies other than patient surveys are required for that area of assessment.
In conclusion, with considerable national attention focused on providing patient-centered care, this project demonstrates the feasibility of obtaining highly reliable measures of patients' experiences with individual physicians and practices. Physician-specific samples were created by pooling sample across multiple payers, and with samples of 45 completed surveys per physician, highly reliable indicators of patients' experiences were established. The finding that individual physicians and sites account for the majority of system-related variance indicates the appropriateness of focusing on these levels for measuring and improving these dimensions of quality.
The importance of adding measures of patients' experiences of care to our nation's portfolio of quality measures is underscored by recent evidence that we are losing ground in these areas.46,47
The erosion of quality on those dimensions stands in sharp contrast to recent improvements noted in the technical
quality of care.48–50
The improvements in technical quality, seemingly spurred by the routine monitoring and reporting of performance on these measures, lend credence to the aphorism that “what gets measured gets attention.” With key methodological barriers to measuring patients' experiences with individual primary care physicians and practices addressed, it is time to add this balance and perspective to our nation's portfolio of quality measures.