|Home | About | Journals | Submit | Contact Us | Français|
The validity of quality of care measurement has important implications for practicing clinicians, their patients, and all involved with health care delivery. We used empirical data from managed care patients enrolled in west coast physician organizations to test the hypothesis that observed changes in health-related quality of life across a 2.5-year window reflecting process of care.
Patient self-report data as well as clinically detailed medical record review regarding 963 patients with chronic disease associated with managed care from three west coast states.
Prospective cohort study of change in health-related quality of life scores across 30 months as measured by change in SF-12 physical component scores.
Patient self-report and medical record abstraction.
We found a positive relationship between better process scores and higher burden of illness (p<.05). After adjustment for burden of illness, using an instrumental variables approach revealed better process is associated with smaller declines in SF-12 scores across a 30-month observation window (p =.014). The application of the best quartile of process of care to patients currently receiving poor process is associated with a 4.24 increment in δ SF-12-physical component summary scores.
The use of instrumental variables allowed us to demonstrate a significant relationship between better ambulatory process of care and better health-related quality of life. This finding underscores the importance of efforts to improve the process of care.
The validity of quality of care measurement has important implications for practicing clinicians, their patients, and all involved with health care delivery. The classic strategy for assessing the internal validity of quality of care measures is to examine whether patient outcomes are mediated by process (Donabedian 1982; Field et al. 2001). This can be accomplished either with randomized trials (Antiplatelet Trialists' Collaboration 1994; ISIS-2 [Second International Study of Infarct Survival Collaborative Group] 1988) or, in the case of observational data, with cohort studies. In either case, the fact that sicker patients receive more process, presents challenges in revealing a relationship between process and outcomes. A decade ago, a significant relationship between processes of care and outcomes for hospitalized patients was noted, but even then a paradoxical relationship between process and outcomes for the sickest patients was noted (Kahn, Keeler et al. 1990; Kahn, Rogers et al. 1990). We hypothesize that this paradoxical relationship is a consequence of the association between unmeasured burden of illness and both more process and also worse outcomes. Angrist, Newhouse, McClellan, and Brooks have documented a methodology and a clinical context for using instrumental variables to disentangle the endogeneity of unmeasured burden of illness and both processes and outcomes (McClellan, McNeil, and Newhouse 1994; Angrist, Imbens, and Rubin 1996; Brook, McGlynn, and Cleary 1996; Brooks, McClellan, and Wong 2000; McClellan and Newhouse 2000; Brooks et al. 2003).
We applied this methodology to the study of the quality of processes of care to evaluate whether better process was associated with better outcomes. We used empirical data from managed care patients enrolled in west coast physician organizations to test the hypothesis that after adjustment for burden of illness, the changes in health-related quality of life across a 2-year window reflecting process of care. To date, few studies have shown a relationship between explicit process measures and outcomes in the ambulatory setting (Nobrega et al. 1977; Romm and Hulka 1980; Berlowitz et al. 1998; Safran et al. 1998; Asch et al. 2005; Haynes et al. 2005; Higashi et al. 2005), and none have done so for insured managed care patients, or for a health status measure.
We used patient self-report data from a cohort of 963 chronically ill patients in 1996 (Damberg and Bloomfield 1997) and 2.5 years later in 1998 (Pacific Business Group on Health 2004), as well as clinically detailed data abstracted from medical records of patients associated with 39 west coast physician organizations to assess quality of care for 30 consecutive months (Kahn et al. 1999, 2003). Registered nurses experienced with both clinical practice and medical record abstraction used an abstraction instrument designed specifically for this project to abstract records from 963 patients representing data from 5,095 unique patient–physician dyads.
We analyzed adherence (yes or no) to 120 explicit process criteria based upon data from the 1996 Core Survey (Damberg and Bloomfield 1997), the 1998 Chronic Condition Survey (Pacific Business Group on Health 2004), and medical record review of all visits that occurred between the two surveys. Explicit process criteria were based on clinical practice guidelines, literature, and clinical judgment. To assess interrater reliability we compared the performance of 11 pairs of abstractors who independently assessed adherence to explicit process measures from the medical records of 54 unique patients. Concordance between abstractors was excellent with no significant difference noted across abstractors in overall process scores. The aggregate kappa score across process measures was 0.87 (Landis and Koch 1977).
The cohort for this analysis of the relationship between process scores and health-related quality of life outcomes is defined as patients with at least one of three diseases where literature and trials suggest such a relationship between process and outcomes might be expected. Specifically, we studied 963 patients with at least one of: ischemic heart disease, asthma and/or emphysema, or diabetes diagnoses described as present with 1996 self-report and corroborated with evidence from the 1996 and 1998 surveys and medical record data.
Study patients were evaluated with disease-specific explicit process criteria pertinent to hypertension, diet/nutrition, obesity, exercise, smoking, hyperlipidemia, thyroid disease, menopause, depression, medication management, substance abuse, and follow-up or continuity for patients. The 120 process measures (58 generic and 62 disease specific) were specified as applicable to individual patients according to their age, gender, and clinical characteristics.
The explicit process measures were selected to collectively represent one of six domains of clinical care. Domain 1, Cognitive Diagnostic Process, uses 31 explicit process criteria to evaluate the extent to which the provider systematically collected patient historical data (e.g., presence or absence of symptoms, precipitating or relieving factors) (Hunt and Gerstein 1999) necessary for the clinician to make an adequate assessment of the patient's clinical needs. Domain 2, Physical Examination, uses 20 explicit criteria to evaluate the provider's use of pertinent components of the physical exam (e.g., lung exam for patients with asthma, foot exam for diabetic patients) (American Diabetes Association 1998). Domain 3, Laboratory Studies, uses 24 criteria to evaluate the use of laboratory studies for diagnostic or surveillance purposes (e.g., monitoring of creatinine in patients using angiotensin converting enzymes) (Knight and Avorn 2001). Domain 4, Procedures, uses 7 criteria to evaluate the provider's use of diagnostic procedures (ACC/AHA 1997). Domain 5, Medications, uses 26 criteria to evaluate provider recommendations for, and patient use of medications (e.g., use of β-blocker for patients with myocardial infarction and no contraindication) (Ryan et al. 1999). Domain 6, Counseling, uses 12 criteria to score the provider's counseling interventions (as an alternative or supplement to pharmacological or procedural interventions) (U.S. Preventive Services Task Force 1996).
For each patient, the proportion of applicable domain-specific process criteria passed was used to calculate domain-specific process scores. Process criteria were weighted equally within domains and domains were weighted equally regardless of the number of criteria comprising the domain. Aggregate observed process is specified as the mean of six domain-specific process scores, with each of those scores representing a proportion defined as the number of THEN criteria met, conditional on the IF being applicable to the patient. This variable defines the aggregate of adherence to the 120 explicit process measures in the ordinary least squares (OLS) regression model.
For each patient, we calculated three dimensions of burden of illness: a count of up to 39 patient comorbidities noted by either patient self-report or medical record review; the severity of cardiac, pulmonary, or diabetic disease; and body mass index (BMI) (Iezzoni 1994;Field et al. 2001) (see Table 1). Items eligible for scoring a point in the comorbidity index include: cardiovascular problems; cerebrovascular disease; cancer; diabetes; chronic lung disease; common ambulatory problems; depression; measures of functional impairment; habits associated with medical problems; and patient report of worsening health status as documented in Table 1. Severity of the patient's coronary heart disease, lung disease (asthma or emphysema), or diabetes were calculated in a disease-specific manner defined to be independent of use of services (Table 1). To test the validity of the comorbidity and staging systems, we checked the relationships between the comorbidity and staging scores and the construct of burden of illness as measured by the number of drug categories the patient used.
Patient demographics were categorized as age, gender, race, Hispanic ethnicity, education, and income.
Health-related quality of life scores were computed for each patient in 1996 and again, 2 years later. We calculated change in SF-12 scores as the simple arithmetic difference between each patient's raw 1998 and earlier 1996 SF-12 SF-12-PCS (Ware, Kosinski, and Keller 1996). A positive change value is interpreted to mean the patient's health-related quality of life improved with time; a negative value indicates a decrement across the 2-year period.
We had an a priori concern that better process might be associated with both greater measured burden of illness, and also with greater unmeasured burden of illness (and/or provider challenges to implementing process and facilitating better patient outcomes), and that the greater burden of illness was an important predictor of both more process and worse outcomes. This concern led to the use of instrumental variables methods to address the potential endogeneity of process (McClellan et al. 1994). We postulated instrumental variables would be useful because unmeasured burden of illness is, by definition unobservable (i.e., not measurable); unmeasured burden of illness influences both processes (a key independent variable) and outcomes (the dependent variable); and unmeasured burden of illness (the omitted variable) is an important predictor of both processes and outcomes when using OLS regression. We used the augmented test (Davidson and MacKinnon 1993) as a way to evaluate whether instrumental variables would be useful in addressing a potential bias.
To gain unbiased estimates of the influence of process on change in SF-12-PCS, we used the structure of care associated with study patients as an instrument for process. Structure of care meets the two essential criteria for an instrument (McClellan et al. 1994; McClellan and Newhouse 2000): structure theoretically is a major determinant of process (Donabedian, 1980; Ann Arbor, Michigan) and structure influences outcomes only as mediated by process. Using structure as the instrument allows us to exploit treatment variation across structure-specific patient cohorts. This allows us to evaluate the effect of process on health-related quality of life outcomes for patients whose process might change at the margin if they were engaged in a different structural arrangement (Angrist et al. 1996).
We use indicator variables for each of the physician organizations as the instruments for structure when predicting outcomes using two separate models. With instrumental variables, observed process in Stage 1 is modeled using patient-level burden of illness (defined as comorbidity score, the severity of heart, lung and diabetes diseases, and BMI); demographics; the frequency of clinical visits; and an indicator for each of the physician organizations. In Stage 2, we model change in SF-12-PCS score as a function of predicted process from Stage 1, patient-level burden of illness, demographics, and the frequency of clinical visits. To account for the unique ways in which adherence to individual process measures vary by comorbidity, disease-specific stage, and BMI, we supplement the three patient-level burden of illness variables in Stage 1 with the aggregate predicted process criteria. This detailed modeling allows us to improve the prediction in the first stage model to improve the estimation of the process-outcome link.
The first stage equation for instrumental variable uses OLS to predict observed aggregate process as follows:
where aggregate predicted process criteria is specified as the aggregate of predicted adherence for each explicit process criteria. It is introduced into the model for the same reason that standard burden of illness measures such as comorbidity, severity, and BMI are included in the prediction of process. However, because we have both clinical and empiric evidence that providers consider the importance of burden of illness uniquely for each individual process measure, we have included in the model a measure of predicted process criteria. This is generated for each individual process measure by a regression:
The aggregate predicted process criteria is the aggregate version of these 120 individual predicted variables. In summary, this variable is a supplemental measure of burden of illness included in Stage 1 of the instrumental variables model to represent the ways providers uniquely consider burden of illness as they approach the implementation of individual process measures. Note that at the end of this process, the aggregated predicted process criteria is only a function of comorbidity, severity, and BMI.
The second stage equation for instrumental variables uses OLS to predict change in SF-12-PCS using aggregate predicted process criteria to account for the unique ways in which adherence to individual process measures varies by comorbidity, disease-specific stage, and BMI as follows:
where predicted process is defined as Y-hat from instrumental variables Stage 1. This is the model resulting from the regression of aggregate observed process on the full set of predictors in Stage 1 of instrumental variables.
We used SAS 8 and Stata 8 for analyses. We present regression results adjusted for clustering of patients within physician organizations using Huber–White correction (White 1980) displaying regression coefficients, 95 percent confidence intervals, and p-values from OLS and both stages of the instrumental variables models. Augmented tests for endogeneity are presented (Davidson and MacKinnon 1993; Baum, Schaffer, and Stillman 2003). Analyses are weighted as the product of sampling weight in 1996, survey nonresponse weight in 1996, disease sampling, survey nonresponse weight in 1998, and medical record abstraction nonresponse weight.
The final study cohort includes 963 patients from 39 physician organizations with baseline 1996 and 30-month follow-up survey data, as well as abstracted medical record data spanning 30 months after the baseline patient self-report survey. The study cohort has a mean age of 60 years (SD 9), with 41 percent at least 65 years and 52 percent female. Forty-four percent of patients had no more than a high school education; 30 percent reported annual income less than $30,000. This cohort of patients with at least one chronic illness had frequent clinical encounters with a mean of one clinical visit per month.
Patients had a mean of 7.90 (SD 3.49) comorbidities of 39 studied (Table 1). The severity of patients' heart, lung, and diabetes conditions ranged from 0 to 1.0 with a mean of 0.49 (SD 0.29). The mean BMI was 29 (SD 6) with 38 percent overweight, and 34 percent obese. As a test of the construct validity of the comorbidity and severity scores, we evaluated the relationship between them and the number of medications used by patients. We found a positive relationship between the patient's overall medication count and the number of comorbid conditions (p<.0001) and BMI (p =.0003); and the disease-specific medication count and the disease severity scores for heart disease (p<.001), lung disease (p =.049), and diabetes (p =.097). The mean 1996 SF-12-PCS was 42 (SD 12), 8 points lower than the national average for adults (Ware et al. 1996), reflecting our cohort having been defined as patients with at least one chronic condition. In 1998, the mean SF-12-PCS was 41 (SD 12) with a mean change of −1.20 (SD 9.83).
Overall, we note the relationships between overall process and the dimensions of severity, comorbidity, and BMI are all positive with statistical significance (p<.0001) for comorbidity and severity. The pattern is reproduced with domain-level process scores and severity and comorbidity. To better understand the nature of the relationship between adherence to process indicators and burden of illness, we compared patients who passed and failed individual process criteria with respect to severity, comorbidity, and BMI. Table 2 presents examples of one explicit process criterion (columns B–D) from each domain of process with documentation regarding the number of patients for whom the criterion was applicable (column C), and the number of applicable patients for whom the criterion was met (column D). Table 2 (columns G–K) shows the bivariate (column J) and multivariable (column K) relationships between burden of illness scores (severity, comorbidity, and BMI) and process measure adherence varies from measure to measure (shown as rows). This provides support for the clinical observation that providers adhere to process measures after consideration of the patients' burden of illness, with their consideration of burden being individualized for each process measure. This also provided support for our analytic strategy of adjusting adherence to each individual process measure for comorbidity, severity, and burden of illness before grouping adherence to the measures together into a single aggregate predicted process score. Note in the Stage 1 regression, the three individual components of burden of illness are included as separate independent variables. The small negative relationships associated with these variables in the prediction of aggregate observed process criteria (Stage 1), should be interpreted in the context of these variables already being in the model in the alternative form of aggregate predicted process criteria.
After standardization, the mean observed aggregate process score was (−0.02), SD 0.62 [minimum (−2.34), maximum (1.40), 25th percentile (−0.38), 75th percentile (0.45), interquartile range (0.83)]. The mean predicted overall process score was −0.03, SD 0.62, [minimum (−1.84), maximum (1.36), 25th percentile (−0.45), 75th percentile (0.43), interquartile range (0.89)].
A test of potential bias using the augmented test (Davidson and MacKinnon 1993) shows OLS is not appropriate for predicting changes in SF-12. The F-test for the instruments in the first-stage regression is 349 with 38 degrees of freedom confirming a very strong joint effect. The model rejected the null hypothesis of no endogeneity (p =.001) suggesting the need for instrumental variables as we have done (Davidson and MacKinnon 1993).
Table 3 column one presents the OLS-model with no significant relationship between process and delta SF-12-PCS; the negative sign on process predicting delta SF-12-PCS is noted. In contrast, column 3 shows process is associated with a significant improvement in delta SF-12-PCS (p =.014) with instrumental variables, Stage 2. We repeated the analysis with the subset of the 120 explicit measures that are included in HEDIS scores (23 measures) and then again with the subset of measures (27 measures) meeting criteria for measures with Grade A evidence based upon randomized trials and found comparable results with virtually no difference in either the coefficient or the p-value for the instrumental variables model.
Using the instrumental variables model to estimate effect size, we note an improvement of 4.24 points in SF-12-PCS from 1996 to 1998 as process changes from the first (worst) quartile to the third quartile of process scores, of 2.21 as process changes from the worst quartile to the median value, and of 2.03 SF-12 points as process changes from the median to the fourth (best) quartile of process.
We observed a mean decrement of 1.20 (SD 9.83) in SF-12-PCS scores across the study's 30-month abstraction window (Table 1). One quarter of the patients dropped their score by 6 points, and 10 percent of patients dropped by 14 points. Almost 25 percent of the cohort dropped their SF-12-PCS by more than 5 (1/2 of the SD). The estimated effect size of 4.24 SF-12 points noted in association with a change in process from the first to the third quartile of process is associated with a substantial improvement in relation to the decrement observed for the cohort after aging 2.5 years. These data suggest the application of better process of care to patients currently receiving poor process would alter their SF-12-PCS change scores in a manner comparable to the eradication of aging three years.
Two important findings emerge from this work. Patients with more burden of illness had better process scores; and patients with better process of care sustain better health-related quality of life outcomes. Challenges in finding a link between better process and better patient outcomes have long been recognized (Sperl-Hillen et al. 2000; Kerr et al. 2001; Leatherman et al. 2003). Clinicians put patients in a higher venue of care (e.g., intensive care) to increase the clinician–patient ratio and facilitate the patient receiving a greater proportion of their many needed services. Yet, every clinician also knows that patients in the intensive care unit are more likely to die than patients cared for elsewhere. This counter–clinical relationship between the delivery of more needed process and higher death rates highlights the problem others and we faced in revealing the expected link between process and outcomes.
We postulated a three-stage clinically plausible sequence to account for the counter–clinical findings of worse health-related quality of life for patients with better-measured process. First, incentives exist to deliver needed process at higher rates for sicker as compared with less sick chronically ill patients. Second, patients with more measured burden of illness also had more unmeasured burden of illness (despite the rich set of burden of illness measures derived both from the patient and the medical record). Third, we postulate a correlation between more unmeasured burden of illness and process, as well as a correlation between more unmeasured (and measured) burden of illness and worse outcomes. We postulate this latter relationship is responsible for the observed counter–clinical OLS regressions which showed better process was associated with worse outcomes. We think the unmeasured (and measured) severity predict worse outcomes. However, as more burden of illness is associated with more process, we observe that more measured process (actually reflecting more burden of illness) is associated with worse outcomes. With this ongoing challenge, it is no wonder that researchers have struggled to find a relationship between process and outcomes. In contrast, clinicians note the dynamic nature of patient's burden of illness and how good patient care is defined by ongoing responsiveness to each patient's ever changing clinical need.
The reversal of the counter–clinical finding (that more process was associated with worse outcomes using OLS) with the instrumental variables approach, as well as the significance of the augmented test, supports our decision to use instrumental variables. Instrumental variables allow us to put more realistic standard errors around the relationship between burden and process.
We found evidence that providers consider different dimensions of burden of illness as they decide whether or not to intervene with specific recommended processes. Use of the instrumental variable model allowed us to account for the ways in which clinicians consider unique components of burden of illness as they approach each clinical process decision. The instrumental variables model closely represents clinical practice by adjusting providers' process scores uniquely for each criterion as a function of three dimensions of burden. This method revealed a statistically significant relationship between better process and health-related quality of life outcomes that patients' value so highly.
These data suggest providers, systems, and patients are rising to the challenges associated with the care of sicker patients. We observe the proportion of needed care that is delivered is greater for sicker patients than for patients with less comorbidity, less severity, or less obesity. We do not know whether providers, patients, or organizational structure is the main determinant of why patients with more (versus less) burden of illness receive a greater proportion of needed services than patients with less burden of illness. However, as process criteria are constructed to measure use of an intervention believed to improve outcomes for a selected set of patients (as defined by explicit conditional logic), good quality of care should deliver comparable rates of adherence to explicit process measures for patients regardless of burden of illness.
Advances in process for very sick patients are to be valued and emulated. In contrast, low process score for chronically ill patients who have not yet demonstrated major decline represents a missed opportunity pertinent to underuse of primary, secondary, and tertiary preventive strategies. This finding of a strong relationship between burden of illness and processes should stimulate efforts to improve the quality of care for patients regardless of whether they are acutely or chronically ill; both instances provide important opportunities to improve patient outcomes. Better understanding of the determinants of higher adherence rates for sicker patients may provide a valuable clue to the reorganization of the current health care system.
This demonstration that ambulatory process of care is a significant predictor of changes in health-related quality of life across 2 years should reassure patients, providers, and those involved with health care delivery that the net result of better process is realized by patients in terms that matter to them. The estimated effect size from changing process from a moderate (50th percentile) process to the next best quartile of process (75th percentile) was found to be associated with an improvement in physical health, roughly equivalent to the decrement observed with aging from 1996 to 1998. The application of the best quartile of process of care to patients currently receiving poor process is associated with a 4.24-point increment in SF-12-PCS change scores comparable to a change in function from New York Heart Association Class II to III or III to IV (Bennett et al. 2002). Our results suggest the delivery of evidence-based processes of care will greatly benefit patients by improving health-related quality of life.
Because the conduct of processes is cost-sensitive, it is important to develop methods for understanding the evaluation of the link between processes and outcomes. This analysis is robust across a number of model assumptions suggesting it can provide the basis for important future analyses that will evaluate costs associated with observed improvements in outcomes, as a function of process.
Demonstrating a clinically important and statistically significant link between better process of care and better health status highlights the importance of clinicians delivering good process. For people interested in process measurement, this analysis provides a process-outcome link, traditional evidence that process matters. For those interested in outcomes measurement, identification of a substantial process-outcome link reinforces the need for taking specific actions to improve outcomes.
We need to explore how these lessons apply to patients across the burden of illness spectrum. Regardless of whether the incentive systems focus on outcomes or process measures, the actions of potential recipients of incentives will be to try to improve process of care. A better understanding of how patient characteristics such as burden of illness vary according to challenges providers face in delivering good process is likely to affect how providers and/or patients will be able to respond to incentives. This may advance our understanding of how organizational structure, process, and outcomes fit together even further.
This should reassure those with concerns about the value of measuring process of care for patients. Having established a link between process and health related quality of life outcomes, providers can use this analysis as both a motivation and a challenge to provide better process.
The analysis of the relationships between burden of illness, process, and process and outcomes was supported by grants from the Agency for Healthcare Research and Quality, American Association of Health Plans, and Robert Wood Johnson Foundation. The Pacific Business Group on Health supported baseline data collection activities. Dr. Diana Tisnado was supported by a Ruth L. Kirschstein National Research Service Award (Training grant number T32-HS00046).
We are indebted to Dr. Arnold Milstein for thoughtful reviews of early drafts of this manuscript, to Sarah Gee for research assistance, and to Corinna Koehnenkamp for manuscript preparation.
Disclosures: There are no disclosures or conflicts of interest.