The sample is drawn from HealthCare for Communities (HCC), which is part of the Robert Wood Johnson Foundation's Health Tracking Initiative. Data are from the HCC phone survey, which was conducted during 1997–1998. The design of HCC is described in detail elsewhere (Sturm et al. 1999
). The HCC respondents were a stratified probability sample of participants in the Community Tracking Study (CTS), a nationally representative study of the U.S. civilian, noninstitutionalized population during 1996–1997 (Kemper et al. 1996
). The CTS includes a sample clustered within 60 randomly selected U.S. communities and a national sample. To improve power for analyses, HCC oversampled the following CTS respondent groups: the national sample, individuals with family income less than $20,000, individuals with high psychological distress on a scale consisting of two items from the 12-item Short Form Health Questionnaire (SF-12) (Ware, Kosinski, and Keller 1996
) and individuals with any mental health specialty use during the prior year. Of the 14,985 respondents selected for HCC, 9,585 were reinterviewed for a response rate of 64 percent. These analyses focus on the sample of individuals with any anxiety or depressive disorder (n
=1,290) and any alcohol, drug, or mental (ADM) disorder (n
=1,943). In analyses, we weight the data to be representative of the population of the United States by using CTS data to adjust for the probability of selection, unit nonresponse, and nontelephone households. The sociodemographic profile of our sample is shown in .
We used a five-level scale of overall satisfaction with the mental health care available for personal or emotional problems during the past 12 months. Because not all individuals with disorders received treatment, this variable reflects experience with treatment and respondents’ perceptions of potentially available treatment.
Appropriate Technical Quality of Care
In our analyses of the satisfaction–technical quality relationship among adults with a 12-month anxiety or depressive disorder, we used a previously described indicator of technical quality (Young et al. 2001
). Briefly, this indicator was developed for a hierarchy of disorders and was defined as use of either appropriate counseling or psychotropic medications during the prior year for a probable depressive or anxiety disorder. Appropriate counseling was defined as four or more visits with a mental health specialist or primary care provider that included counseling for mental health problems. Appropriate pharmacotherapy was based upon AHCPR and other published guidelines, and included parameters for the type of medication, dosage, and duration of treatment.
The above measure of appropriate technical quality approximates quality of care for individuals with depressive or anxiety disorders. However, it would be difficult to fit a similar measure of technical quality across a broad range of psychiatric disorders. Therefore, to study the satisfaction–technical quality relationship among adults with any 12-month ADM disorder we developed a broader indicator of technical quality, active treatment. This variable measures active treatment, beyond assessment, for an ADM disorder in the past year including: use of inpatient, day treatment, or residential care; use of prescribed psychotropic medications daily for a month or more; or a period of potentially therapeutic outpatient treatment for ADM conditions, such as four or more outpatient visits or visits to a provider trained in counseling methods, improving skills in relationships or coping with loss, teaching ways to relax, encouraging enjoyable activities or taking responsibility for substance abuse problems, or teaching how to avoid recurrences. While the use of a broader definition of quality could potentially decrease the strength of the association between satisfaction and quality, it would lead to greater statistical power and more precise estimates.
Psychiatric morbidity covariates included measures of probable 12-month mental health disorders; major depression, dysthymia, generalized anxiety disorder, and panic disorder were assessed (Kessler et al. 1998
) using short-form versions of the World Health Organization's Composite International Diagnostic Interview (CIDI) (World Health Organization 1995
). Lifetime mania corresponded to a positive score on the CIDI stem, and psychosis corresponded to a report of ever having had an overnight hospital stay for psychotic symptoms or having received a diagnosis of schizophrenia from a physician (Sturm et al. 1999
). Probable 12-month substance use/abuse disorders were assessed using the AUDIT (World Health Organization 1992
). Two mental health composites were included in our analyses. The MCS12 uses items from the Short Form Health Questionnaire (SF-12) (Ware, Kosinski, and Keller 1996
); the MHI-5 contains additional items on the respondent's mental health status in the past four weeks.
Indicator variables were included for one, two, or more chronic conditions, as well as the PCS12, an aggregate measure of physical health functioning from the 12-item Short Form Health Questionnaire (SF-12) (Ware, Kosinski, and Keller 1996
Other covariates include: gender, income (0–15k, 15–30k, 30–50k, 50–70k, 70–100k, and greater than 100k), number in household, insurance status, race (non-Hispanic White, African American, Hispanic, other), employment status (employed, unemployed, not working and not looking for employment), age (between 18 and 45, greater than 45), education (not high school graduate, high school graduate, some college, college graduate), and marital status (married or living with partner, single).
Our first set of models used the subsample of individuals with a 12-month anxiety or depressive disorder, excluding individuals with lifetime mania or psychosis. We first tested the association between technical quality of care and satisfaction in a bivariate model. We then investigated the possibility of selection bias by adding the covariates previously described to the model. Selection bias is suggested by both a large difference in the magnitude of technical quality coefficient across the two regression equations and statistical significance of the additional covariates. In our second set of models we performed a parallel set of analyses using the sample of individuals with a 12-month anxiety, depressive, or substance disorder, or lifetime mania or psychosis to investigate the relationship between satisfaction and active treatment.
We used ordered multinomial logistic regression techniques (Hosmer and Lemeshow 2000
). Similar to logistic regression, statistical significance occurs when the odds ratio (OR) differs statistically from 1.00. However, in multinomial logistic regression the OR represents the increased (OR>1) or decreased odds (OR<1) of reporting a higher level of satisfaction (e.g., satisfied versus very satisfied) that is associated with a one-point increase in the independent variable. For example, for dichotomous variables (e.g., female=0 or 1), the OR represents the odds of a woman reporting a higher or lower level of satisfaction, relative to a man. When the range of values for the independent variable is larger, as with the physical and mental health components of the SF-12 (approximate range 15–66) and MHI-5 (range 0–100), the effect of a one-unit change and the magnitude that the OR deviates from 1.00 is relatively smaller, although statistical significance is not affected by the scale of the metric. Estimates were calculated using SUDAAN
software, version 8.0.0, and take into account the complex survey design and survey clustering (SUDAAN 2001
). The degrees of freedom in the denominator of our F-tests differ for covariates due to our variance corrections for imputed data.
A large majority of respondents with an anxiety or depressive disorder were either satisfied or very satisfied with the care available to them for mental health problems (). In a bivariate model among individuals with an anxiety or depressive disorder, technical quality of care was a significant predictor of satisfaction (odds ratio=1.55, t=2.11, p=0.04) (). The OR in the full model was slightly increased and more highly significant (OR=1.64, t=2.51, p=0.01). Measures of psychiatric morbidity (F-stat (8, 323)=2.08, p=0.04) and employment status (F-stat (2, 329)=3.63, p=0.03) were also significant in the full model. Other covariates including gender (t-stat=0.70, p=0.48), age (t-stat=1.28, p=0.20), income (F-stat (5, 227)=1.77, p=0.12), number in household (t-stat=0.66, p=0.51), race (F-stat (3, 328)=1.69, p=0.17), education (F-stat (3, 328)=0.33, p=0.81), insurance (t-stat=0.99, p=0.32) and physical health (F-stat (3, 328)=0.04, p=0.99) were not significant.
The Relationship between Appropriate Quality of Care and Patient Satisfaction for Respondents with a 12-Month Depression or Anxiety Disorder (n=1,290)
A broader definition of quality, active treatment, was also significantly associated with satisfaction (OR=1.40, t=2.60, p=0.01) among respondents with any ADM disorder (). The OR was slightly increased in the adjusted model (OR=1.56, t=3.04, p=0.003). In this model, income (F-stat (5,200)=5.54, p<0.001), and psychiatric morbidity (F-stat (10, 488)=2.75, p=0.003) were also significantly associated with satisfaction, while gender (t-stat=0.94, p=0.35), age (t=1.96, p=.0504), number in household (t=0.06, p=0.95), employment status (F-stat (2, 496)=2.96, p=0.053), race (F-stat (3, 495)=2.21, p=0.09), education (F-stat (3, 495)=1.88, p=0.13), insurance status (t=1.53, p=0.13), and physical health status (F-stat (3, 495)=0.17, p=0.91) were not significant.
The Relationship between Active ADM Treatment and Patient Satisfaction for Respondents with Any 12-Month Disorder (n=1,943)
Satisfaction measures have their own intrinsic merit as measures of consumer preference. Satisfaction is also theoretically a relatively easy way to obtain inexpensive markers of technical quality of care. However, it has been suggested patients have difficulty distinguishing technically appropriate care (Welton and Parker 1999
; Pascoe 1983
), and concerns have been raised about using satisfaction as a marker for technical quality of care (Cleary and McNeil 1988
We have shown that satisfaction with available mental health services is significantly associated with two measures of appropriate care. Further, the association between satisfaction and quality was more robust to model specification than we anticipated. The strength of the association between satisfaction and appropriate technical quality of care was comparable in a bivariate model and in a model that included measures of mental and physical health and sociodemographic status. The broader definition of quality, active treatment, was also significantly associated with satisfaction in both unadjusted and adjusted models, although the strength of the relationship was slightly stronger in the adjusted model. Thus, although our measures of psychiatric health were significantly associated with satisfaction, their exclusion from the model did not lead to substantive changes in the quality coefficients. However, we believe it would be premature to state that case-mix adjustment is not needed in these types of analyses. Indeed, defining the relevant groups for analyses is a type of case-mix adjustment. Further study with other samples and other groups of disorders is necessary to clarify the utility of case-mix adjustment.
While the strength of the satisfaction–technical quality association was only moderate, our estimates might be larger with more perfect measures of technical quality or mental health status. Our case identification relied on a brief diagnostic screening instrument, which has been found to be highly specific but not very sensitive (Murphy et al. 2000
), and due to the constraints of population surveys we were not able to control for all potentially relevant mental health disorders. Our measure of appropriate counseling, at least five minutes of counseling in four different sessions, is unsophisticated and contains no information on session content or process. While the definition for appropriate counseling has been shown to affect outcomes for depressive disorders in the Medical Outcomes Study, it has not been studied for anxiety disorders (Sturm and Wells 1995
). We assessed technical quality through self-report of use of specific services; however, Katon et al. (1996)
found moderate associations between self-reports of medication use and administrative data in an HMO (Katon et al. 1996
). Due to these limitations, we feel that our estimate of the strength of the satisfaction–technical quality relationship should be viewed as a theoretical lower bound, although in practice it will be difficult for any health plan or provider to implement more sophisticated methodologies.
One other data limitation deserves comment. While satisfaction with care is multidimensional, our study used only a one-item global measure of satisfaction. However, this is likely to mirror “real world” applications of satisfaction measures where the mental health items in any general survey of satisfaction with health care will be limited. For example, the core CAHPS questionnaire has no mental health care items, and only three items in the optional supplement.
Our study demonstrates that technical quality of care correlates with satisfaction, but what does this mean? One explanation for our findings is that individuals are able to distinguish quality care for common mental disorders, but there are other plausible explanations (Orlando and Meredith 2002
). For example, providers might be more apt to give quality care to individuals who are more easily satisfied, or clinicians who provide appropriate technical quality of care might also have better interpersonal skills, leading to greater patient satisfaction. Therefore, to better understand the causes of the quality–satisfaction correlation will require a more detailed investigation of the complex clinician–patient dynamic.
If the strength of the satisfaction–technical quality association using currently feasible techniques is only moderate, attempts to differentiate technical quality among health plans or provider groups using satisfaction as a proxy will likely require a large number of observations or large differences in satisfaction between plans. The robust relationship between satisfaction and quality is encouraging. However, our findings may not generalize to populations that were not well represented in this study, such as individuals with schizophrenia or bipolar disorder. Case-mix adjustment will still be necessary to define the relevant groups. Because estimation of technical quality using satisfaction as a proxy measure will likely require large samples and case-mix adjustment, it may be more difficult for plans or provider groups to implement than directly using indicators of quality. More importantly, satisfaction is not the same as technical quality, and our results suggest that at this time they cannot be made to approach each other closely enough to eliminate either.