|Home | About | Journals | Submit | Contact Us | Français|
Providing culturally competent care shows promise as a mechanism to reduce healthcare inequalities. Until the recent development of the CAHPS Cultural Competency Item Set (CAHPS-CC), no measures capturing patient-level experiences with culturally competent care have been suitable for broad-scale administration.
We performed confirmatory factor analysis and internal consistency reliability analysis of CAHPS-CC among patients with type 2 diabetes (n=600) receiving primary care in safety-net clinics. CAHPS-CC domains were also correlated with global physician ratings.
A 7-factor model demonstrated satisfactory fit (χ2(231)=484.34, p<.0001) with significant factor loadings at p<.05. Three domains showed excellent reliability – Doctor Communication- Positive Behaviors (α=.82), Trust (α=.77), and Doctor Communication- Health Promotion (α=.72). Four domains showed inadequate reliability either among Spanish speakers or overall (overall reliabilities listed): Doctor Communication- Negative Behaviors (α=.54), Equitable Treatment (α=.69), Doctor Communication- Alternative Medicine (α=.52), and Shared Decision-Making (α=.51). CAHPS-CC domains were positively and significantly correlated with global physician rating.
Select CAHPS-CC domains are suitable for broad-scale administration among safety-net patients. Those domains may be used to target quality-improvement efforts focused on providing culturally competent care in safety-net settings.
Culturally competent care has been defined as “the ability of health care providers and institutions to deliver effective services to racially, ethnically and culturally diverse patient populations.”(1) Cultural competence is associated with better interpersonal processes of care(2) and has been identified as a promising tool in combating racial/ethnic and socioeconomic inequalities in healthcare.(3–6) While the utility of culturally competent care has been extensively theorized,(7–9) little empiric research links cultural competence to clinical outcomes.(3, 10, 11) The relative paucity of outcomes research related to culturally competent care has made health plans reticent to develop comprehensive strategies to improve cultural competence.(12) A significant barrier in linking culturally competent care to outcomes is the lack of quantitative, patient-centered culturally competent care measures that can be used in diverse settings. To date, culturally competent care research has measured physician knowledge and attitudes rather than patients’ assessments of the cultural competence of the care they receive.(5, 13) The Consumer Assessment of Healthcare Providers and Systems Cultural Competency Item Set (CAHPS-CC) captures patients’ assessments of the degree to which their care is culturally competent, reasoning that patient assessments will be more closely linked to clinical outcomes than physician assessments.(14)
CAHPS-CC is intended as a supplement to the Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys. The Agency for Healthcare Research and Quality (AHRQ) developed CAHPS as a patient-administered assessment of healthcare quality in diverse settings.(15) Medicare managed care plans are required to report CAHPS data to the federal government as a quality of care metric. Low CAHPS scores have been associated with poor medication adherence,(16) increased emergency room use,(17) and lower utilization of colon cancer screening.(18) While CAHPS has been extensively validated among insured patients, it is less clear how it performs with ethnically diverse, uninsured, safety net, and other vulnerable populations.(19, 20) Adding a set of culturally competent care domains makes CAHPS more relevant to these populations.
CAHPS-CC was originally designed for and psychometrically evaluated in an insured population.(21) Given the high prevalence of uninsured and underinsured minority populations in the US, we evaluated its performance in an uninsured/underinsured sample of patients with type 2 diabetes. Provision of culturally competent care may be particularly important among patients with chronic diseases such as diabetes, where treatment requires patient engagement, competent self-management, and an ongoing relationship between patient and clinician.
CAHPS-CC was administered in participants’ preferred language (English or Spanish) as part of the Immigration, Culture, and Health Care (ICHC) study. ICHC was a cross-sectional study investigating factors that impact diabetes self-management and health outcomes in vulnerable populations. The study included African American, Spanish- and English-speaking Mexican American, and non-Latino white adults with diabetes who received primary care in safety-net clinics in Chicago and the San Francisco Bay Area. Eligible patients were adults (age 18 years or older) with chart-confirmed type 2 diabetes. Exclusion criteria, assessed by trained interviewers prior to study enrollment, included cognitive impairment, active substance abuse, or psychosis severe enough to interfere with survey administration. The ICHC survey and CAHPS-CC component were administered during face-to-face interviews. The Institutional Review Boards of the University of California, San Francisco, the Cook County Health and Hospital System, and other participating clinics approved all study activities.
CAHPS-CC is a multidimensional set of 26 questions designed to measure patients’ overall experience of their physician’s interpersonal and cultural competence as well as their experience of their physician’s office. As Weech-Maldonado et al describe in a companion manuscript,(21) literature review and examination of cultural competence theory yielded 5 domains of cultural competence: 1) Patient-provider communication; 2) Respect for patient preferences/ shared decision-making; 3) Experiences leading to trust or distrust; 4) Experiences of discrimination; and 5) Language services. In order to develop the 26 survey items, Weech-Maldonado et al gleaned relevant items from the existing CAHPS survey, from other cultural competency instruments in the literature and public domain, and from a Federal Register call for measures. Items then underwent extensive cognitive interviewing, field testing, and rigorous translation.(21) Exploratory factor analysis of the original items - conducted on a sample of patients insured through the Medicaid program - suggested that CAHPS-CC had a first order factor structure comprised of 8 domains: Doctor Communication- Positive Behaviors – taken directly from the existing CAHPS survey (5 items), Doctor Communication- Negative Behaviors (3 items), Trust (5 items), Equitable Treatment (2 items), Doctor Communication- Health Promotion (4 items), Doctor Communication- Alternative Medicine (2 items), Shared Decision Making (2 items) and Access to Interpreter Services (3 items). Given the multidimensional, first order factor structure, CAHPS-CC was thought to represent several separate domains of cultural competence, rather than measuring a single cultural competence construct.
We also used a measure of global physician rating to evaluate the validity of the CAHPS-CC adjusting for potential confounders. The global physician rating asks participants to rate their primary care doctor using any number from 0 to 10, where 0 is the “worst possible” doctor and 10 is the “best possible” doctor.(15) This item is a widely reported patient satisfaction measure and is associated with healthcare quality(22) and some clinical outcomes.(23) Measured confounders included age (18–34, 35–64, or 65+ years), gender, race/ethnicity (non-Latino white, African American, or Mexican American), language (English or Spanish), self-reported health status (Excellent/very good/good vs. fair/poor/very poor), and depressive symptoms. We considered scores ≥10 on the Patient Health Questionnaire (PHQ-9) to be consistent with depressive symptoms.(24) The PHQ-9 is a nine-item scale which asks respondents how often over the previous two weeks they have been bothered by each of the nine symptom clusters in the Diagnostic and Statistical Manual’s (Fourth edition) criteria for diagnosis of depression.
CAHPS CC domain scores were calculated by linearly transforming items to a 0–100 scale (with a higher score representing more favorable assessments), and then averaging items within each domain. We performed a confirmatory factor analysis using a weighted least squares estimator with a mean and variance adjustment (Mplus 6.12, estimator WLSMV) to assess the validity of the hypothesized factor structure.(25) We assessed global model fit using the chi-square test of exact model-data fit, the comparative fit index (CFI), root mean square error of approximation (RMSEA), and weighted root mean square residual (WRMR). Ideal values for these tests are as follows: chi-square test of exact model fit, p > .05; CFI ≥ 0.95; RMSEA ≤ 0.06; and WRMR ≤ 1.00. Because the chi-square test of exact model fit may be sensitive to trivial departures of model-data fit, we followed the recommendations of Hu and Bentler by a priori accepting the confirmatory factor analysis results if two of the three descriptive statistics (CFI, RMSEA, WRMR) met the desired fit criteria.(26)
We used SAS 9.2 to measure the internal consistency reliability of the factors identified in the confirmatory factor analysis using a Cronbach’s alpha statistic. (27, 28) We considered a Cronbach’s alpha of 0.70 indicative of adequate reliability.(29)
We then evaluated the relative importance of the CAHPS-CC domains on perceptions of care by measuring the variance in global physician ratings attributable to culturally competent care. We trichotomized global physician rating (0–4, 5–8, 9–10) in accordance with CAHPS recommendations(30) and previous studies of global physician rating in diverse populations.(31) We calculated the adjusted pseudo-r2 and odds ratios from an ordinal logistic regression model to determine the relative contribution of each of the CAHPS-CC domains to the global physician rating and adjusted for age, gender, race/ethnicity, language, self-reported health status and depressive symptoms. These analyses were performed in SAS 9.2.
Of 782 eligible patients approached, 91% consented to participate in the study (n = 711). For this analysis we excluded 111 patients who could not identify a primary care provider, resulting in an analytic sample of 600 participants. The sample was predominantly of Mexican descent (53%), low-income (65% uninsured or Medicaid only), and middle-aged (Table 1). Excluded participants were more likely to be of Mexican descent (77.5% vs 53%; p<.0001), young (mean age 50 vs 55 years; p<.001), and fluent in Spanish (36.9% vs 25.3%; p=.01).
We were unable to evaluate the Access to Interpreter Services domain because few participants reported using interpreter services (n=36). We therefore conducted a confirmatory factor analysis of the 7 remaining domains. Sample size, means and standard deviations for each domain can be found in Table 2. The chi-square test of exact model data-fit rejected the null hypothesis of exact fit (χ2(209) = 450.92, p < .0001). However, two of the three descriptive model fit statistics indicated acceptable approximate fit (CFI =.96, WRMR = 1.07, and RMSEA=.04) suggesting that the 7-factor model performed well in our sample. Standardized factor loadings for all items were significant at p < 0.001. One item exhibited a negative variance estimate, a common problem among factors with a small number of items. Re-specifying the model by setting the residual variance of this item to a small positive value did not substantially change the results.(32) The correlations among latent constructs exhibited a wide range from essentially zero (r = .004) to strongly correlated (r = .77) (Table 3).
We used Cronbach’s alpha statistics to assess the internal consistency reliability of each factor (Table 2). Three of the 7 factors had adequate to excellent reliability (Doctor Communication- Positive Behaviors, α =0.82; Trust, α =0.77; and Doctor Communication- Health Promotion, α =0.72). Reliability was lower than 0.70 for the Doctor Communication- Negative Behaviors, Equitable Treatment, Doctor Communication- Alternative Medicine, and Shared Decision-Making domains.
Based on our previous work, in which the performance of some scales measuring trust in healthcare differed in English- and Spanish-speaking populations(33), we stratified the internal consistency reliability analysis by language (Table 2). The internal consistency reliability for Doctor Communication- Negative Behaviors and Equitable Treatment was substantially worse among Spanish speakers than English speakers (α=.45 vs α=.57 and α=.35 vs α=.75 respectively).
We examined the association of each of the 7 CAHPS-CC domains with global physician rating using an ordinal logistic regression adjusted for age, gender, race/ethnicity, language, self-reported health status and depression (Table 4). The Doctor Communication- Positive Behaviors and Trust domains exhibited the largest associations with global physician ratings, with each one point increase in Doctor Communication- Positive Behaviors associated with an 8% (95%CI 7–10%) increase in odds of rating one’s physician favorably and each one point increase in Trust associated with a 7% (95%CI 6–8%) increase in odds of rating one’s physician favorably (Table 4).
The confirmatory factor analysis of CAHPS-CC demonstrated that a 7-factor model performed well in a sample of predominantly uninsured and underinsured low-income adults with diabetes. This finding is consistent with the factor analysis of CAHPS-CC conducted among non-elderly enrollees in two Medicaid managed care programs in California and New York.(21) The robustness of the 7-factor model suggests that CAHPS-CC is able to capture multiple aspects, or domains, of the multidimensional concept of “cultural competency”.
All CAHPS-CC domains were significantly and positively associated with global physician rating, with a single point increase in scores on two domains (Doctor Communication – Positive Behaviors and Trust) each explaining a 7–8% change in odds of higher global physician rating. This finding provides evidence that the domains of culturally competent care measured by CAHPS-CC are related to patient experience of physician quality, a finding also supported by the work of Weech-Maldonado and colleagues in an insured population.(21) Our analysis may underestimate the magnitude of the effect of cultural competency (as measured by CAHPS-CC) on patient experience of physician quality, as much of the variance in global physician rating in our sample was at the higher end of the scale.
The Doctor Communication- Positive Behaviors, Trust, and Doctor Communication- Health Promotion domains of CAHPS-CC showed acceptable internal consistency reliability among English- and Spanish-speakers. However, other CAHPS—CC domains showed poor internal consistency reliability or differed in internal consistency reliability.(33) Of the four domains with low reliability, three (Equitable Treatment, Doctor Communication- Alternative Medicine and Shared Decision-Making) were two item scales. Cronbach’s alpha is dependent on scale length and may underestimate reliability for two item scales. Further research is necessary to more adequately understand the performance of these small scales in underinsured and uninsured populations, particularly since the internal consistency reliability appears stronger for these two item domains in insured populations.(21) In addition, the internal consistency reliability of the Shared Decision-Making domain may have been affected by its small sample size (n=204). The skip pattern structure of the ICHC survey excluded all respondents who reported that in the last 12 months their doctor did not “tell you there was more than one choice for your treatment or health care.”
Unlike the findings of Weech-Maldonado et al, we found relatively low internal consistency reliability for the Doctor Communication – Negative Behavior domain. The reasons for this discrepancy are unclear. Additional studies are needed to understand whether the low internal consistency reliability we observed is an isolated finding or related to the particular sample in which we conducted our study.
Carle et al. used multiple group confirmatory factor analyses to examine differences in the performance of CAHPS-CC across language (English and Spanish). They found general support for the equivalence of the English and Spanish versions of CAHPS-CC for the Doctor Communication-Positive Behaviors, Doctor Communication-Negative Behaviors, Doctor Communication-Health Promotion, Equitable Treatment, and Trust domains. Observed measurement bias was generally small and did not substantively impact findings for any of the domains. In contrast, we found that the internal consistency reliability estimates for the Equitable Treatment, Doctor Communication- Negative Behaviors, and Doctor Communication- Alternative Medicine domains were substantially lower among Spanish speakers than among English speakers. The difference we observed in the internal consistency reliability estimates for the Equitable Treatment domain among Spanish speakers (alpha = 0.35) and English speakers (alpha = 0.75) is particularly striking. Differential findings by language may be due to translation, cultural orientation of Spanish-speaking participants, or another unique characteristic of this population. Spanish-speaking patients may experience difficulty distinguishing whether inequitable treatment in the healthcare setting stems from their language, race, or health insurance coverage. In these individuals, Equitable Treatment questions may more logically function as individual items rather than a scale.
Because many of the physicians at the institutions where our survey was conducted are fluent Spanish speakers and monolingual speakers of other languages were excluded, we were unable to evaluate the Access to Interpreter Services domain. However, given the impact of interpretation services on clinical outcomes,(34, 35) we believe it is important that a cultural competency assessment include assessment of access to interpreter services.(36) Previous efforts to add interpreter access questions to CAHPS have also faltered due to insufficient sample size.(37) Additional psychometric analysis of this domain should be conducted in samples specifically recruited to examine access to interpreter services.
This study is limited to an assessment of the internal consistency reliability of CAHPS-CC. Future studies using larger samples must assess other forms of reliability, including test-retest reliability and inter-rater reliability. In addition, we administered the survey orally because of limited literacy skills of many patients in our population. Because CAHPS is traditionally self-administered, further testing is necessary to determine the acceptability and reliability of self-administering CAHPS-CC domains in a safety-net population.
Although our sample was ethnically and linguistically diverse, other large population groups represented in the US (e.g., Latinos of non-Mexican descent, Asian Americans) were not included. Our results cannot be generalized to patients of racial, ethnic or linguistic groups not represented in our sample. There were significant differences between our included and excluded participants by race/ethnicity, language, and age. However these differences likely reflect the demographics of patients with access to primary care in the San Francisco Bay Area and Chicago.
In summary, our study supports the 7-factor model structure of CAHPS-CC and suggests that these factors are important contributors to patients’ overall assessments of provider quality. Internal consistency reliability estimates were high in the Doctor Communication – Positive Behavior, Trust, and Doctor Communication - Health Promotion domains, but lower for the other domains. Further research is necessary to understand the discrepancies in internal consistency reliability estimates in our sample compared to Weech-Maldonado et al’s sample.
Culturally competent care is thought to be a critical component of patient satisfaction and high quality care,(38) yet few patient-centered measures of cultural competence exist(12). CAHPS-CC has been proposed to examine patients’ perceptions of the cultural competence of their care, investigate the relationship of culturally competent care to clinical outcomes, and to facilitate the development of cultural competence policies for health plans. It may also provide a metric for quality improvement interventions. Our data support these applications in the safety net setting for the three domains with acceptable internal consistency reliability: Doctor Communication – Positive Behavior, Trust, and Doctor Communication - Health Promotion. The hope is that broad-scale dissemination of CAHPS-CC with CAHPS will enable improvements in healthcare for racially and ethnically diverse patient populations, including members of the safety-net.
Funds were provided by the Russell Sage Foundation and the Commonwealth Fund.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.