PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Ethn Dis. Author manuscript; available in PMC Oct 17, 2012.
Published in final edited form as:
Ethn Dis. 2009 Autumn; 19(4): 447–453.
PMCID: PMC3474598
NIHMSID: NIHMS405811
A MIXED-METHODS APPROACH TO DEVELOPING A SELF-REPORTED RACIAL/ETHNIC DISCRIMINATION MEASURE FOR USE IN MULTIETHNIC HEALTH SURVEYS
Salma Shariff-Marco, PhD, Gilbert C. Gee, PhD, Nancy Breen, PhD, Gordon Willis, PhD, Bryce B. Reeve, PhD, David Grant, PhD, Ninez A. Ponce, PhD, Nancy Krieger, PhD, Hope Landrine, PhD, David R. Williams, PhD, Margarita Alegria, PhD, Vickie M. Mays, PhD, Timothy P. Johnson, PhD, and E. Richard Brown, PhD
Cancer Prevention Fellowship Program, Office of Preventive Oncology, National Cancer Institute and Applied Research Program, Division of Cancer Control and Population Sciences, National Cancer Institute (SSM), Applied Research Program, Division of Cancer Control and Population Sciences, National Cancer Institute (NB, GW, BBR), School of Public Health, University of California, Los Angeles (GCG, NAP, VMM, ERB), University of California, Los Angeles Center for Health Policy and Research (DG, NAP, ERB), Harvard School of Public Health (NK, DRW), American Cancer Society (HL), Harvard Medical School and Cambridge Health Alliance (MA), University of Illinois Chicago (TPJ), Center for Research, Education, Training and Strategic Communication on Minority Health Disparities, University of California, Los Angeles (VMM)
Address correspondence and reprint requests to Salma Shariff-Marco, PhD; Health Services and Economics Branch, Applied Research Program; Division of Cancer Control and Population Sciences; National Cancer Institute; 6130 Executive Blvd, EPN-4009C, MSC 7344; Bethesda, MD 20892-7344; 301-435-4986; 301-435-3710 (fax); shariffs/at/mail.nih.gov
Objective
The development of measures of self-reported racial/ethnic discrimination is an active area of research, but few measures have been validated across multiple racial/ethnic and language groups. Our goal is to develop and evaluate a discrimination measure that is appropriate for use in surveys of racially and ethnically diverse populations.
Methods
To develop our measure, we employ a mixed-methods approach for survey research, drawing from both qualitative and quantitative traditions, including literature review, cognitive testing, psychometric analyses, behavior coding as well as two rounds of field testing using a split-sample design. We tested our new measure using two different approaches to elicit self-reported experiences of racial/ethnic discrimination.
Results
Our new measure captures four dimensions of racial/ethnic discrimination: 1) frequency of encounters with discrimination across several domains (e.g., medical care, school, work, street and other public places); 2) timing of exposure (e.g., recent, lifetime); 3) appraisal of discrimination as stressful; and 4) responses to discrimination.
Conclusions
Because of the growing interest in measurement of racial/ethnic discrimination in health surveys, we think this report on the methods informing the development and testing of the discrimination module that will be used on the California Health Interview Survey would be useful to other researchers. The application of mixed methods to rigorously test the validity and reliability of our instrument proves to be a good roadmap for measuring racial/ethnic discrimination in multicultural and multilingual populations.
Keywords: Cross-Cultural Comparison, Discrimination, Survey Methods
Racial/ethnic discrimination is associated with negative health outcomes, ranging from depression to coronary calcification to mortality.14 Few measures of self-reported racial/ethnic discrimination have been validated across multiple racial/ethnic and language groups.5 Moreover, it is not clear that the experience of discrimination is qualitatively equivalent across different racial/ethnic and language groups. The growing multicultural demographic landscape of the United States and the documented relationship between discrimination and health outcomes highlight the need for new, brief measures that are valid across multiple populations and that can be used in health surveys.
The National Cancer Institute (NCI) is collaborating with the UCLA Center for Health Policy Research to improve discrimination measures in the California Health Interview Survey (CHIS). In this article, we describe our mixed-methods design for evaluating two approaches to elicit self-reported experiences of racial/ethnic discrimination as part of telephone health surveys.
In 2006, an NCI-led workgroup, called the CHIS Discrimination Module Workgroup, began developing an instrument for measuring self-reported racial/ethnic discrimination using the CHIS. Our goal was to develop and evaluate a discrimination measure that is appropriate for use in surveys of racially and ethnically diverse populations. The CHIS is a statewide, random digit-dial telephone health survey administered in the nation’s most ethnically- diverse state. The CHIS questionnaire was developed in collaboration between the CHIS staff and academic researchers, policymakers, funders and several standing committees, including the CHIS Multicultural Technical Advisory Committee (TAC). These features, and the fact that the CHIS is fielded in the nation’s most populous and linguistically and ethnically diverse state, make the CHIS an ideal vehicle for testing measurement issues of racial/ethnic discrimination. (www.chis.ucla.edu)
We drew from both qualitative and quantitative traditions for our mixed-methods approach. The CHIS DM Workgroup received input from the TAC as well as from an external group of advisors at multiple steps in the process. Each methodological step in our process of evaluation was led by an internal expert, reviewed by the entire workgroup, and then reviewed again by the panel of external advisors. The multiple methods used are shown in Figure 1 and summarized below:
Figure 1
Figure 1
Overview of the development of a self-reported racial/ethnic discrimination module for the California Health Interview Survey (CHIS)
  • Structured review of existing racial/ethnic discrimination scales and studies from the literature;
  • Secondary analyses of existing racial/ethnic discrimination items fielded from the National Institute of Mental Health-funded National Latino and Asian American Study and the National Survey of American Life;6 these analyses focused on the psychometric properties of the Everyday Discrimination Scale including the validity and reliability of the individual items and scale in multiracial/ethnic samples;
  • Cognitive testing (with primary data) to identify cross-cultural equivalence of item wording;
  • Field-testing two alternative ways of asking about racial/ethnic discrimination in English using the CHIS 2007, and analyses comparing the two approaches;
  • Behavior coding of a subset of field-test interviews to identify strategies to improve survey administration of the measures;
  • Cultural review of the items by cultural/linguistic experts to further improve cross-cultural item and translation equivalence;
  • Multi-step, team-based translation (i.e., team of translators working together instead of a single translator);
  • Field-testing two alternative ways of asking about racial/ethnic discrimination in English, Spanish, Vietnamese, Korean, Cantonese and Mandarin using the CHIS 2009, and comparing the two approaches; and
  • Feedback from members of the scientific community (e.g., external advisors, and CHIS Multicultural TAC).
Three well-known, validated instruments that measure a variety of dimensions and domains of discrimination were identified in our literature review as particularly relevant to our project. These were the Everyday Discrimination Scale (EDS),7 Experiences of Discrimination Scale,8 and General Ethnic Discrimination Scale.9 The first two lend themselves to telephone administration in omnibus public health surveys such as the CHIS because they are short and have been administered in various health surveys in multicultural and multilingual settings. The third instrument was not designed for brief telephone administration. However, this instrument was important in informing the types of dimensions that could be measured, including the appraisal of discrimination as being stressful. Other instruments were reviewed but not used due to their length or limited validation across multiple racial/ethnic groups.1024
Based on the review, the workgroup agreed that the new measure should capture four dimensions: 1) frequency of encounters with discrimination across a number of domains (e.g., medical care, school, work, street, other public places, police, and courts); 2) timing of exposure (e.g., recent, lifetime); 3) appraisal of discrimination as stressful; and 4) responses to discrimination. These decisions were supported by our panel of external advisors and the CHIS Multicultural TAC.
Subsequent to our literature review we conducted secondary and primary data analyses to evaluate and refine discrimination items from the EDS. The EDS had been administered in many studies, including the National Latino and Asian American Study (NLAAS) and the National Survey of American Life (NSAL), to three racial/ethnic groups (Blacks, Latinos and Asian Americans). We conducted factor analyses on the EDS, which had been conceptualized as a unidimensional scale in previous research.8,25 Our factor analysis confirmed that the EDS can be conceptualized as a unidimensional scale. We also found that two items (“You are treated with less courtesy than other people” and “You are treated with less respect than other people”) were highly correlated. Qualitative results from cognitive testing supported our hypothesis that the term respect is more meaningful to respondents than courtesy. Accordingly, the courtesy item was dropped. Descriptive analyses showed that few people reported experiencing frequent discrimination (at least once a week) and that Black participants reported experiencing the most unfair acts of any racial/ethnic group we studied. We found that the item, “You are called names or insulted,” functioned differentially for Asian Americans and Blacks. Differential item functioning was also found for the item, “People act as if they are afraid of you,” between men and women.2629
The cognitive testing revealed that the majority of items performed as intended. Respondents did not raise any major or consistent concerns with the phrasing of these items. However, a few items were not interpreted as intended, or produced cognitive difficulties. “You received poorer service than other people,” was perceived as being too vague because respondents felt that everyone has received poor service at some point in time. Confirming other cross-cultural research,30 our cognitive testing showed that respondents preferred vague quantifiers (e.g., never, often) to quantitative categories (e.g., once, twice, three or more times, or once a year, once a month, once a week). We used behavioral coding to identify any unanticipated reactions during the course of interviews. Details are described in the next section.
Quantitative and qualitative results informed instrument development for our field test. Our external advisors confirmed that the main objective of the field test should be to assess the two most common ways to ask about self-reported experiences of racial/ethnic discrimination.1 The first approach, early attribution, is to ask participants specifically about discrimination based on race/ethnicity (e.g., Have you been unfairly treated because of your race/ethnicity?). The second, late attribution, is to first ask participants about unfair treatment, and then ask participants about the reasons for this unfair treatment (e.g., Have you been unfairly treated? What was the reason?).
Two studies suggest that results may vary with the way questions on racial/ethnic discrimination are asked.31,32 The first study compared explicit versus generic measures of discrimination (parallel to the two approaches in our field test, early and late attribution, respectively) and found that prevalence, correlates, and associations with mental health vary depending on question framing.31 The second study, using data from the NLAAS, found that these two different approaches yielded discrimination scores that were only modestly correlated (r=.43).32 Among the participants who reported that they had experienced racial/ethnic discrimination in response to the direct question (early attribution), nearly one-third stated that they had not experienced unfair racial/ethnic treatment in response to the two-stage question (late attribution). Conversely, among participants who, in response to the late-attribution question, stated that they had experienced unfair treatment, fully half stated, in response to the early-attribution question, that they had not experienced racial/ethnic discrimination. In the NLAAS example, the early attribution approach had 3 items, whereas the late attribution approach had 9 items and there was incomplete overlap in the content between the two approaches. For instance, early attribution included a question on unfair treatment of friends that was not asked in late attribution. Because the questions directly assessing racial/ethnic discrimination and those assessing unfair treatment were not identical, we do not know the contribution of variation in question wording to the observed differences.33 This is an important limitation of these studies.
Our study will directly address this limitation. To do this, we have developed one instrument with two versions (each with 8 items on recent experiences and 5 items on lifetime experiences). The two versions differ only in their approach for eliciting self-reported experiences of racial/ethnic discrimination. Respondents will be randomized into the early and late attribution versions of the module. Responses will be compared across racial/ethnic groups. Thus, findings from our field test will enable researchers to directly address this critical methodological question.
Our first field test, on CHIS 2007, was in English and yielded a multi-racial/ethnic sample who identified as: Latino (n=2056), Asian American and Pacific Islander (n=1357), African-American (n=1069), Native American (n=763), Non-Hispanic White (n=2037), or multiracial (n=223). Data was collected from October 2007 through February 2008. In addition to conducting psychometric analysis on the split-sample described above, the two instruments will be evaluated for validity, cultural comparability and administration properties using qualitative and quantitative techniques. One example is behavior coding. We used this technique to conduct a qualitative study designed to identify patterns related to question performance along lines of racial/ethnic and sex groups with a subsample of approximately 500 respondents (100 for each of the 5 racial/ethnic groups). Findings from this analysis will complement psychometric and other findings.
The two approaches will be tested in multiple languages on the CHIS 2009 and compared. For this second round of field testing, both versions of the instrument are undergoing English simplification, cultural review, and translation in order to field the two approaches in Spanish and some Asian languages. We will use these data to refine the two approaches for asking about racial/ethnic discrimination. We anticipate that our final module will be ready for administration in CHIS 2011 and will be released for public use.
We seek a concise, reliable and valid way to elicit reports of racial/ethnic discrimination experiences that will work equally well in all racial/ethnic populations. Although this module is intended for health-related population-based surveys, it may also be used to measure racial/ethnic discrimination in other types of studies. Our instrument includes dimensions that are theoretically important (e.g., appraisal of discrimination as stressful) but are not routinely measured in instruments administered by telephone. A goal of our evaluation is to further balance content validity (e.g., lifetime vs. recent experiences, occurrences in various settings) with practical considerations related to parsimony, respondent burden, and costs of administration. Although the discrimination module is still under development, we feel that the growing interest in the measurement of racial/ethnic discrimination in health surveys warranted this early report. We hope that our preliminary results will contribute to development and refinement of measuring racial/ethnic discrimination in multicultural and multilingual populations. Our intention is to provide periodic updates to the scientific community as the data are analyzed and showcase a roadmap on best practice methods for the assessment or racial/ethnic discrimination in health surveys.
Table 1
Table 1
Self-Reported Racial/Ethnic Discrimination Field Test Instrument Sections and Sample Items by Early and Late Attribution Approach
Acknowledgments
We would like to thank the NIH Office of Behavioral Social Sciences Research for funding the field test and NCI Division of Cancer Control and Population Sciences for providing staff for this project. We would also like to acknowledge the Cancer Prevention Fellowship Program, Office of Preventive Oncology, National Cancer Institute for providing financial support for S. Shariff-Marco in her post-doctoral fellowship; Westat for their work on the cognitive interviews and behavior coding; and the Collaborative Psychiatric Epidemiology Surveys project for providing access to data from the National Survey of American Life, and the National Latino and Asian American Survey. Support for this work was also provided by the National Center for Minority Health and Health Disparities (MD 000508).
We would also like to acknowledge Dr. David Takeuchi for his participation as a member of the panel of external advisors for this research project and Drs. Martin Brown and Rachel Ballard-Barbash for their thoughtful reviews of our manuscript.
Footnotes
AUTHOR CONTRIBUTIONS
Design concept of study: Shariff-Marco, Gee, Breen, Willis, Reeve, Grant, Ponce, Krieger, Alegria, Johnson, Brown
Acquisition of data: Shariff-Marco, Gee, Breen, Brown
Data analysis and interpretation: Shariff-Marco, Breen, Willis, Reeve, Ponce, Krieger, Williams, Mays
Manuscript draft: Shariff-Marco, Gee, Breen, Willis, Reeve, Ponce, Krieger, Landrine, Alegria, Mays, Johnson
Statistical expertise: Ponce, Krieger
Acquisition of funding: Shariff-Marco, Breen, Grant, Brown
Administrative, technical, or material assistance: Shariff-Marco, Breen, Willis, Grant, Williams, Mays, Johnson
Supervision: Shariff-Marco, Gee, Breen, Ponce, Brown
1. Williams DR, Mohammed SA. Discrimination and racial disparities in health: evidence and needed research. J Behav Med. 2009;32(1):20–47. [PMC free article] [PubMed]
2. Mays VM, Cochran SD, Barnes NW. Race, race-based discrimination, and health outcomes among African Americans. Annu Rev Psychol. 2007;58:201–225. [PubMed]
3. Taylor TR, Williams C, Makambi K, et al. Racial discrimination and breast cancer incidence in US Black women. Am J Epidemiol. 2007;166(1):46–54. [PubMed]
4. Paradies Y. A systematic review of empirical research on self-reported racism and health. Int J Epidemiol. 2006;35(4):888–901. [PubMed]
5. Kressin NR, Raymond KL, Manze M. Perceptions of race/ethnicity-based discrimination: a review of measures and evaluation of their usefulness for the health care setting. J Health Care Poor Underserved. 2008;19(3):697–730. [PMC free article] [PubMed]
6. Pennell B, Bowers A, Carr D, et al. The development and implementation of the National Comorbidity Survey Replication, the National Survey of American Life, and the National Latino and Asian American Survey. Int Journal Methods Psychiatr Res. 2004;13(4):241–269. [PubMed]
7. Williams DR, Yu Y, Jackson JS, Anderson NB. Racial differences in physical and mental health: socioeconomic status, stress and discrimination. J Health Psychol. 1997;2(3):335–351. [PubMed]
8. Krieger N, Smith K, Naishadham D, Hartman C, Barbeau EM. Experiences of discrimination: validity and reliability of a self-report measure for population health research on racism and health. Soc Sci Med. 2005;61(7):1576–1596. [PubMed]
9. Landrine H, Klonoff EA, Corral I, Fernandez S, Roesch S. Conceptualizing and measuring ethnic discrimination in health research. J Behav Med. 2006;29(1):79–94. [PubMed]
10. Green NL. Development of the perceptions of racism scale. Image J Nurs Sch. 1995;27(2):141–146. [PubMed]
11. Utsey SO, Ponteretto JG. Development and validation of the index of race-related stress (IRRS) J Couns Psychol. 1996;43(4):490–501.
12. McNeilly MD, Anderson NB, Armstead CA, et al. The perceived racism scale: a multidimensional assessment of the experience of white racism among African Americans. Ethn Dis. 1996;6(1–2):154–166. [PubMed]
13. Landrine H, Klonoff EA. The schedule of racist events: a measure of racial discrimination and a study of its negative physical and mental health consequences. J Black Psychol. 1996;22(2):144–168.
14. Auslander WF, Thompson SJ, Dreitzer D, Santiago JV. Mothers’ satisfaction with medical care: perceptions of racism, family stress, and medical outcomes in children with diabetes. Health Soc Work. 1997;22(3):190–199. [PubMed]
15. Schneider KT, Hitlan RT, Radhakrishnan P. An examination of the nature and correlates of ethnic harassment experiences in multiple contexts. J Appl Psychol. 2000;85(1):3–12. [PubMed]
16. Finch BK, Kolody B, Vega WA. Perceived discrimination and depression among Mexican-origin adults in California. J Health Soc Behav. 2000;41(3):295–313. [PubMed]
17. Bird ST, Bogart LM. Perceived race-based and socioeconomic status (SES)-based discrimination in interactions with health care providers. Ethn Dis. 2001;11(3):554–563. [PubMed]
18. Loo CM, Fairbank JA, Scurfield RM, et al. Measuring exposure to racism: development and validation of a race-related stressor scale (RRSS) for Asian American Vietnam veterans. Psychol Assess. 2001;13(4):503–520. [PubMed]
19. Iyer DS, Haslam N. Body image and eating disturbance among south Asian-American women: the role of racial teasing. Int J Eat Disord. 2003;34(1):142–147. [PubMed]
20. Liang CT, Li LC, Kim BS. The Asian American racism-related stress inventory: development, factor analysis, reliability, and validity. J Couns Psychol. 2004;51(1):103–114.
21. Contrada RJ, Ashmore RD, Gary ML, et al. Measures of ethnicity-related stress: psychometric properties, ethnic group differences, and associations with well-being. J Appl Soc Psychol. 2001;31(9):1775–1820.
22. Vines AI, McNeilly MD, Stevens J, Hertz-Picciotto I, Baird M, Baird DD. Development and reliability of a telephone-administered perceived racism scale (TPRS): a tool for epidemiological use. Ethn Dis. 2001;11(2):251–262. [PMC free article] [PubMed]
23. Corrigan P, Thompson V, Lambert D, Sangster Y, Noel JG, Campbell J. Perceptions of discrimination among persons with serious mental illness. Psychiatr Serv. 2003;54(8):1105–1110. [PubMed]
24. Brondolo E, Kelly KP, Coakley V, et al. The perceived ethnic discrimination questionnaire: development and preliminary validation of a community version. J Appl Soc Psychol. 2005;35(2):335–365.
25. Clark R, Coleman AP, Novak JD. Brief report: initial psychometric properties of the everyday discrimination scale in black adolescents. J Adolesc. 2004;27(3):363–368. [PubMed]
26. Reeve BB, Hays RD, Bjorner JB, et al. Psychometric evaluation and calibration of health-related quality of life item banks: plans for the patient-reported outcomes measurement information system (PROMIS) Med Care. 2007;45(5):S22–S31. [PubMed]
27. Edelen MO, Reeve BB. Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement. Qual Life Res. 2007;16(suppl 1):5–18. [PubMed]
28. Teresi JA. Statistical methods of examination of differential item functioning with applications to cross-cultural measurement of functional, physical and mental health. J Ment Health Aging. 2001;7:31–40.
29. Thissen D, Steinberg L, Wainer H. Detection of differential item functioning using the parameters of item response models. In: Holland PW, Wainer H, editors. Differential Item Functioning. Hillsdale, NJ: Lawrence Erlbaum Associates; 1993. pp. 7–113.
30. Johnson TP, Cho YI, Holbrook A, O’Rourke D, Warnecke R, Cha’vez N. Cultural variability in the comprehension of health survey questions. Ann Epidemiol. 2006;16(9):661–668. [PubMed]
31. Brown TN. Measuring self-perceived racial and ethnic discrimination in social surveys. Sociol Spectr. 2001;21(3):377–392.
32. Chae DH, Takeuchi DT, Barbeau EM, et al. Unfair treatment, racial/ethnic discrimination, ethnic identification and smoking among Asian Americans in the National Latino and Asian American Study (NLAAS) Am J Public Health. 2008;98:485–492. [PubMed]
33. Williams DR, Neighbors HW. Racism, discrimination and hypertension: evidence and needed research. Ethn Dis. 2001;11:800–816. [PubMed]