The purpose of this study was to investigate the impact of residence time and language preference on the likelihood of experiencing problems with survey questions. Summarizing across question domain and problem type, the frequency of problems reported by two or more respondents ranged from 86% in low residence time/Spanish respondents to 67% and 62% respectively in high residence time/Spanish and English speaking groups. Translation problems were found to be infrequent across residence time/language conditions and question topic areas, suggesting that translation processes were effective (Table ). On the other hand, a variety of non-translation problems were identified, for all three subject groups, and for both question topic areas (Table and below).
Three main results are relevant. First, for both physical activity and acculturation items, the percentage of items exhibiting problems was marginally higher for acculturation-related questions (Chi2, p = 0.0524). Second the number of problems differed significantly by language group (Chi2, p = 0.0204), with the fewest problems for English interviews related to physical activity (50% of questions) and the most for low acculturated people interviewed in Spanish (97% of questions). These p values have to be interpreted with caution, as some observed and expected cell sizes are below five.
Acculturation Items: Key Qualitative Findings
Given our small sample sizes, and lack of control for potentially confounding effects, we relied on intensive qualitative analysis of observed problems to identify potential sources and causes. For acculturation items, non-translation problems were especially prevalent for the less-acculturated subjects. A particular item format seemed to cause many difficulties, as illustrated below. For these items, field interviewers use a pre-coded set of categories to code open-ended responses (that is, the question is asked open-ended, but then coded into a pre-selected set of response categories).
Where was your mother born? (RESPONDENTS REPORT AN OPEN-ENDED ANSWER AND INTERVIEWERS CIRCLE A NUMBER TO CODE THE RESPONSE).
CENTRAL AMERICA (Name of Country).................6
SOUTH AMERICA (Name of Country)...........7
Six of the 29 acculturation items used this general format, which produced difficulties for interviewers because respondents often reported the names of villages, towns or regions. If interviewers were unfamiliar with the countries respondents meant to refer to, then additional unscripted probing was necessary to identify the appropriate country or territory. Hence, this finding represents a classic response-matching problem [46
], as subjects both understand the question and know the answer, but simply provide an answer at a different (geographic) level.
In reviewing these issues by group, we found that there was a decrease in problem frequency depending on how long the participant had been in the U.S., and a further decrease for those who also preferred to be interviewed in English. For those with less than 5 years in U.S., all nine respondents answered with the name of their village or town. Among those who were interviewed in Spanish but had been in the U.S. at least 15 years, 6 respondents answered with the name of the country (as intended), and 3 with the town or state. On the other hand, all nine of those who were interviewed in English and had been in the U.S. for at least 15 years answered with the name of the country.
Qualitative information revealed a further example of how interpretation of acculturation-related items may vary depending on how long the respondent has been in the U.S., and language preference. For the items: "When you were growing up, how many of your friends were of Anglo origin?" and "How many of your friends now are of Anglo origin?", most respondents who lived in the U.S. less than 5 years were found to be unfamiliar with the term "Anglo." Further, some respondents who lived in the U.S. 15 years or more and who preferred to complete the interview in Spanish were also unfamiliar with the term 'Anglo' and provided ambiguous explanations when asked to define it.
In contrast, all nine respondents who preferred to complete the interview in English were familiar with "Anglo." However, they did interpret the term in somewhat different ways in response to cognitive interview probes. Examples include "Anglo" as "from another nationality" (e.g., "non-Salvadoran; non-Puerto Rican"), "those who speak English", "non-Latino", "white" or "Caucasian"; "born in the U.S."; "white American"; "North American"; "of English descent"; and "northern European" (e.g., France, Germany, Holland, Switzerland). Because of the multiple interpretations of the term, we recommended that "Anglo" be replaced with an alternative, well-defined term that makes intended measurement goals clearer to all groups. "English Speaking European American" is a long but specific alternative. We are unaware of efforts to test this or other alternative phrases.
A further example of problems with acculturation-related items and involved the key term "ethnic identification" within the item "What ethnic identification (does/did) your mother use?" In general, the term "ethnic identification" was unfamiliar to respondents in all three design conditions. Several subjects who lived in the U.S. less than 5 years adopted an unexpected interpretation of the item, inferring that "ethnic identification" referred to official paperwork related to proof of citizenship or legal status (e.g., "your certificate of baptism" or "birth certificate"). This seems important, because absent cognitive testing, it is not obvious that this problem would be detected from respondent responses and the resulting misclassification would go undetected.
Further, testing of the follow-up question "Would you say she is/was Latino, Hispanic, American, North American, Cuban, Mexicano, or something else? ", which was asked when respondents were unable to reply in open-ended form, yielded further evidence of interpretive variation across groups. Most respondents who had lived in the U.S. less than five years were unfamiliar with terms such as "Latino" or "Hispanic ". However, several subjects in the two groups who lived in the U.S. 15 years or more also failed to identify with any of the response options, stating that these labels over-simplify ethnic background by glossing over important distinctions among groups from different regions and groups with different nationalities. This example suggests that items on ethnic background are problematic, but may present somewhat different problems for more and less acculturated respondents.
Respondents also had trouble with categories for language and thinking activities, such as with the categories "only Spanish", "mostly Spanish", "Spanish and English about the same," "mostly English", "only English", or "another language ". The qualifiers 'only' and 'mostly' were either ignored or did not fit respondents' situation. The phrase "In which language do you think?" was sometimes interpreted by respondents to mean, "What language do you think about. "
Physical Activity Items: Key Qualitative Findings
For the physical activity questions, question design issues (as opposed to translation problems) were again common, but appeared to affect all subject groups equally, from both quantitative and qualitative perspectives. Problems mainly related to either vague wording, difficulties in recalling necessary information, or estimating a response. Vague terms that respondents in all three groups identified within the physical activity items included "vigorous activity", "light or moderate" activities, "leisure" activities and "physical activities specifically designed to strengthen your muscles." Respondents had problems consistently recalling information and estimating activity frequency and activity duration across the set of items on walking and exercise activities. For example, in response to probe questions concerning walking for transportation, one respondent reported walking to the mall every day during the past 7 days. She stated that she knew that these walks were more than 10 minutes because she usually spent about 3 hours walking to, from, and around in the mall each time. On the other hand, for another question about walking, the same respondent reported walking about 15 minutes a day during the past 7 days. In general, question about vigorous activity involving lengthy definitions (e.g., activity for 10 minutes, that causes heavy sweating or large increases in breathing or heart rate...) had to be repeated multiple times due to question wording complexity.
Another subject answered "yes" to a question concerning walking, reporting that the prior day she had walked to a store (a walk that took her 20-25 minutes, round trip), and she also walked to get to the bus (a 12 minute walk). However, her responses to related questions suggested that it is challenging to summate these different episodes in a consistent manner. Finally, subjects in all three groups often provided answers in terms of ranges (e.g., "10 to 15 minutes") or multiple answers, that is, separate responses for different walking activities. These response patterns are an additional indication that respondents may have difficulty estimating the requested time duration. Again, these problems were fairly equally distributed across Latino subgroup tested.
Although cross-cultural differences were not strongly reflected in PA items, we did find some hints of cross-cultural variation. Concepts such as weekends or weekdays were ambiguous to some respondents. Further, some respondents did not divide their week into 5 weekdays and 2 weekend days. When asked: (a) Outside of work, how many hours do you spend per day during WEEKDAYS sitting?; and (b) Outside of work, how many hours do you spend per day during the WEEKEND sitting, some subjects who preferred Spanish for the interview reported that they were thinking of "either everyday, 5-6 working days, or that day" when asked about the specific term "weekday"; and that weekend included only Sunday, or else included Friday, Saturday, and Sunday. English speakers, on the other hand seemed to accept without comment that the weekend was Saturday and Sunday.
In one case, a translation problem arose where Spanish translation included words that conveyed the wrong meaning."Aficiones" was the word used to convey hobby even though in many Spanish-speaking countries it refers to a passion toward something. This kind of problem also arose for other words, but survey questions in other domains, beyond acculturation and physical activity, could well present even more translation problems due to meaning and variation in pronunciation.
There were differences in the specific kinds of problems reported for acculturation versus physical activity items that are apparent when problem types are categorized according to the Q-bank criteria. Table presents the tabulation of problem types in the acculturation and PA domains for each of the interview conditions. PA questions resulted in more problems involving recall and judgment processes, whereas acculturation questions resulted in many more problems concerning interviewer difficulty and response selection. There were many problems related to question wording in both domains. This difference in the frequency of problem types across domains was statistically significant based on a chi-squared test comparing the frequency of problems between acculturation items and physical activity items (p < 0.05). Inspection indicates no evidence that language use was related to problem type in this more inclusive tabulation.