|Home | About | Journals | Submit | Contact Us | Français|
Few studies have evaluated the reliability of lifetime sun exposure estimated from inquiring about the number of hours people spent outdoors in a given period on a typical weekday or weekend day (the time-based approach). Some investigations have suggested that women have a particularly difficult task in estimating time outdoors in adulthood due to their family and occupational roles. We hypothesized that people might gain additional memory cues and estimate lifetime hours spent outdoors more reliably if asked about time spent outdoors according to specific activities (an activity-based approach). Using self-administered, mailed questionnaires, test-retest responses to time-based and to activity-based approaches were evaluated in 124 volunteer U.S. radiologic technologist participants: 64 females and 60 males 48 to 80 years of age. Intraclass correlation coefficients (ICCs) were used to evaluate the test-retest reliability of average numbers of hours spent outdoors in summer estimated for each approach. We tested the differences between the two ICCs, corresponding to each approach, using a t-test with the variance of the difference estimated by the Jackknife method. During childhood and adolescence, the two approaches gave similar ICCs for average numbers of hours spent outdoors in summer. By contrast, compared with the time-based approach, the activity-based approach showed significantly higher ICCs during adult ages (0.69 vs. 0.43, P=0.003) and over the lifetime (0.69 vs. 0.52, P=0.05); the higher ICCs for the activity-based questionnaire were primarily derived from the results for females. Research is needed to further improve the activity-based questionnaire approach for long-term sun exposure assessment.
Exposure to ultraviolet radiation from the sun (solar UVR) is the principal environmental risk factor for skin cancers (1). There is also a growing body of evidence that solar UVR exposure may reduce risk of some diseases, including non-Hodgkin lymphoma (2–4), prostate cancer (5–7), breast cancer (7, 8), colon or colorectal cancer (7, 9), and multiple sclerosis (10).
Estimates of cumulative lifetime UVR are commonly derived from questionnaire-based recall of hours spent outdoors on weekdays or weekend days during different age intervals; i.e., a time-based approach. For example, participants have been asked “on weekdays, how many hours did you generally spend outdoors in sunlight (between 9:00 AM and 3:00 PM) in summer when you were [X] to [Y] years old?” (11). Time-based questionnaires have asked participants to estimate a typical number of hours outdoors in all activities in a given age interval without directing them to use any particular estimation method. We hypothesized that participants may gain additional memory cues and estimate time outdoors more reliably if asked about time spent outdoors in specific activities in particular periods of life; i.e., an activity-based approach.
Most studies examining the reliability of responses to questions about time spent outdoors have done so using information from interviewer-administered questionnaires (12–15); no study has assessed their reliability using self-administered questionnaires. It would be informative to evaluate the reliability from self-administered questionnaires since they are the primary source of information in large cohort studies.
One interview-based reliability study has reported that women’s responses were less reliable than men’s and attributed this difference to women’s combined roles in work and family (13). Other such studies, although reporting no difference between males and females, have not reported gender-specific reliability estimates (12, 14, 15). In addition to the fact that women multi-task in their frequently combined work and family roles (13), women report different outdoor activity patterns to men (13, 16–18).
In this investigation we have evaluated the test-retest reliability of responses to time-based and activity-based questions on lifetime time spent outdoors in self-administered mailed questionnaires in the same subjects at two different times; and analyzed reliability in men and women separately.
The investigation (Figure 1) was carried out among volunteers from the U.S. Radiologic Technologists (USRT) Study, a collaborative study between the U.S. National Cancer Institute, University of Minnesota, and the American Registry of Radiologic Technologists. The USRT cohort consists of 146,022 radiologic technologists who are predominantly female (73%), Caucasian (95%), and born before 1950 (58%; mean, 1944) (19, 20). Identification of the cohort was initiated in 1982, and three mail surveys were conducted during 1984–1989, 1993–1998, and 2003–2005.
Details of the selection of volunteers for the assessment of solar UVR exposure have been published previously (21, 22). Briefly, the third USRT mail survey containing time-based sun exposure questions was mailed to 1,800 USRT cohort members residing in the greater Minneapolis metropolitan area (latitude: 44.96° N; longitude: 93.27° W) and surrounding areas of Wisconsin (northern latitudes) or in Raleigh, North Carolina (latitude: 35.82° N; longitude: 78.66° W) and Atlanta, Georgia (latitude: 33.76° N; longitude: 84.42° W) (southern latitudes) in the summer of 2004. Participants from two geographic regions were studied because there were potential differences in outdoor behavior patterns by latitude (23), as latitude influences outdoor temperatures, which to some extent determine outdoor activities at different times of the year. A stratified sample of 300 participants (including 150 individuals in each geographic region) was randomly selected from respondents who returned the questionnaire within 4 weeks. Within each region, the sample was further stratified by age group (40–59, 60+ years) and gender. Our goal was to enroll 120 of the 300 targeted participants, including approximately 15 in each age-gender-region category. A total of 127 participants were initially enrolled and 124 (98%) completed subsequent questionnaires and other data collection. The numbers of participants were similarly distributed by gender (female, n =64; male, n=60), age (ages 40–59, n=60; ages 60+, n=64), and geographic region (northern latitudes, n=63; southern latitudes, n=61).
The institutional review boards at the National Cancer Institute and University of Minnesota approved the protocol. We focused on assessment of time spent outdoors from 9:00 AM to 3:00 PM in summer, because a very high proportion of annual ambient UVR irradiance is incident during this period in the summer (24) and is considered most hazardous for UV exposure in the continental U.S. (25). In addition, individual outdoor activity patterns may vary substantially by season (23). Therefore, in this methods study we inquired about time spent outdoors in summer during the defined age intervals (<13, 13–19, 20–39, 40–64, 65+ years). These age intervals are similar to those used in other studies (12, 14, 26).
We used a test-retest design to answer our research questions; participants were asked to complete each questionnaire type twice at similar intervals (mean = 223 days for time-based, = 195 days for activity-based) but at different times (Figure 1). All questionnaires were sent by mail, and formatted for self-administration. The baseline (test) time-based questionnaire was administered during the summer of 2004, and the baseline activity-based questionnaire in the spring of 2005.
In the time-based questionnaire, participants were asked to report the number of hours they usually spent outdoors, according to the period during the week (weekday or weekend) and age intervals. Participants were given the following response categories for estimating the average number of hours spent outdoors each day between 9:00 AM to 3:00 PM in the summer: 0, <1, 1 to 2, 3 to 4, and 5 to 6 hours per day.
The origin of the activity-based questionnaire was an activity diary. Participants were asked to complete an activity diary for seven consecutive days between September 1 and October 5 in 2004. In the diary, participants were asked to list their primary activity type (indoor or outdoor) and specific outdoor activities in every 30-minute interval between 9:00 AM and 5:00 PM (Figure 1). Using categories for grouping similar types of activities that were derived from two large national surveys (23, 27) and expert judgment, the original 4,697 entries were eventually reduced to 7 major categories, consisting of driving, gardening, outdoor home maintenance, walking and light exercise, water activities, outdoor recreation, and outdoor leisure. Outdoor recreation included playing golf, soccer, biking, hiking, running, jogging, or moderate to heavy exercise. Outdoor leisure included sitting or relaxing outside, eating or cooking outside, smoking, supervising children or watching sports games. These seven categories captured more than 94% of total time spent outdoors among the study participants (17).
For summer outdoor exposures recalled for the age intervals 20–39, 40–64, and 65+, we used these seven categories, along with time spent outdoors in the participant’s job, in our activity-based questionnaire. For recall of summer outdoor exposures during the age intervals 0–12 and 13–19, the only categories included in the activity-based questionnaire were 1) organized sports, 2) other than organized sports (e.g., sitting or relaxing outside, running or playing outside, selling lemonade), 3) water activities, and 4) time spent outdoors in a job (ages 13–19 only). Participants were asked, for instance, “how many hours in total from Monday through Friday did you generally spend outdoors doing activity [A] between 9:00 AM and 3:00 PM during the summer when you were [X] to [Y] years old?” The following response categories were provided: 0, <1, 1 to 2, 3 to 4, 5 to 6, 7 to 8, 9 to 10, >10 hours in total from Monday through Friday or on the combined weekend days (Saturday and Sunday). The activity-based questionnaire consisted of a total of 62 questions: 31 questions for each of summertime weekday and weekend day exposures, including 3, 4, 8, 8, and 8 questions for weekday and for weekend day exposures during age intervals 0–12, 13–19, 20–39, 40–64, and 65+ respectively.
The maximum total number of hours participants reported spending outdoors during the weekdays and on weekend days between 9:00 AM and 3:00 PM in the activity-based questionnaire should not have exceeded 30 hours or 12 hours respectively. Participants, however, tended to over-estimate the number of hours spent on each activity, and thus the total hours (across activities) often exceeded 30 or 12 hours. The proportion of participants who over-estimated the number of hours ranged from 0% (ages 0–12, weekdays, males and females) to 57.6% (ages 20–39, weekend days, males). Participants were more likely to over-estimate their exposure during weekend days than during weekdays, and more males than females over-estimated their time spent outdoors. Although these over-estimated responses raised concerns about the validity of the number of hours reported in the activity-based questionnaire, to avoid distorting the reliability of estimated hours spent outdoors, we did not transform these over-reported values; participants with over-reported values were retained in the analysis.
Cumulative lifetime sun exposure during 9:00 AM to 3:00 PM in summer was calculated by summing weekday and weekend day exposures across different age intervals (time-based questionnaire), or across activity categories and age intervals (activity-based questionnaire) using the midpoint values of the time categories. To account for the effect of current age (e.g., more total hours spent outdoors for older than for younger participants), the cumulative measure was then averaged to give the rate of sun exposure (i.e., average number of hours spent outdoors in the summer per year).
For ordinal variables (e.g., original response categories that were used in both assessment approaches), the test-retest reliability was evaluated by the proportion of participants with perfect agreement and weighted kappa, which compares the observed agreement with that expected by chance while giving more weight to more disparate responses (28). If the participants’ responses were limited to two categories in both test and retest, the weighted kappa reduces to the simple unweighted kappa. Guidelines have been proposed for qualitative interpretations of the kappa statistic: 0.81–1.0 almost perfect, 0.61–0.80 substantial, 0.41–0.60 moderate, 0.21–0.40 fair, 0.00–0.20 slight, and <0.00 poor (29).
For continuous variables (e.g., average number of hours spent outdoors per year), we first used a log transformation to stabilize the variance, and then used a one-way ANOVA to obtain the intraclass correlation coefficient (ICC) (28). The ICC can be interpreted as the correlation between the test and retest responses for the same individual (28). ICCs were evaluated separately according to the exposure time window: childhood and adolescence ages (0–19 years), adult ages (20+ years), and lifetime (0+ years). Guidelines have been proposed for the interpretation of ICC: 0.75–1.00 excellent, 0.40–0.74 fair to good, and 0–0.39 poor (30).
We compared weighted kappas, as well as ICCs, from the two questionnaire approaches to evaluate the reliability of the two approaches. For weighted kappa comparisons, however, because different numbers of questions and time categories were used for the two questionnaire approaches, we had to reduce the numbers of response categories in the activity-based approach to obtain “average” weighted kappa values, as weighted kappa values are affected by the number of response categories. The reduction of categories was performed as follows: for weekdays and for weekend days within each age interval, we first calculated the proportion of participants, separately for males, females, and the total population, within each response category for the time-based questions at baseline (test data); in some cases, the response categories were combined to ensure sufficient sample size in each response category to calculate weighted kappas. We then regrouped the response categories for the baseline data for activity-based questions (using the sum of hours from all activities within the time period during the week, separately for weekdays and weekend days), so that the regrouped activity-based response categories would have a similar distribution to the time-based response categories. The new hour categories for the activity-based questions were then applied to the activity-based retest data, and used to calculate the average weighted kappa values for the activity-based questionnaire. For instance, for weekday exposure during ages [X] to [Y], the categories for the time-based questions had a distribution of a%, b%, c%, d%, and e% for 0, <1, 1 to 2, 3 to 4, and 5 to 6 hours per day, respectively. For data from the activity-based questionnaire, we summed up the total hours spent outdoors across activities carried out during weekdays for ages [X] to [Y]. The total hours in the test data of the activity-based approach were combined into five new categories to provide a distribution of a%, b%, c%, d%, and e%, respectively (e.g., < 5, 5–9.9, 10–14.9, 15–24.9, and 25+ hours as the five new categories for all weekdays combined for ages [X] to [Y]). The same five new categories were used to provide frequency distributions for the activity-based retest data, and an average weighted kappa value was calculated based on the new combined categories.
To test the null hypothesis H0: D = 0, where D is the unknown parameter representing the difference between the weighted kappa estimates from time-based and activity-based questionnaires, we calculated the value of T using the formula:
where is the estimate of D based on the data from individuals who answered both the test and the retest questionnaires in both questionnaire approaches, and JK is the variance of the difference estimated by the Jackknife method (31).
To calculate the P-value for rejecting the null hypothesis, we then compared T to a t distribution with d degrees of freedom, where d is the number of individuals minus 1. The same statistical method was applied to compare the ICCs between the activity-based and the time-based estimates.
Among the 124 participants who returned the retest questionnaires, the mean age was 60.1 years (range, 48–80) in males and 59.7 years (range, 45–74) in females. The mean time interval between the test and retest was 223 days (range, 181–290) for the time-based questionnaire and 195 days (range, 125–261) for the activity-based questionnaire.
We first compared the estimated number of hours spent outdoors in the summer per year (averaged between the test and retest questionnaires) from the two questionnaire approaches (Table 1). During childhood and adolescence (ages 0–19 years), the reported number of hours spent outdoors was similar between the two questionnaires for all participants combined. During adult ages (20+ years), the estimated total hours from the activity-based approach were higher than those from the time-based approach in both genders.
For all participants, assessment of test-retest reliability by weighted kappa for the time-based questionnaire generally showed fair to moderate reliability; weekday exposure between ages 20–39, for which reliability was slight, was an exception (Table 2). For women, kappa generally indicated slight to moderate reliability with the lowest reliability on weekdays during ages 20–39 and 40–64. For men, reliability was slight to moderate with the lowest reliability observed on weekdays during ages 20–39.
Test-retest agreement for the activity-based questionnaire assessed by weighted kappa showed fair to moderate reliability for all participants combined (Table 2). For women, weighted kappa values gave slight to moderate reliability, with slight reliability found for exposures in weekdays during childhood and at ages 65+. For men, weighted kappa statistics indicated fair to moderate reliability for weekdays and weekend days during all ages.
We compared the weighted kappa values for time spent outdoors from activity-based questionnaire with those from the time-based questionnaire (Table 2). Only three of the 30 comparisons gave P-values that were <0.05 and of those that did, two were in favor of greater reliability for the activity-based questionnaire (all participants and weekday exposures during ages 20–39 and females and weekday exposures during 40–64 years of age) and one for the time-based questionnaire (all participants and weekend day exposures at 13–19 years of age). There is, thus, no strong evidence in these comparisons for greater reliability of either questionnaire with respect to the categorical estimates of outdoor hours in individual age groups.
Table 3 shows the test-retest reliability for average hours spent outdoors from time-based and activity-based questionnaires by gender and period of life. The ICCs for the two questionnaire types were similar in childhood and adolescence in all subjects and in males and females. They were, however, consistently higher for the activity-based questionnaire in adult years and over the lifetime in all subjects and both genders, with ICCs ranging from 0.64 to 0.72 for the activity-based questionnaire and 0.25 to 0.65 for the time-based questionnaire. These differences were statistically significant for all subject and females but not for males, in whom the ICCs for the time-based questionnaire in adulthood and over the lifetime were substantially higher (0.55, 95% CI 0.34–0.71, and 0.65, 95% CI 0.46–0.79, respectively) than in females (0.25, 95% CI 0.00–0.47, and 0.33, 95% CI 0.09–0.54).
To our knowledge, this is the first study to compare the reliability of time-based and activity-based sun exposure questionnaires whether self-administered (as in this case) or interviewer administered. The retest questionnaires were sent 4–9 months after the baseline (test) questionnaire. The activity-based questionnaire produced higher estimated time spent outdoors, especially in adult years and during lifetime. We also wanted to learn whether cumulative estimates of time spent outdoors could be improved, especially for women, using an activity-based questionnaire instead of the more usual time-based questionnaire. While there was little apparent difference between the two questionnaire types in reliability of age-group specific categorical estimates of time spent outdoors on weekdays and weekend days, as assessed by kappa statistics (Table 2), the activity-based approach showed significantly higher ICCs for continuous estimates of average hours outdoors in adult years and over the lifetime than the time-based questionnaire (Table 3). This difference was more evident in females than in males and was not evident for childhood and adolescence exposure.
One concern is the large difference between the estimated numbers of hours spent outdoors based on the two questionnaire approaches. A key potential strength for the activity-based questionnaire is the opportunity to trigger respondent’s memory to recall the full spectrum of outdoor activities. This is particularly true for adult years, during which participants perform various types of outdoor activities that may have not been captured using the traditional time-based questionnaire. For instance, one activity category that is likely not captured in the time-based approach is “driving”, which has been shown to account for 39% of total time spent outdoors in our study population during the assessment of short-term sun exposure (17). Despite the potential advantage discussed above, the activity-based approach also allows a greater chance of inaccuracies because of the number of questions asked. It is likely that the activity-based questionnaire approach gave much higher estimates in time spent outdoors than the time-based questionnaire approach due to the combination of these factors.
Other studies of reliability using time-based questionnaires have observed similar or higher reliability than the activity-based questionnaires in this study. These previous studies were all interviewer-administered (12–15), and thus could presumably have used cues to help elicit responses. In an Australian population of 62 cases with basal cell carcinoma and 162 controls (84 females and 80 males, aged 40–64 years at the first interview), English et al. reported an ICC of 0.77 for lifetime hours spent outdoors using face-to-face interviews 5 years apart. In another study in Australia of 52 cases with multiple sclerosis and 52 controls (69 females and 35 males, mean age = 43.6 for cases and 44.7 for controls) who were interviewed 11 weeks apart, the authors reported weighted kappa values ranging from 0.43 to 0.74, according to four age intervals and two seasons and disease status, using face-to-face interviews (15). In a multi-center European study of 115 cases of basal cell and squamous cell carcinoma and 119 controls (106 females and 130 males, mean age = 61.2 for cases and 62.4 for controls), Rosso et al. obtained ICC values of 0.68, 0.79, and 0.56 for weighted outdoor hours in a lifetime for work, vacation, and sports, respectively, using face-to-face interviews 18–26 months apart (14).
For the current investigation, the test-retest reliability for the time-based questionnaire was lower in females than males (Table 3). Some, but not all, studies have also reported gender differences in time spent outdoors and types of outdoor activities (16–18). Compared with males, estimating lifetime cumulative sun exposure in females during adult years has been reported to be particularly difficult, possibly because of females’ frequently combined work and family roles (13). Kricker et al, evaluating reliability of a shortened telephone questionnaire against its longer face-to-face version 9–20 months apart in an Australian population of 15 ocular melanoma cases and 45 controls (34 males and 26 females), also reported lower reliability of cumulative sun exposure among females (ICC = 0.54) than in males (ICC = 0.73) (13). The same study also reported that females had lower reliability of working day sun exposure than males (ICC = 0.48 for females and 0.71 for males), whereas females had higher reliability of nonworking day exposure than males (ICC = 0.67 for females and 0.53 for males) (13). As a second component of their study, Kricker et al compared the correlations between occupational exposure and total exposure (occupational + recreational) in males and females. The correlation was lower in females than in males (R = 0.54 and 0.78, respectively) (13). In contrast, other studies reported no differences in reliability by gender but did not give gender-specific estimates of reliability (12, 14, 15). Lastly, in the USRT cohort, using self-administered, mailed questionnaires, test-retest responses to the time-based questionnaire approach was evaluated in a previous study of 481 U.S. radiologic technologists (ages 44 to 80 years old, including 238 males and 243 females from throughout the U.S.) who volunteered to complete the same questionnaire about lifetime UVR solar exposures at two times approximately 9 months apart. In this larger nationwide population, reliability of responses about lifetime sun exposure was fair to good for both women (ICC = 0.62, 95% CI 0.53–0.70) and men (ICC = 0.53, 95% CI 0.42–0.62) (Thomas Fears, personal communication). These mixed findings suggest that the potential gender difference in test-retest reliability is still an evolving area. Additional studies are needed to examine the potential gender differences in the reliability of cumulative sun exposure, by properly accounting for various types of activities.
Our study has several strengths, including high participation rates, a wide range of ages, substantial numbers of participants in both genders, inclusion of participants from two very different latitudes, and systematic assessment of sun exposure at specified intervals. For the activity-based approach, we provided a comprehensive list of activities in our questionnaire to potentially improve participants’ recall.
The study also has some limitations. Most importantly, when answering the activity-based questionnaire, participants over-estimated the total hours spent outdoors when the number of hours spent in a given activity were summed, raising questions about the validity of the activity-based questionnaire. The degree of over-estimation (compared to the maximum number of hours of possible response) was lower during weekdays (particularly for females), and higher on weekend days. Part of the overestimation may have been due to study participants recalling their time outdoors during 9 A.M. to 5 P.M., which corresponds more closely to the working day, rather than the 9 A.M. to 3 P.M. time frame asked in the questionnaires. The over-reporting on the activity-based questionnaire may have contributed to the higher number of total outdoor hours recorded using the activity-based questionnaire (Table 1).
We cannot completely rule out chance as an alternative explanation for the higher ICC in the activity-based questionnaire, because of the multiple comparisons made. Nonetheless, given the body of literature indicating gender differences in outdoor activity patterns, we consider that the activity-based questionnaire approach provides a promising new direction to improve reliability in the questionnaire assessment of sun exposure. This position is supported by the fact that the activity-based questionnaire showed the greatest improvement in women’s recall of their outdoor hours, which is consonant with the basis for which it was designed.
In conclusion, a newly-developed activity-based questionnaire, which asks detailed contribution from each activity category, produced an increase in overall reliability of sun exposure in adult years and over the lifetime, in comparison with a time-based questionnaire, which was greater in women than men. There is, however, uncertainty about the validity of the activity-based questionnaire in light of many participants estimating daily outdoor hours that were greater than the specified time window during the day. Research is needed to improve the accuracy of the activity-based questionnaire while retaining its high reliability, for instance, by alerting participants when the total hours across activities exceed the maximum.
Funding: This research was supported by the Intramural Research Program of the National Institutes of Health and the National Cancer Institute.
The authors gratefully acknowledge the participation of the USRT cohort members. Thanks to Diane Kampa and Rick Hoffbeck (University of Minnesota) and Jeremy Miller (IMS) for their assistance.