Development of a Computer-Based Survey Instrument for Organophosphate and N-Methyl-Carbamate Exposure Assessment among Agricultural Pesticide Handlers

doi:10.1093/annhyg/meq038

Ann Occup Hyg. 2010 August; 54(6): 640–650.

Published online 2010 April 22. doi: 10.1093/annhyg/meq038

PMCID: PMC2918489

Development of a Computer-Based Survey Instrument for Organophosphate and N-Methyl-Carbamate Exposure Assessment among Agricultural Pesticide Handlers

Jonathan N. Hofmann,¹^* Harvey Checkoway,^1,² Ofelio Borges,³ Flor Servin,³ Richard A. Fenske,² and Matthew C. Keifer²

¹Department of Epidemiology, University of Washington, Seattle, WA 98195–7236, USA

²Department of Environmental and Occupational Health Sciences, University of Washington, Seattle, WA 98195–7234, USA

³Washington State Department of Agriculture, Olympia, WA 98504–2560, USA

^*Author to whom correspondence should be addressed. Tel: +1-301-402-2605; fax: +1-301-402-1819; e-mail: hofmannjn/at/mail.nih.gov

Received November 12, 2009; Accepted March 11, 2010.

Abstract

Background: Assessment of occupational pesticide exposures based on self-reported information can be challenging, particularly with immigrant farm worker populations for whom specialized methods are needed to address language and cultural barriers and account for limited literacy. An audio computer-assisted self-interview (A-CASI) survey instrument was developed to collect information about organophosphate (OP) and N-methyl-carbamate (CB) exposures and other personal characteristics among male agricultural pesticide handlers for an ongoing cholinesterase biomonitoring study in Washington State.

Objectives: To assess the feasibility of collecting data using the A-CASI instrument and evaluate reliability for a subset of survey items.

Methods: The survey consisted of 64 items administered in Spanish or English on a touch-screen tablet computer. Participants listened to digitally recorded questions on headphones and selected responses on the screen, most of which were displayed as images or icons to facilitate participation of low literacy respondents. From 2006–2008, a total of 195 participants completed the survey during the OP/CB application seasons on at least one occasion. Percent agreement and kappa coefficients were calculated to evaluate test–retest reliability for selected characteristics among 45 participants who completed the survey on two separate occasions within the same year.

Results: Almost all participants self-identified as Hispanic or Latino (98%), and 97% completed the survey in Spanish. Most participants completed the survey in a half-hour or less, with minimal assistance from on-site research staff. Analyses of test–retest reliability showed substantial agreement for most demographic, work history, and health characteristics and at least moderate agreement for most variables related to personal protective equipment use during pesticide applications.

Conclusions: This A-CASI survey instrument is a novel method that has been used successfully to collect information about OP/CB exposures and other personal characteristics among Spanish-speaking agricultural pesticide handlers.

Keywords: agriculture, audio computer-assisted self-interview (A-CASI), exposure assessment, pesticides, survey instrument

INTRODUCTION

Accurate classification of exposures is critical in occupational epidemiology studies. Often, direct measurements of exposures are unavailable, and researchers must rely on questionnaires to collect exposure information retrospectively. Although questionnaires can be an efficient means of data collection, exposure misclassification due to inaccuracies in self-reporting can obscure true exposure–response relations (Perry et al., 2006). Even standardized survey instruments and methods that have been well validated in the general population may not be effective with population subgroups that differ with respect to primary language and culture and have low levels of formal education and literacy (Zahm et al., 2001). Recently developed survey methods and technologies may be useful for improving the accuracy of self-reported occupational exposure information in studies involving immigrant and low literacy worker populations.

One increasingly popular method that has been used successfully in various epidemiologic studies is audio computer-assisted self-interviewing (A-CASI). With A-CASI software, questions and potential responses are displayed on the computer screen and are narrated using digitally generated or recorded audio. Responses are entered directly into the computer on a touch screen, an adapted keyboard, or a traditional keyboard. Previous studies utilizing A-CASI instruments have demonstrated that data are reliable and that participants may be more willing to report sensitive information with an A-CASI instrument than in a paper-based self-administered questionnaire or face-to-face interview (Turner et al., 1998; Kurth et al., 2004). Relative to traditional paper-based self-administered questionnaires, potential advantages of A-CASI instruments include privacy of responses, participation of individuals with low literacy levels, branching logic and automated skip patterns, and efficient data entry and management (Bloom, 1998). Other strengths relative to face-to-face interviews include standardized question delivery and the ability to administer surveys in multiple languages.

A-CASI instruments have been more widely accepted in studies of sensitive behaviors (i.e. sexual history, substance use) or stigmatized conditions (i.e. psychiatric symptoms; Brown et al., 2008). However, relatively few studies have used A-CASI for occupational exposure assessment. In this paper, we describe an A-CASI instrument that was developed to collect information about organophosphate (OP) and N-methyl-carbamate (CB) exposures among Washington State agricultural pesticide handlers in an ongoing longitudinal study of risk factors for cholinesterase (ChE) inhibition (Hofmann et al., 2009; Hofmann et al., 2010). This study assessed the feasibility of collecting data using the A-CASI instrument and evaluated the reliability of reported information for a subset of items in the survey.

METHODS

Survey development

This A-CASI instrument was developed by a collaborative team including investigators from the University of Washington and pesticide safety educators from the Washington State Department of Agriculture, all of whom were bilingual in English and Spanish. A preliminary version of the survey was developed in English and translated into Spanish by a professional medical translator. Subsequent revisions to the survey were based on the Spanish version. The final English and Spanish versions were reviewed for consistency by the team that developed the questions.

The final survey included 64 items. Information was collected about factors potentially related to pesticide exposure during the 30-day period prior to the interview, including use of specific OP/CB insecticides, crops treated, handling activities performed, spray equipment used, duration and frequency of handling activities, use of personal protective equipment (PPE), personal hygiene, decontamination practices, and acute exposure events. We also asked about pesticide safety training, non-occupational exposures, symptoms of pesticide-related illness, concern about pesticide-related health effects, health status, medication use, and demographic characteristics. Examples of specific items within each of these categories are provided in Table 1.

Table 1.

Examples of information collected using the computer-based survey instrument^a

The survey was administered on a touch-screen tablet computer. A software application that was originally developed to collect data from patients receiving treatment for sexually transmitted infections was adapted for use in this study (Kurth et al., 2004; Hendry et al., 2005). An example of the user interface is shown below (Fig. 1). Questions and possible responses were narrated by research team members (O.B. for the Spanish version and M.K. for the English version). The survey instrument also featured an avatar (i.e. a computer representation of an interviewer) to explain how to enter information. Pictures or icons were used to represent responses for most questions. As each response is read out loud, the background of the picture/icon and the response text simultaneously change color on the screen.

Fig. 1.

Example of the user interface for the computer-based survey instrument (photographs reprinted with permission courtesy of the following individuals/organizations: (i) air blast sprayer photo—Steve Ringman, Seattle Times; (ii) boom sprayer photo—Paul (more ...)

The questions were revised after nine face-to-face pilot interviews; most changes involved replacing technical language regarding pesticide application practices with lay terminology used by Spanish-speaking pesticide handlers. Several additional changes were made to the survey instrument after the first year of data collection (2006) based on field observations, participant feedback, and preliminary reviews of the data. In the 2006 version of the survey, some questions required numerical responses on a calculator-style keypad (e.g. days since last OP/CB exposure, frequency of alcohol consumption, length of employment handling pesticides). Because some participants had difficulties with these questions and several response errors were observed in 2006, these items were changed to categorical questions in subsequent years. Several questions were added to the survey after 2006, and the questions about literacy were changed from ‘yes/no’ responses to categorical responses. (Question: How well can you read in Spanish/English? Responses: not at all, not very well, fairly well, or very well.)

Recruitment and record selection

The study population was comprised of male agricultural pesticide handlers in a ChE monitoring program administered by the Washington State Department of Labor and Industries (L&I). In this statewide program, employers are required to offer ChE testing for agricultural workers who handle OP/CB insecticides, though workers have the option to decline ChE monitoring. In 2006, 11% of the handlers who were offered ChE testing declined to participate in the program (SAC, 2006), and a similar proportion declined testing in 2007 and 2008 (L&I, 2008). The workers who participate in this program are tested for ChE activity annually at baseline before the OP/CB application season (typically in February or March). Workers are re-tested during the application season (typically April–July) if they handle OP/CBs for 30+ h in a 30-day period. Of the 1889 handlers who had baseline tests in 2006, 471 (25%) had at least one follow-up test (SAC, 2006). The proportion who returned for follow-up testing was somewhat lower in 2007 and 2008 (21 and 16%, respectively; L&I, 2008). Although most of the returning handlers had only one follow-up test, some had multiple follow-up tests during the same season if they continued to handle OP/CB insecticides over a longer time period. ChE inhibition relative to baseline levels may be indicative of overexposure; employers are required to conduct work practice investigations or remove handlers from exposure (with wage protection) depending on the degree of ChE inhibition observed.

Handlers were recruited for this study through two collaborating clinics that provide ChE monitoring services in a highly productive agricultural area in central and eastern Washington State. Many tree fruit orchards are located in this region (primarily apples but also pears, cherries, peaches, and other crops). Several OP/CBs including chlorpyrifos, azinphos-methyl, and carbaryl are widely used in apple orchards (USDA, 2008).

Participants were recruited by Spanish–English bilingual field research staff from the University of Washington in the clinic or at the worksite when they were seen for follow-up ChE testing from April through July. Field staff were present for recruitment at the clinic when most handlers in the ChE monitoring program came in for follow-up testing. Most handlers were seen for follow-up testing in groups with fellow co-workers; up to 10 participants were able to complete the survey at the same time (limited only by the number of computers that were available). During the interviews, research personnel were available to address questions about the survey or any technical difficulties with the instrument. All study procedures were approved by the Institutional Review Board at the University of Washington.

From 2006–2008, a total of 195 participants completed the survey on at least one occasion, which represents 60% of the agricultural pesticide handlers invited to participate in the study. Study participants were similar to all pesticide handlers in the statewide ChE monitoring program in terms of demographic characteristics including age, race/ethnicity, and sex (SAC, 2006). One record per participant was selected at random for the frequencies reported in this manuscript. A subset of 45 participants (23%) completed the survey on at least two different visits during the same year. Among these participants, we selected two survey records from the same year at random to evaluate test–retest reliability for variables with responses that were considered unlikely to change over time. These analyses included demographic characteristics, employment history, health status, and PPE use.

Statistical analyses

Frequencies for personal characteristics and main exposure variables were calculated using data from the entire study population (N

195). Percent agreement and kappa coefficients were used to evaluate reliability of responses for selected characteristics among the subset of participants who completed two separate interviews in the same year (N

45). The 95% confidence intervals for kappa coefficients were calculated using the analytical method for dichotomous variables and the bias-corrected method with 1000 bootstrap replications for categorical variables with three or more possible responses (Efron and Tibshirani, 1986). Weighted kappa coefficients were also calculated for ordered categorical variables using quadratic weights. Statistical analyses were performed using Intercooled Stata 9.2 (StataCorp, College Station, TX, USA).

RESULTS

Surveys were completed by 195 male pesticide handlers. The median duration of the computer-based interview was 27 min (range: 9–62 min). Almost all participants (98%) self-identified as Hispanic/Latino, and all but five participants completed the survey in Spanish (Table 2). The median age among study participants was 33 years (range: 19–57 years). Nearly all participants were able to read in Spanish at least to some degree, though most (56%) reported having a primary school education or less. Relatively few participants (29%) reported being able to read in English. Approximately half (48%) had been employed as pesticide handlers for three years or less. The participants in the test–retest reliability subsample were similar to the other study participants in terms of demographic and other personal characteristics (e.g. age, level of education, and length of employment as a pesticide handler).

Table 2.

Characteristics of study participants, 2006–2008 (N

195)

In terms of pesticide handling activities in the last 30 days, almost all participants (94%) reported applying pesticides and most (69%) reported mixing/loading pesticides (Table 3). Almost all participants (95%) applied OP/CBs in apple orchards; some participants also performed pesticide applications in pear and cherry orchards (24 and 14%, respectively). Chlorpyrifos was the most widely used pesticide (61%), followed by carbaryl (31%) and azinphos-methyl (19%). Of note, handlers who completed the interview in April or May were significantly more likely to report recent use of chlorpyrifos than handlers who were interviewed in June or July (71 versus 23%, P

0.001, chi-squared test) and reported recent use of azinphos-methyl was significantly higher in June–July than in April–May (66 versus 7%, P

0.001, chi-squared test). These findings are consistent with expectations based on typical pesticide use patterns in Washington State tree fruit orchards (Smith, 2006).

Table 3.

Reported exposures in the last 30 days and use of PPE, 2006–2008 (N

195)

In terms of PPE use, the vast majority of study participants reported wearing chemical-resistant gloves (97%), chemical-resistant footwear (95%), and a rain suit (88%). Almost all participants reported wearing some type of respirator; use of half-face respirators was most common (74%). Among half-face respirator users, safety glasses were the most common type of eye protection (80%).

Results of reliability analyses for participant characteristics and reported PPE use are shown in Table 4. The median time span between interviews was 40 days (range: 12–92 days). In general, responses for demographic and other personal characteristics showed substantial agreement across interviews. Percent agreement ranged from 71–98%, and unweighted kappa coefficients ranged from 0.16–0.97. When responses for ordered categorical variables were weighted for partial agreement, kappa coefficients ranged from 0.36–0.98. The only variable with less than moderate agreement was ‘level of concern about pesticide-related health effects’, which is a more subjective question, and may reflect changing perceptions of risk over the course of the spray season. Participants were significantly more likely to report being ‘very concerned’ in the second interview relative to the first interview (93 versus 68%, P

0.008, chi-squared test). Several exploratory analyses were performed to evaluate changes in exposures or other characteristics among the 11 participants who expressed a higher level of concern in the second interview than in the first interview. The only notable differences between interviews were related to reported OP/CB use; more participants reported using chlorpyrifos and carbaryl in the first interview, and reported use of azinphos-methyl was more common in the second interview. Although use of different OP/CBs could potentially explain the increased concern among handlers in the second interview, azinphos-methyl is typically applied later in the spray season, so this observation may simply reflect the timing of pesticide applications with respect to the interview dates. Alternately, the increase in concern may have resulted from greater awareness of pesticide safety issues due to continued participation in the ChE monitoring program and/or completion of the questionnaire.

Table 4.

Test–retest reliability for selected characteristics, 2006–2008 (N

45)

For reported PPE use, percent agreement ranged from 62–100%, and kappa coefficients ranged from 0.15–1.0. We observed substantial agreement for respirator type, glove type, locker use, respirator fit testing, and respirator seal check frequency (κ ≥ 0.63). Moderate agreement was observed for type of footwear (κ

0.48) and use of a chemical-resistant rain suit (κ

0.54). It is possible that PPE use may have changed between interviews, which might account for inconsistencies in some responses. When we restricted this analysis to participants whose second interview took place within 40 days of the first interview (using the median time interval between interviews as the cut point), we observed slightly higher agreement for most PPE variables including respirator type (κ

0.84), eyewear type (κ

0.27), glove type (κ

0.78), use of a chemical-resistant rain suit (κ

0.65), and reported condition of PPE (κ_weighted

0.59).

DISCUSSION

The A-CASI instrument that we used was developed to collect information about recent OP/CB exposures and other personal characteristics from agricultural pesticide handlers who were predominantly Spanish speakers. Results of test–retest analyses showed substantial agreement for most demographic and personal characteristics. Substantial or moderate agreement was observed for some but not all questions regarding PPE use. Although we could not evaluate reliability for reported use of specific OP/CBs, the timing of reported recent use among study participants was consistent with expectations based on typical seasonal pesticide use patterns in Washington State tree fruit orchards (Smith, 2006).

Previous studies have utilized other specialized interviewing methods with agricultural workers. Zahm et al. (2001) described a life-events/icon calendar questionnaire that was used to collect information about lifetime occupational histories among migrant and seasonal farm workers. In an interview, agricultural workers were asked to delineate their lifetime work history on a calendar with assistance from an interviewer. Icons representing major life events (e.g. weddings, births, major illnesses) were placed on the calendar; participants then filled in their work histories (including jobs held, work activities performed, and crops grown) around these memorable events. Engel et al. (2001b) found that it was possible to obtain a more comprehensive and detailed lifetime occupational history in an interview using the icon/calendar questionnaire rather than a traditional questionnaire. Engel et al. (2001a) observed good agreement for job count and crop-specific agricultural work activities when they evaluated test–retest reliability for the icon/calendar questionnaire in a sample of farm workers who were interviewed twice (8–14 months apart). As with the icon/calendar questionnaire, the computer-based survey instrument described in this paper used icons and an interactive platform to facilitate data collection among Spanish-speaking agricultural workers with limited formal education and literacy.

Other computer-based instruments have been utilized to collect data from agricultural workers in previous studies. Rohlman et al. (2003) developed a computer-based test system to assess neurobehavioral function; this system was used successfully in a study among adult and adolescent Latino farm workers in Oregon (Rohlman et al., 2007). To facilitate participation of individuals with limited education and computer use experience, the investigators developed the ‘9BUTTON’ keyboard cover with keys that were larger and more widely spaced. For similar reasons, we used an instrument that could be administered on a touch-screen computer rather than asking participants to enter responses on a standard keyboard.

In addition to using computer-based instruments for data collection in epidemiologic studies with agricultural worker populations, these instruments have also been employed for worker training and safety education. Anger et al. (2004) demonstrated that computer-based training was an effective tool for safety education among immigrant Latino nursery workers with limited formal education. In this training instrument, the content is divided into ‘information sets’, which consist of a series of information screens followed by quiz screens to assess comprehension (Anger et al., 2001). Participants receive feedback based on their responses to quiz questions, and information sets are repeated when participants enter incorrect responses. The computer-based instrument that we used was developed for research purposes and presently does not provide training or feedback to participants. However, it is possible that future versions of this instrument could be developed to provide targeted messages regarding safe pesticide handling practices based on an individual's responses.

Strengths and limitations

Other investigators have noted many challenges in conducting research studies of pesticide exposures and health outcomes with agricultural worker populations, including language barriers, low literacy levels, and limited formal education (Arcury et al., 2006; McCauley et al., 2006). Many of the participants in this study had relatively little formal education, and some reported limited literacy. Moreover, comments from open-ended follow-up interviews with 11 participants indicated that most participants likely had very limited, if any, previous experience using computers (Cox, 2009).

Nonetheless, we found that it was possible to collect information about OP/CB exposures and other participant characteristics using the A-CASI survey instrument. This interviewing method had several strengths that may have facilitated more accurate reporting in this population of Spanish-speaking agricultural pesticide handlers. First, the touch-screen user interface—which included pictures or icons representing possible responses for most questions—and the audio narration made it easier for participants to enter responses, even if they had limited literacy. Second, questions were standardized and delivered in the same manner for all participants. Third, the privacy of entering responses on the computer allowed participants to report potentially sensitive information, such as exposure-related behaviors that are considered to be inconsistent with accepted safe pesticide handling practices (i.e. not washing hands before breaks during pesticide applications).

Although the response rate in this study was 60% and information about handlers who declined to participate was unavailable, the study population appeared to be demographically representative of the overall population of pesticide handlers in the statewide ChE monitoring program (SAC, 2006). The participants who were included in the reliability testing were similar to other study participants in terms of demographic and other personal characteristics, which suggest that this subsample should be representative of the entire study population.

There were several limitations in the design and evaluation of this survey instrument. The lack of heterogeneity in responses for some questions made it difficult to evaluate agreement using kappa coefficients, which are strongly influenced by the prevalence of the characteristic being evaluated (Sim and Wright, 2005). In situations where the prevalence of the characteristic is very high or very low, the kappa coefficient can be low even if there is a high level of exact agreement. This was the case for some of the characteristics that were evaluated in this study. For example, because a high proportion (90%) of participants were educated in Mexico, the kappa coefficient for country of education was 0.48 even though there was 96% exact agreement for this variable.

The inability to evaluate the reliability of all study variables was also a limitation. Because of the relatively long time interval between interviews (median

40 days), we were unable to evaluate the reliability of exposures that were expected to change over time. Although we did evaluate the reliability of PPE variables, depending on the chemicals applied and the handling activities performed during the 30-day time period prior to each interview, it is possible that PPE use may have changed between interviews. Pesticide label requirements for PPE use are different for chlorpyrifos, carbaryl, and azinphos-methyl (the three most commonly reported OP/CBs in this study). We found that wearing a protective face shield was more common among azinphos-methyl users than among chlorpyrifos users (21 and 2%, respectively), which is consistent with the label requirements for these chemicals. It is possible that this may have affected the results of the reliability analysis for type of eyewear, which showed relatively poor agreement (κ

0.22). That is, different PPE use practices during the 30-day period before each interview may explain, in part, the lower levels of agreement observed for some characteristics of PPE use relative to other characteristics that are fixed or less likely to vary over a short time period. To the extent that PPE use differed between interviews, the observed kappa coefficients may be falsely low estimates of true agreement. Interestingly, when we restricted this analysis to the 12 participants who reported using the same chemical prior to both interviews, there was perfect agreement in terms of respirator type. However, the impact of changes in PPE use between interviews on test–retest results was otherwise likely minimal because: (i) we did not observe any other notable differences in PPE use between handlers who reported using chlorpyrifos, carbaryl, or azinphos-methyl and (ii) there did not appear to be any systematic differences in reported PPE use between the first and second interviews (data not shown).

As discussed above, during the first year of data collection some participants had difficulties with questions that required numerical responses, and these items were reformatted as categorical questions in subsequent years. Although some information was sacrificed by replacing continuous responses with categorical responses, we considered this change to be necessary in order to obtain valid data from a greater number of study participants.

As with other questionnaire-based studies, reliance on self-reported data was an inherent limitation of this research project. Missing data were a concern for some questions. In particular, 25 participants (13%) reported on the survey that they were uncertain about which specific OP/CBs they used in the last 30 days. It is possible that recall might be improved for these specific items in future versions of the survey through changes in the question format (i.e. larger response icons, fewer response options per screen).

Finally, the relatively high initial costs of A-CASI instrument development may be a barrier to implementation in some research settings. However, Brown et al. (2008) found that computer-based surveys can be more cost effective than self-administered questionnaires for studies with large sample sizes or when questionnaires are used in multiple studies. Moreover, improvements in technology have reduced the unit cost of portable touch-screen computers, which will likely facilitate the use of computer-based surveys in future studies.

CONCLUSIONS

This study suggests that computer-based survey instruments can be useful tools for occupational exposure assessment. We developed a computer-based survey that has been used to assess self-reported OP/CB exposures and collect other information from Spanish-speaking agricultural pesticide handlers with limited formal education and literacy levels. Analyses of test–retest reliability showed substantial agreement for most personal characteristics and moderate to substantial agreement for most questions related to PPE use. In addition to its present use for data collection and exposure assessment, in the future, this instrument could potentially be adapted to promote safe pesticide handling practices by providing immediate feedback and delivering tailored messages based on the participants' responses.

FUNDING

US National Institute for Occupational Safety and Health/Centers for Disease Control and Prevention (U50-OH07544, T42-OH008433-01); US National Institute of Environmental Health Sciences (P30-ES07033, P42-ES04696, T32-ES07262).

Acknowledgments

We would like to thank the participants in this study. We also gratefully acknowledge the contributions of the following individuals: Joe Cozzetto for assistance with participant recruitment and field data collection; Kit Galvin for comments on the survey instrument; Stacey Holland for providing the photos and illustrations that were used in the survey instrument; Tycen Hopkins and Jim Larkin with Resources Online for software development; and Maria Negrete and Pablo Palmandez for conducting the field data collection.

References

Anger WK, Rohlman DS, Kirkpatrick J, et al. cTRAIN: a computer-aided training system developed in SuperCard for teaching skills using behavioral education principles. Behav Res Methods Instrum Comput. 2001;33:277–81. [PubMed]
Anger WK, Tamulinas A, Uribe A, et al. Computer-based training for immigrant Latinos with limited education. Hisp J Behav Sci. 2004;26:373–89.
Arcury TA, Quandt SA, Barr DB, et al. Farmworker exposure to pesticides: methodologic issues for the collection of comparable data. Environ Health Perspect. 2006;114:923–8. [PMC free article] [PubMed]
Bloom DE. Technology, experimentation, and the quality of survey data. Science. 1998;280:847–8. [PubMed]
Brown JL, Vanable PA, Eriksen MD. Computer-assisted self-interviews: a cost effectiveness analysis. Behav Res Methods. 2008;40:1–7. [PMC free article] [PubMed]
Cox I. Testing the validity of the audio computer-assisted self-interview (A-CASI) among low literate pesticide handlers [thesis] Seattle, WA: University of Washington; 2009.
Efron B, Tibshirani R. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Stat Sci. 1986;1:54–77.
Engel LS, Keifer MC, Thompson ML, et al. Test-retest reliability of an icon/calendar-based questionnaire used to assess occupational history. Am J Ind Med. 2001a;40:512–22. [PubMed]
Engel LS, Keifer MC, Zahm SH. Comparison of a traditional questionnaire with an icon/calendar-based questionnaire to assess occupational history. Am J Ind Med. 2001b;40:502–11. [PubMed]
Hendry D, Mackenzie S, Kurth A, et al. Evaluating paper prototypes on the street. Portland, OR: Conference on Human Factors in Computing Systems; 2005.
Hofmann JN, Keifer MC, De Roos AJ, et al. Occupational determinants of serum cholinesterase inhibition among organophosphate-exposed agricultural pesticide handlers in Washington State. Occup Environ Med. 2010 Published online 9 October 2009; doi:10.1136/oem.2009.046391. [PMC free article] [PubMed]
Hofmann JN, Keifer MC, Furlong CE, et al. Serum cholinesterase inhibition in relation to paraoxonase (PON1) status among organophosphate-exposed agricultural pesticide handlers. Environ Health Perspect. 2009;117:1402–8. [PMC free article] [PubMed]
Kurth AE, Martin DP, Golden MR, et al. A comparison between audio computer-assisted self-interviews and clinician interviews for obtaining the sexual history. Sex Transm Dis. 2004;31:719–26. [PubMed]
L&I. Cholinesterase monitoring of pesticide handlers in agriculture: 2008 report. Tumwater, WA: Washington State Department of Labor and Industries; 2008.
McCauley LA, Anger WK, Keifer M, et al. Studying health outcomes in farmworker populations exposed to pesticides. Environ Health Perspect. 2006;114:953–60. [PMC free article] [PubMed]
Perry MJ, Marbella A, Layde PM. Nonpersistent pesticide exposure self-report versus biomonitoring in farm pesticide applicators. Ann Epidemiol. 2006;16:701–7. [PubMed]
Rohlman DS, Gimenes LS, Eckerman DA, et al. Development of the Behavioral Assessment and Research System (BARS) to detect and characterize neurotoxicity in humans. Neurotoxicology. 2003;24:523–31. [PubMed]
Rohlman DS, Lasarev M, Anger WK, et al. Neurobehavioral performance of adult and adolescent agricultural workers. Neurotoxicology. 2007;28:374–80. [PubMed]
SAC. Cholinesterase monitoring of pesticide handlers in agriculture: 2004-2006. Tumwater, WA: Washington State Department of Labor and Industries; 2006.
Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005;85:257–68. [PubMed]
Smith T. Crop protection guide for tree fruits in Washington. Pullman, WA: Washington State University Extension; 2006.
Turner CF, Ku L, Rogers SM, et al. Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science. 1998;280:867–73. [PubMed]
USDA. Washington's agricultural fruit chemical usage, 2007: apples. Olympia, WA: US Department of Agriculture, National Agricultural Statistics Service, Washington Field Office; 2008.
Zahm SH, Colt JS, Engel LS, et al. Development of a life events/icon calendar questionnaire to ascertain occupational histories and other characteristics of migrant farmworkers. Am J Ind Med. 2001;40:490–501. [PubMed]

Articles from Annals of Occupational Hygiene are provided here courtesy of
Oxford University Press