OccIDEAS is a web application written in Java which links the steps of the expert assessment system and automates some of the assessment steps. There are interfaces which allow users to do a range of tasks such as: develop new JSMs or edit existing ones; change the if/then rules in the JSMs; manage job history data; undertake interviews; view data and automatic assessments; and manually assess exposures.
When developing a JSM for a particular industry, a researcher needs to investigate the industry, the job, and the tasks within the job as well as the main agent exposures. In order to do this, the person who is developing the questionnaire reviews the literature, talks with experts, and collects questionnaires developed for the previous studies (particularly from [3
]). For each JSM there is an associated online collaborative discussion board which contains references used in creating the JSM and the rationale behind decisions to include questions or what level of exposure to assign.
Within OccIDEAS, questions are tagged with the exposure agents relevant to that question. An important design philosophy was to keep questions narrowly focussed to facilitate this tagging. So for example, instead of asking a question such as “What were other workers doing in the area where you were working?” a question might ask “Were you working in the area where metal was poured?” Tagging each question with its associated exposure agents allows automatic removal of a question from an interview if that agent is not of interest to the study, thus shortening the interview. For example, if the hypothesis of a study is that solvent exposure is the causative agent, only those questions relating to solvents would be retained, while ones tagged with other agents such as ionizing radiation or diesel exhaust would be dropped. In our prostate cancer study we based our questionnaires on questionnaires used in a study of non-Hodgkin lymphoma. Because of different hypotheses in the two studies we needed to remove questions relating to solvents and PCBs and add questions relating to oils, fertilizers, and exhaust fumes. This process previously took us several weeks of intensive reviewing and editing to modify the questionnaires. In OccIDEAS it takes less than half an hour as one simply selects the agents of interest and the template JSM is automatically modified to only include questions relating to those agents.
The assessment involves deciding on probability of exposure (none, possible, probable), level (none, low, medium, high), and frequency of exposure (weeks per year and hours per week). The tasks which result in probable exposure (and therefore the questions relating to those tasks) are usually clear, so that decision making rules can be assigned, for example, welding will result in probable UV exposure. The designation of “possible exposure” is used to highlight more difficult cases in which experts may need to examine the context of the job or free text answers in order to assign exposure, for example, not all welders are exposed to high levels of metal fume.
Generally, we define low level as above background but <10% TLV, medium as 10%–100% TLV, and high as >100% TLV at current TLV levels [14
]. The option of “unknown level” is also available. For some agents such as shiftwork, physical activity, or sun exposure there is no TLV and the levels are related to a standard level. For example, shiftwork might be categorized as work over the graveyard (1 AM to 5 AM) shift (high exposure), work at night but not the graveyard shift (medium), and changing shifts but not involving night work (low). All levels used are recorded in the online documentation.
During the questionnaire development, the expert simultaneously develops exposure rules. These rules are if/then statements relating to particular answers to questions and provide an automatic exposure assessment. As a very simple example, in the JSM for forestry workers, questions and their answers in lead to the automatic rules assigning wood dust exposure of “Probable” if the person chopped down trees with “High” level of exposure category if a chain saw was used and a “Medium” level of exposure if a hand saw was used.
Forestry worker JSM questions leading to exposure assessment rule for wood dust.
Rules can also include information from the job history such as the country in which the job was done or the year of employment. Thus, for example, rules can specify that the exposure is “high” before 1983 and “low” afterwards. Level of exposure can be modified by the use of different types of personal protective equipment or ventilation.
During the data collection phase of a study, the job histories of the subjects are obtained and entered, either from a written questionnaire, or directly into the system by an interviewer or the subject. The study researchers or the interviewer then manually link the appropriate JSMs to each job using the title and main tasks as described by the subject. We explored the possibility of linking the JSMs to jobs automatically possibly using fuzzy logic, but the range of descriptions for jobs was found to be too broad to do this accurately.
The participant is then ready to have a computer-assisted interview, which may be done by an interviewer (in person or by phone) or online by the subjects themselves. Status reports can be used to track subjects who require interviews or are awaiting assessments. Interviewers can be trained to do the JSM assignment and to administer the JSM at the same time as the job history is taken; however if the subject enters their own job history, it is necessary to have a two-step process. Once the data collection for a subject is complete, the data from the job history and the answers to any JSMs are ready to be assessed for exposure.
The assessment of the probability, level, and frequency of exposure is performed on an agent-specific level. Assessments can be performed automatically by invoking the rules, or can be done manually. The invoking of the rules to produce an automatic assessment is controlled by the expert, who can run the rules for just one person or for a subset of subjects. The manual and automatic assessments are held separately so it is possible to compare independent assessments. Each triggered rule is displayed for the expert so that he or she can understand why a particular subject was assigned a specific exposure assessment. For each subject, the expert assessors can choose to assess the exposure independently, accept the automatic assessment, or modify the automatic assessment and provide comments on why they chose to do that. The comments are used to improve the template JSMs.
Given the low prevalence of occupational exposures in community-based studies, the rules are designed to be sensitive to possible exposure circumstances. If the exposure is very probable in a task, the rule will usually include a level of high, medium, or low. If there is less certainty whether the task involves exposure, or if the answer to a question is “do not know” then the automatic assessment would assign an “unknown” level. These cases would be priorities for the expert to review manually.
In population-based studies, the large number of jobs with no exposure result in a huge and unrewarding burden for the expert to review manually. In our prostate cancer study [4
], 43% of the subjects had no exposure to any of the agents being assessed (metals, wood, oils, pesticides, fertilizers, exhaust fumes). However each of these unexposed subjects needed to be reviewed by the expert and we estimate this took about a quarter of the time, that is, over 250 hours. In OccIDEAS, all the subjects with no exposure can be reviewed easily in one report and batch confirmed as having no exposure (or individually assigned exposures if required). Since the prevalence of most agents in community-based studies is typically 1%–20% [10
], this means that the expert can concentrate on examining the minority of jobs where exposure is likely rather than the large number of jobs with very low likelihood of any exposure. This represents a big time and hence cost saving for the exposure assessment process, reduces the repetitive nature of the work, and reduces the probability of misclassification. In addition, some agent/job combinations are likely to be less variable than others (e.g., in most nursing jobs an individual would be exposed to blood borne viruses, whereas only a few individuals would be exposed to ionizing radiation). The expert can therefore rapidly accept some of the auto assessments from the consistent combinations and spend their time on the more variable, difficult, and interesting assessments. This is less likely to result in expert assessor “burn out.”