Per participant, interviews revealed most barriers and facilitators for using a new genetic test. On average, interview participants produced 1.9 items and 8.4 relevant remarks per participant, in comparison to 0.9 items and 3.8 remarks for focus group participants and 0.7 items and 1.7 remarks for questionnaire respondents. Although interviews revealed more items per participant, the total number of different items was similar to that revealed by the focus groups. Both methods were needed to reveal all different items present in the study population. In total, interviews revealed 29, focus groups 30 and questionnaires only 21 items. All three methods disclosed items that could influence the use of the test that had not been revealed by the literature search.
To our knowledge, there are only a few studies comparing the output of involvement methods (Fern 1982
; Folch-Lyon et al. 1981
; Kaplowitz 2000
; Ward et al. 1991
; Wutich et al. 2010
). Kaplowitz (2000
) studied the value of mangrove wetlands among residents living in Yucatan, Mexico and compared focus groups and interviews. The authors showed that the interviews revealed more different discussion topics than the focus groups, while we found that the total number of items was about equal. Fern (1982
) who compared the number of unique items (ideas) regarding communication strategies or concerns on job opportunities for women suggested in focus groups and interviews concluded that focus group participants produced only 60% to 70% of the items that would have been produced in an individual interview. In our focus groups, participants produced 47% (0.9/1.9 pp) of the items of the interview participants. Unfortunately, both Kaplowitz (2000
) and Fern (1982
) did not study the differences and similarities of the output contents. Fern (1982
) investigated the differences between interviews and questionnaires (“individuals working alone”) and between questionnaires and focus groups. They also found that interviews revealed more relevant items than questionnaires. However, in contrast to our study, the authors concluded that questionnaires revealed more relevant items than focus groups. Possibly, the complexity of our study topic (genetics and genetic testing) in comparison to the topic of the study of Fern and colleagues (job opportunities for women) could account for the observed differences. Participants in our focus groups and interviews often asked for clarification concerning genetics and genetic testing. The questionnaire participants did not have this opportunity. Clearly, complex topics are less suitable for the detection of new items through questionnaires. Furthermore, combining qualitative methods (triangulation) is mentioned to be an important criterion for finding all different opinions and views in a particular population (Bryman 2001
; Denzin and Lincoln 2000
; Kvale 1996
). Similarly, in our study, both focus groups and interviews were needed to reveal all different items in the study population. The questionnaires did not add any items that were not already mentioned during the other two methods.
In contrast to our findings, Folch-Lyon et al. (1981
), who compared the attitudes towards contraception in Mexico with focus groups and questionnaires, found no apparent differences between the attitudes (items) revealed by the two methods. Similarly, Ward et al. (1991
) who compared the outputs (items) of focus groups and questionnaires of three studies on family planning also found that the outputs of both methods were highly similar. The authors concluded, however, that focus groups brought forward more in depth-information than questionnaires. Wutich et al. (2010
) who compared the output (remark percentage) of these methods about sensitive topics in water-policy development concluded that the methods revealed similar remark percentages when they concerned low or moderately sensitive topics. The authors also concluded that the focus groups could reveal higher statement percentages when discussing very sensitive topics with an opportunity to exchange important information or present a solution. In our study, we discussed hand eczema, an occupational disease that is very common among nurses. Although HE is probably not a very sensitive topic, participants may consider HE to be serious, and viewing the test as an opportunity for HE prevention may have stimulated discussion. Again, the complexity of our study topic may have enlarged the difference between the output per participant of the focus groups and interviews and that of the questionnaires.
This study is one of the first comparing stakeholder involvement methods on revealing items that could influence the use of a new health-related knowledge product, such as a genetic test. Our study has several limitations. Although we carefully developed the protocols for all three involvement methods based on experience and literature, the reliability and validity of the involvement methods can be affected by the way it is conducted and evaluated. This topic needs some consideration. A limitation could be the effect of the interviewers (MR, MV and MMV) and focus group (MR) moderator on the output (Denzin and Lincoln 2000
). Although they are supposed to stimulate discussion, making use of a moderator or interviewer may induce socially desirable answers from the participants. This in turn may decrease the reliability and validity of the findings. Another issue may concern participant recruitment and compensation. We tried to minimize the effects of these issues by standardization between methods, by for example, matching the recruitment technique and the amount of compensation. Also the coding process and its resulting taxonomy was a subjective process that included the interpretation of data by MR and MV. Possibly, other researchers would have preferred different domains and items. Furthermore, some items may overlap or fit in more than one domain. Nevertheless, the large differences in output (per participant) between interviews and questionnaires and between focus groups and questionnaires would most likely have remained. Another limitation concerns our method used to establish the point of data saturation and its potential influence on the output per participant. As customary, we established the point of data saturation as part of an ongoing process in data collection. Based on experience, we expected to need between four and six focus groups, between nine and 15 interviews and between 15 and 50 questionnaires to reach saturation. As a rule of thumb we used 30% of the minimum expected number, as the number of successive focus groups, interviews or questionnaires needed to indicate saturation (respectively, 1, 3 and 5). However, we chose to use two subsequent focus groups instead of one because using the output of only one subsequent focus group may be too dependent on chance. One may hypothesize that one focus group with five to eight participants has a larger impact on the output per participant than one individual interview or questionnaire. Nevertheless, a second analysis excluding the last two focus groups, three interviews and five questionnaires shows a largely similar distribution of the number of relevant remarks per participant: 7.5 for focus groups, 10.5 for interviews and 2.7 for questionnaires.
Another constraint is the observed group difference in training level and gender. The group of questionnaire respondents included more high training level student nurses (78%) than the focus groups (55%) and interview participants (53%). An expected effect of this difference is that more items and remarks would be revealed in the group with high training level nursing students because they may possibly have had more reflection on this topic. However, a subgroup analysis showed the opposite. A similar analysis on possible effects of gender on the output within the questionnaire group showed that the female respondents revealed a similar amount of items and remarks than male respondents. Next, the percentage of participants that were not willing to use the test was significantly higher for the interviews than for the focus groups and questionnaires. A more thorough inspection of data on individual level showed that not-willing interview participants, on average, revealed more remarks than the participants who were willing or were doubtful. Possibly, interview participants who were not willing to use the test had reflected more extensively on the advantages and disadvantages of the test. However, in the questionnaires, the number of remarks per participant did not show a tendency to differ among the participants who were and who were not willing to use the test. Therefore, it is not clear whether the ratio of participants willing and not willing to use the test influenced the higher number of remarks per participant. Furthermore, the specific nature of our studied research product, a genetic susceptibility test meant for a specific stakeholder group in a specific context, limits the generalisability of our study findings. Still, our findings on the output of different user involvement methods are probably useful when evaluating views of intended users to other genetic tests. We recommend that future research studies repeat our study design for different research products and tools in different contexts. Last, this study only compared the involvement methods on output per participant. Future studies could evaluate the efficiency of the involvement methods more thoroughly, by also addressing the more qualitative aspects of the output, e.g. the quality, depth or breadth, and by including all costs and benefits, e.g. time and effort spent on recruiting participants and transcribing or analysing data. Because in this study questionnaires only revealed a small part of the barriers and facilitators, time spared only using questionnaires was outweighed by the limited output. We estimate that overall, interviews seemed most efficient in terms of cost and benefits. Time spent to recruit participants was in favour of the interviews as we only needed 15 participants. Furthermore, the time needed to prepare and execute the focus groups and interviews was similar, although two researchers were needed to guide the focus groups. We estimate that the time to analyse the output was similar for both methods.