|Home | About | Journals | Submit | Contact Us | Français|
To systematically review comparative research from developing countries on the effects of questionnaire delivery mode.
We searched Medline, EMbase and PsychINFO and ISSTDR conference proceedings. Randomized-controlled trials and quasi-experimental studies were included if they compared two or more questionnaire delivery modes, were conducted in a developing country, reported on sexual behaviours, and occurred after 1980.
28 articles reporting on 26 studies met the inclusion criteria. Heterogeneity of reported trial outcomes between studies made it inappropriate to combine trial outcomes. 18 studies compared audio computer-assisted survey instruments (ACASI) or its derivatives (PDA or CAPI) against another self-administered questionnaires, face-to-face interviews, or random response technique. Despite wide variation in geography and populations sampled, there was strong evidence that computer-assisted interviews lowered item-response rates and raised rates of reporting sensitive behaviours. ACASI also improved data entry quality. A wide range of sexual behaviours were reported including vaginal, oral, anal and/or forced sex, age of sexual debut, condom use at first and/or last sex. Validation of self-reports using biomarkers was rare.
These data reaffirm that questionnaire delivery modes do affect self-reported sexual ehaviours and that use of ACASI can significantly reduce reporting bias. Its acceptability and feasibility in developing country settings should encourage researchers to consider its use when conduct ing sexual health research. Triangulation of self-reported data using biomarkers is recommended. Standardising sexual behaviour measures would allow for meta-analysis.
After almost three decades, the burden of the HIV epidemic remains squarely in developing countries where the disease is primarily spread sexually (UNAIDS 2007; Wilson & Halperin 2008). Sexual behaviour is complex and influenced by many factors such as socio-economic, cultural, biological, and psychological conditions, many of which cannot be easily externally validated or objectively measured.
Another measurement challenge surrounds the private aspect of sexual behaviours. As sexual behaviour cannot ethically be observed directly, we must rely on individuals’ self-reports of their sexual experiences. Although cultural norms differ around the world, most sexual behaviours are socially censured, and strong pressure to conform to societal norms causes self-reports to be fraught with bias, particularly around recall and social desirability.
Over the past two decades a diverse studies (condom usage, discordant couples, condom negotiations) have explored the relationship between risky behaviours and their impact on HIV. Researchers noted a gap between the validity and reliability of the self-reported measure and other outcomes. Ensuing under-reporting of sexual behaviours renders it difficult to interpret trends in HIV prevalence or incidence, to design appropriate behavioural interventions and to interpret their effects. This requires greater attention to improving measurement techniques. The scope for change ranges from questionnaire wording (Wellings et al. 2001; Elam & Fenton 2003; Mavhu et al. 2008) and ensuring privacy and confidentiality (Catania et al. 1990; Fenton et al. 2001; Tourangeau & Yan 2007), to improving questionnaire delivery modes (Mensch et al. 2003, 2008; Tourangeau & Yan 2007).
Traditionally the field has relied on interviewer-administered questionnaires to collect self-reported sexual behaviour information. Growing concern for better greater validity prompted researchers to explore other questionnaire delivery modes. This search has been most radically transformed with the recent advent of computer programming in questionnaire design (Turner et al. 1998; Jones 2003).
In the early 1990s, Catania and colleagues (1990) reviewed the methodological challenges for assessing sexual behaviour, including the effects of questionnaire delivery mode on measurement error. Their conclusion called for more rigorous research into the assessments of sexual behaviours and emphasized that the foundation lay in improving the reliability and validity of its measurement. All 20 studies included in the review on response bias in sexual behaviour research were conducted in North America. At the end of the 1990s, Weinhardt and colleagues (1998) re-examined the literature since Catania’s review on reliability and validity of questionnaire delivery method and assessed the evidence relating to questionnaire delivery mode and reporting of sexual behaviours. Despite developments in the field, such as the use of computer technology, Weinhardt et al. believed that comparative research of questionnaire delivery modes remained limited. Moreover, of the 30 studies included in their review, only three were from developing countries, and only one compared questionnaire delivery modes (Konings et al. 1995).
Two recent developments have provided the impetus for this systematic review. First there has been a rise in the number of comparative studies on questionnaire delivery modes in developing country settings. While the bulk (14/26) of the studies reviewed here took place between 2000 and 2004, almost two-thirds (18/26) were published between 2005 and 2008. Secondly, there has been an increase – albeit still tentative -- in the use of computer-administered modes in developing country settings on sexual behaviour research. Within the comparative research conducted in developing countries, more than half include computer-administered modes. These two developments prompted this systematic review to examine the body of evidence and to summarize the findings around different questionnaire delivery modes regarding sexual behaviour measurement.
Studies were selected for review if they met the following criteria: published or cited in peer-reviewed journals that compared two or more questionnaire delivery modes; conducted in a devel-oping country; including data that reported on sexual behaviour (vaginal, anal, or oral sex, condom use, risky sexual behaviours, contraceptive use), and published between 1 January 1980 and 31 December 2008. In addition, abstracts for the conference proceedings from the International Congress of Sexually Transmitted Infections (ISSTDR) were examined from 2001 onwards (2001, 2003, 2005, and 2007). In this review the countries listed as having ‘emerging’ or ‘developing economies’ by the International Monetary Fund World Economic Outlook report were considered a ‘developing’ country (IMF 2009). Studies were included if they were evaluated in an experimental (RCT), quasi-experimental (i.e. had non-randomised comparison group) or test-retest design. Studies were excluded if they compared one questionnaire delivery mode against a biological marker, the impact of interviewer gender on questionnaire responses or data reported by married couples rather than between questionnaire methods. Unpublished studies emanating from references of published articles and studies published in non-English language journals were considered for inclusion.
Three databases were searched: Medline, EMbase, and PsychINFO using key MeSH terms and text words relevant to each data base (Table 1). Duplicates were manually discarded.
Titles and abstracts were used to screen for relevance to the literature review. If the questionnaire delivery method was not mentioned in the title or abstract, it was assumed there was no mode comparison. Where the title or abstract were not sufficient to make a determination, the article was downloaded and read. Reference lists for all included articles were examined. 3% of articles were re-examined blind by one co-investigator (FC) to check that inclusion criterion were being met. Articles where inclusion criteria were unclear (n=119) and all articles included in the review were jointly discussed by two of the investigators (FC & LL).
Of the 6824 references reviewed (Medline=3261; Embase=1761; PsycINFO=1800, ISSTDR=2), 28 articles reporting on 26 studies met the inclusion criteria (Figure 1). Articles reporting on results from more than one study were analysed separately. Where studies were reported in more than one article; these results were combined for analysis. Studies ranged geographically (from China, to Hanoi, to rural Malawi) and in their selection of respondents (from female sex workers to South African students). Self-administered questionnaires (SAQ) were used as the comparison in 5 studies (Table 2) whereas interviewer-administered questionnaires were the comparison in 16 studies (Table 3). Seven studies included SAQ, an interviewer-administered mode, and at least one other mode for comparison (see Table 4).
This analysis makes the general assumption that an increase in reporting of a socially censured behaviour indicates more accurate reporting (reducing social desirability bias and increasing validity) (Konings et al. 1995; Durant & Carey 2000; Kreuter et al. 2008; Brener et al. 2003).
All 5 studies that compared another method against SAQ used an audio-computer-assisted survey in-strument (ACASI) or a derivative thereof (e.g. personal digital assistant (PDA) or phone-ACASI) (see Table 2). Two were randomized control trials (Rumakom et al. 2005; Seebregts et al. 2008), one was quasi-experimental (Fielding et al. 2006), and two focused on test-retest (Jaspan et al. 2007; Bernabe-Ortiz et al. 2008). Three studies included school attenders (Rumakom et al. 2005; Jaspan et al. 2007; Seebregts et al. 2008).
Where non-response rates were reported, SAQ performed poorly against ACASI and its derivatives. In South Africa, SAQ respondents were seven times more likely to have missing items than respondents using PDA (Jaspan et al. 2007). In China, nonresponse rates to items were 8% to 14% with SAQ and 0.4-3.0% with phone-ACASI (Fielding et al. 2006).
In general, SAQ respondents reported lower levels of risk exposure than respondents using computer self-administered modes (e.g. in China, bisexual sex (SAQ 0.6% versus ACASI 2.6%, p<0.001; sex in the past year SAQ 50.3% versus ACASI 56.5%, p=0.006; and belief that their partner had sex with others SAQ 4.3% versus ACASI 7.0%; p=0.005) (Jaspan et al. 2007). The exception was among South African youth where there were no differences between SAQ and PDA except for ‘ever sex’, which was more often reported in SAQ (36%) than PDA (26%; p=0.003) (Jaspan et al. 2007).
In three studies respondents used SAQ followed by PDA (Jaspan et al. 2007; Seebregts et al. 2008; Bernabe-Ortiz et al. 2008). Response agreement between the two modes was high at 85% and a Kappa value of 0.5 or more.
Respondents were more likely to report that ACASI and its derivatives were more confidential when answering sexual behavioural questions. For example, in the Thai study, 7.8% of college females who used SAQ reported feeling embarrassed answering the sexual behaviour questions, whereas fewer than 1.5% of female ACASI respondents did (Rumakom et al. 2005).
Seven of the 16 studies compared face-to-face interviewing against ACASI (Table 3). Five studies explored an adaptation of face-to-face interviewing which allowed respondents to self-report sensitive questions on a ballot card (Phillips et al. 2007; Gregson et al. 2002, 2004; Hanck et al. 2008; Jaya et al. 2008). One study from India compared an additional mode termed ‘interactive interviewing’ which included a tape-recorded drama and dolls to desensitise respondents around sensitive issues (Jaya et al. 2008). Four other studies compared face-to-face interviewing against in-depth interviewing (Konings et al. 1995), coital diaries (Ramjee et al. 1999; Allen et al. 2007) or used a derivative of SAQ where the questions were read aloud in a group setting (Plummer et al. 2004a,b).
As shown in Table 3, eight studies were randomized control trials (Simoes et al. 2006; Allen et al. 2007; Caceres et al. 2007; Minnis et al. 2007; Phillips et al. 2007; Mensch et al. 2008; Jaya et al. 2008; Hewett et al. 2008), five were quasi-experimental studies (Konings et al. 1995; Gregson et al. 2002, 2004; Bernabe-Ortiz 2008; Hanck et al. 2008), and three focused solely on test-retest (Ramjee et al. 1999; Plummer et al. 2004a,b).
As interviewer presence renders it more difficult for respondents to ignore a question, few studies (3/15) reported non-item response rates. In Peru, where data entry was handled manually (FTFI) or directly into a PDA, missing responses were significantly more common using FTFI (p<0.001) (Bernabe-Ortiz 2008).
Overall, respondents using face-to-face interviewing reported fewer sensitive behaviours than respondents using other questionnaire delivery modes. Six of 7 studies that compared face-to-face interviewing with ACASI (RCT=6) showed increased reporting of various sexual behaviours in ACASI. For example, urban Brazilian women were more likely to report behaviours risking STI in ACASI (e.g. anal sex in the last six months: 33% versus 24%; p<0.01; no condom use with vaginal sex in the last month: 59% versus 51%; p<0.01) (Hewett et al. 2008).
In a study in Zimbabwe on hormonal contraception where eligibility of respondents was conditional on specific behaviours, ACASI users were more likely to report an undesirable behaviour (Minnis et al. 2007). For example, not getting pregnant was a study condition and more pregnancies were reported in ACASI interviews than in face-to-face ones (OR=1.5; 95% CI 1.1-1.9). ACASI users were also more likely to report multiple partners than FTFI users (OR=5.7; 95% CI 2.1-15.2).
Data from India produced less consistent results where face-to-face interviewing was compared against both ACASI and interactive interviewing (Jaya et al. 2008). Among females more sexual behaviours were reported using interactive interviewing than face-to-face interviewing, and they consistently reported fewer sexual behaviours using ACASI than FTFI. However, males using ACASI were more likely to report having had sex (26.9% versus 21.4%; p=0.03) and having been forcibly touched (26.2% versus 21.4%; p=0.09) than males using FTFI.
In two studies there were no reported differences between ACASI and interviewer data collected using a computer (CAPI) (Jasper et al. 2007; Bernabe-Ortiz 2008). This was attributed to smaller sample sizes.
When face-to-face interviewing was compared with a non-ACASI mode (ICVI, coital diaries, in-depth interviewing, and assisted self-completed questionnaires-see definitions at the end of Table 3), all 8 studies found lower rates of reporting of sensitive behaviours using face-to-face interviewing. This was true for both socially desirable and socially undesirable behaviours. While data from coital diaries provided increased reporting of sexual behaviours (Allen et al. 2007), diary loss or incompletion posed a problem (Ramjee et al. 1999; Sedyaningsih-Mamahit et al. 2003). Four studies examined interviewer-controlled voting instrument (ICVI), where respondents marked their responses to sensitive questions privately and then posted them into a locked box (Gregson et al. 2002,2004; Phillips et al. 2007; Hanck et al. 2008). Overall there was an increase in reports of sexual behaviours among ICVI users (e.g. in Zimbabwe: multiple sex partners males OR = 1.33; p = 0.028; females OR = 5.21; p<0.001). Only among transgendered males was there increased report of some behaviours but not others (Phillips et al. 2007) (Au: This sentence is strange English; and what precisely does it mean? Which behaviours? Please rephrase.)
In the three studies which examined respondent agreement where differences occurred, reporting was less likely to have occurred during face-to-face interviews (Konings et al. 1995; Plummer et al. 2004a,b; Mensch et al. 2008). Tanzanian pupils were administered the same questionnaire in two modes (FTFI or ASCQ where respondents had the questions read aloud to them in a single-gender group setting) five weeks apart (Plummer et al. 2004a,b). 62% of males and 41% of females reported having sex in both surveys (Plummer et al. 2004a). There was 64.4% agreement in reporting of age at first sex and 47.3% agreement around the number of sexual partners (Plummer et al. 2004b). Respondents were more likely to report condom use, forced sex, and pregnancy in ASCQ. For example, forced sex was reported by only one female in both surveys, but it was reported by 12.3% of females in ACSQ and by 0.2% in FTFI (Plummer et al. 2004b). In Uganda, 23.3% of respondents reported not engaging in sex when asked in the face-to-face interview but then reported sexual activity in an indepth interview (Konings et al. 1995).
There were seven studies that compared both SAQ and FTFI with at least one other mode .Six of these were conducted as randomized control trials (Mensch et al. 2003; Hewett et al. 2004; Potdar & Koenig 2005; Le et al. 2006; van Griensven et al. 2006; Langhaug et al. 2007, one used a quasi-experimental design (Lara et al. 2004; Table 4).
Data on rates of reporting sexual behaviours are less clear. In India, among college males using SAQ, ACASI or FTFI, men responding to ACASI reported heterosexual sex (AOR = 1.8, p<0.05), oral sex with a female (AOR = 2.08; p<0.05), homosexual sex (AOR = 8.1; p<0.05) and having experienced coercive sex (AOR = 11.35; p<0.01) more often. When using ACASI rather than FTFI, young men living in slums were more likely to report masturbation (AOR=22.53; p<0.001) and oral sex with a woman (AOR = 2.4; p<0.010) than vaginal sex (AOR = 0.23, p<0.001).
In Viet Nam young people’s attitudes and norms around sexual behaviours and condom use were compared using SAQ, FTFI, and ACASI (Le et al. 2006). Respondents using ACASI were more likely to report liberal attitudes around premarital sex (a socially censured norm) and to report less confidence in their ability to access condoms (a socially condoned behaviour). Female refusals to answer questions about condom use were most frequent among SAQ and FTFI users. ACASI users were also more likely to report sex (e.g. unmarried males’ sex with commercial sex workers OR=2.8, p<0.05; males aged 15-19 having sex OR = 2.79, p<0.05).
Three studies compared four modes (Lara et al. 2004; van Griensven et al. 2006; Langhaug et al. unpublished). In the Thai study, data generally showed no difference in reporting of sexual behaviours between SAQ, ACASI, and PASI. However, regarding self-reports of the most sensitive behaviours, there was a statistically significant difference (p<0.001) between PASI and FTFI (history of oral sex 37.3% versus 13.2%; sex today/yesterday 19.3% versus 6.1%; sold sex 8.2% versus 0.9%; bought sex 8.2% versus 2.5%). In the Zimbabwean study, after adjusting for covariates, Audio-SAQ and ACASI users were twice as likely to report sexual activity as SAQ users (Audio-SAQ AOR = 2.05 [95%CI: 1.2-3.4]; ACASI AOR = 2.0 [95%CI: 1.2-3.2]), with no reporting difference between ICVI and SAQ users (ICVI AOR = 1.0 [95%CI: 0.6-1.8) (Langhaug et al. 2007). ACASI users reported a younger age at first sex (0.7-1.7 years lower) (p<0.045). In a post-survey questionnaire, ACASI users reported improved ability to answer questions honestly (p=0.004) and believed their answers would be kept secret.
ACASI only performed poorly compared to the random response technique (RRT). RRT asks a respondent to answer one of two questions picked randomly where the interviewer is blinded to the question being answered. One question is of a sensitive nature whereas the other question is not sensitive and for which there is a known probability for that answer in a population (e.g the question, ‘were you born in April’ has a known probability.). Mathematical techniques allow indirect estimates of the pro-portion reporting the sensitive behaviour (Lara et al. 2004). In three sub-populations in Mexico (Lara et al. 2004), RRT users reported the highest rates of attempted abortion. Differences were only statistically significant in the hospital survey (RRT = 22%, SAQ = 19%, ACASI = 13%, FTFI = 12%, p = 0.012).
Whilst 5 of the 26 studies used biological markers as external comparators, only 4 examined sexual biomarkers (Mensch et al. 2008; Plummer et al. 2002,2004; Langhaug et al. 2007; Hewett et al. 2008), one study evaluated drug use (van Griensven et al. 2006). Sexual biomarkers comprised Chlamydia, gonorrhoea, trichonomoniasis, HIV, and current pregnancy in females. While prevalence of biological markers varied among studies involving adolescents, absolute numbers were relatively small. In the study from Brazil among urban women, those who used ACASI had stronger associations between their reported risk behaviours and STIs (Hewett et al. 2008). STI positive respondents using FTFI were more likely to underreport sexual behaviours than their STI negative peers.
The results outlined here reaffirm that questionnaire delivery modes do affect self-reported sexual behaviour (Tourangeau & Smith 1996; Turner et al. 1998; des Jarlais et al. 1999; Metzger et al. 2000; Tourangeau & Yan 2007). This systematic review is the first to examine data from developing countries that compares self-reports of sexual behaviours between various questionnaire delivery modes. Despite wide variation in geography and populations sampled, we found strong evidence that computer-assisted interviewing decreases non-response rates to items and increases rates of reporting of sexual behaviours. This was true when ACASI and its derivatives (PASI, phone-ACASI, PDA) were compared with other self-administered and interviewer-administered questionnaire delivery modes. Comparative research using ACASI was predominantly conducted among young people (11 out of 16 studies were exclusively with young people and the other 5 included individuals aged 18 and older), whose reporting of sexual behaviours is very likely to be socially censured (Mensch et al. 2003, 2008; Hewett et al. 2004; Rumakom et al. 2005; Potdar & Koenig 2005; Le et al. 2006; van Griesven et al. 2006; Jaspan et al. 2007; Langhaug et al. 2007; Seebregts et al. 2008). Data entry errors were also reduced when controlled by a computer programme: in studies with no differences in reporting of sexual behaviours, ACASI still improved the quality of data entry (Jaspan et al. 2007; Seebregts et al. 2008; Bernabe-Ortiz et al. 2008).
Validation of self-reports against biomarkers for sexual activity were rarely available. Where sample sizes were sufficient, results suggest more accurate reporting using ACASI than face-to-face interviewing.
These studies also support the acceptability and feasibility of using computers in developing country settings. In those studies where it was examined, ACASI and its derivatives were found acceptable, easy to use, and respondents, particularly young women, reported feeling more comfortable using a computer to report sensitive behaviours than they did with other methods (Rumakom et al. 2005; Le et al. 2006; Langhaug et al. 2007). Similar findings have emerged from Zimbabwe and the US (Millstein & Irwin 1983; Kissinger et al. 1999; Metzger et al. 2000; van de Wijgert et al. 2000; Kurth et al. 2004). Acceptability of computer technologies may vary geographically and be related to level of exposure. Generally, an increased sense of trust and sense of privacy is expressed by those who live in countries where computers are less commonly used. Only one study compared ACASI against ICVI and found less reporting of sensitive behaviours using ICVI (Langhaug et al. 2007). Results of studies using ICVI or interactive interviewing which did not compare ACASI emphasize that any effort made to improve privacy levels when answering sensitive questions increases the reporting of sexual behaviours (Gregson et al. 2002; Hanck et al. 2008; Jaya et al. 2008). ACASI has not been compared against interactive interviewing. This research is needed in order to better establish their comparative strengths and limitations.
Equally encouraging is the work of Mensch 2008 is research in adol in Malawi and Hewett 2004 is research in Kenya, which suggests that levels of literacy per se may not affect ACASI, as it was used successfully among rural youth. Potdar and Koenig (2005) used ACASI (and not SAQ) among young people living in Indian slums, where despite differences in reporting to college-educated peers, no mention was made of their inability to use the computers. However, evidence from three studies is not sufficient to suggest that ACASI can always be used in settings with low literacy rates; more feasibility research is required.
While coital diaries also demonstrated increased reporting of sensitive behaviours (Ramjee et al. 1999; Allen et al. 2007), they carry requirements which render them less suitable for large surveys: more logistical support, more time for data entry, and specific training to ensure appropriate completion. Coital diaries have a low completion rate (20% in one study). However, including coital diary data from a sub-sample of a large survey population can complement the data collected by other means.
One of the strengths of this review is that a number of the studies reported both socially censured and socially condoned behaviours (Lau et al. 2003; Sedyaningsih-Mamahit et al. 2003; Potdar & Koenig 2005; Le et al. 2006; Simoes et al. 2006; Minnis et al. 2007; Hanck et al. 2008). Conclusions drawn from these studies are strengthened when users of a mode are not only more likely to report socially censured behaviours but also less likely to report socially acceptable behaviours(Catania et al. 1990). Reports for computer self-administered questionnaires followed this pattern. In India, more college men using ACASI reported behaving violently after drinking than those using FTFI (3.0 vs. 1.7%) (Potdar & Koenig 2005). Similarly, in a study in Zimbabwe where hormonal contraceptive use was a prerequisite for participation, women were more likely to report that they were not using them in ACASI than in FTFI (Minnis et al. 2007). Equally heartening is the growing comparative literature around questionnaire delivery modes in developing countries and the increased interest in ACASI and its derivatives.
There are, some limitations to this review. A number of studies did not show statistically significant differences around reporting of sexual behaviours between questionnaire delivery modes. This is in part attributable to the small sample size of these studies, or when youth were sampled, to the small number who reported sexual behaviours overall. Studies did not report the same sexual behaviour outcomes, nor did they always disaggregate their data by gender or age. This made it difficult to make comparisons across studies. Only four studies included biological markers of sexual behaviour as part of their analysis. Biological markers offer complimentary evidence that can be used to explore directions of effect. As mentioned above, for most sexual behaviors which are socially censured, particularly for young people, it is assumed that an increase in reporting indicates a improved validity of that report, but the ability to triangulate against objective data improves our understanding of the differences in self-reported sexual behaviours between questionnaire delivery modes. Researchers should incorporate biological markers (or other externally valid outcomes) into evaluations whenever possible to broaden the evidence within comparative studies. We did not examine the effect of interviewer age or gender. Research on this has been extensive but inconclusive (Becker & Sosa 1992; Blanc & Croft 1992; Catania et al. 1996; Elam & Fenton 2003; Wellings et al. 1990).
In 2003, a technical meeting on “Measurement of Trends in Sexual Behaviour” called for more rigorous comparative studies before anything more definitive could be concluded (Cleland et al. 2004). Since then, there has been a noteworthy increase in the number of published articles in peer-reviewed journals comparing questionnaire delivery modes. Most articles considered in this review, which focussed exclusively on research performed in development country settings, were published after 2003 (n=21/28). Their data strongly suggest the use of computer-assisted methods.
This is important in view of the fact that the principal data collection tool for sexual behaviours in developing countries remains the interviewer-administered questionnaire. While interviewer-administered questionnaires remain an important tool for collecting survey data, this review provides good evidence that self-administered options, especially those using computers, will enhance data quality, particularly of socially sensitive data.
We express our profound gratitude to Angela Young, the wonderful librarian who assisted in structuring this systematic review.