Use of a Web-based Questionnaire in the Black Women's Health Study

doi:10.1093/aje/kwq310

Am J Epidemiol. 2010 December 1; 172(11): 1286–1291.

Published online 2010 October 11. doi: 10.1093/aje/kwq310

PMCID: PMC3025633

Copyright American Journal of Epidemiology © The Author 2010. Published by Oxford University Press on behalf of the Johns Hopkins Bloomberg School of Public Health. All rights reserved. For permissions, please e-mail: journals.permissions@oxfordjournals.org.

Use of a Web-based Questionnaire in the Black Women's Health Study

Cordelia W. Russell,^* Deborah A. Boggs, Julie R. Palmer, and Lynn Rosenberg

^*Correspondence to Cordelia W. Russell, Slone Epidemiology Center, Boston University, 1010 Commonwealth Avenue, Boston, MA 02215 (e-mail: drussell/at/bu.edu).

Received May 17, 2010; Accepted August 6, 2010.

This article has been cited by other articles in PMC.

Abstract

The authors assessed the utility and cost-effectiveness of using a World Wide Web-based questionnaire in a large prospective cohort study, the Black Women's Health Study (BWHS). In 1995, 59,000 African-American women were recruited into the BWHS through a paper questionnaire. Follow-up paper questionnaires have been mailed every 2 years since then. During the 2003, 2005, and 2007 questionnaire cycles, participants were given the option of completing a Web-based questionnaire. The cost of developing and processing a returned paper questionnaire was 4 times that of a returned Web questionnaire, primarily because of return postage costs and greater processing time for paper questionnaires. The proportion of respondents who completed a Web questionnaire doubled from 2003 to 2007, from 10.1% to 19.9%, but the characteristics of those completing the Web questionnaire remained the same. Web response was greatest at younger ages (20.9% of those aged <30 years) and declined with age to 3.6% among women aged 60 years or more. Web questionnaires were filled out more completely than paper questionnaires, regardless of the sensitivity of a question. The use of a Web questionnaire in the BWHS resulted in cost savings and more complete responses. Although there are advantages to using a Web questionnaire, the use of multiple means of soliciting questionnaire responses is still needed.

Keywords: African Americans, data collection, follow-up studies, Internet, questionnaires, women

Over the past decade, the use of the Internet in North America has increased by an astounding 134%, with 74% of the population having access to the Internet in 2009 (1). Younger age groups continue to have the highest rate of Internet use, but rates are increasing in older age groups. Since 2005, persons aged 70–75 years have had the largest rise in Internet usage, with prevalence increasing from 26% to 45% (2). The increase in familiarity and trust in all age groups has allowed survey researchers to develop Internet-based questionnaires as an alternative method for obtaining survey responses.

In studies of physicians, patients, and laypersons in both the United States and abroad, use of the Internet for health surveys has been shown to decrease the number of missing responses, contribute to cost savings in person-time and printing and postage, speed up response time, and increase the reliability of the data obtained (3–15). For small samples, however, mailing of multiple paper questionnaires may be a more cost-effective method for eliciting completed questionnaires, due in part to the cost of designing and implementing a World Wide Web survey (16). Use of a mixed-mode method, offering participants the option to use either a Web questionnaire or a paper questionnaire, has been shown to yield a higher response rate (17, 18). We assessed the utility and cost-effectiveness of using a Web-based questionnaire in conjunction with a paper questionnaire in a large cohort study, the Black Women's Health Study (BWHS).

MATERIALS AND METHODS

The BWHS, a prospective follow-up study of 59,000 African-American women, began in March 1995. Black women aged 21–69 years from all regions of the United States were enrolled in the study through postal questionnaires sent to subscribers to Essence magazine, members of black professional organizations, and friends of respondents (19). Enrollment was completed in 1995. Twice a year, participants have received a newsletter updating them on the status of the research and informing them of current and future questionnaire mailings. Follow-up questionnaires have been mailed every 2 years (20). Eight mailings of paper questionnaires were made throughout each 2-year period. The first paper mailing of each cycle was carried out using nonprofit postage; the fourth and seventh mailings were done by priority mail; and the other 5 questionnaires were mailed first class. Since 1999, participants have been asked to provide us with their e-mail address. During the 2003, 2005, and 2007 questionnaire cycles, all participants in the BWHS were given the option of completing either the paper version of the questionnaire or the Web version. Data collected from these 3 follow-up cycles were used for the present analyses.

The BWHS questionnaires contained a wide range of questions assessing current health status and exposures. The participants were asked to report on their medical conditions, current medication usage, weight, menstrual/reproductive history, physical activity, cigarette smoking, and alcohol use. Dietary intake was ascertained with a modification of the short Block-National Cancer Institute food frequency questionnaire (21). Additional questions about education, occupation, health care, religion, insurance, waist/hip measurement, violence, night work, and other factors were added to particular questionnaires. The women's addresses have been linked to 2000 US Census data at the block group level. A neighborhood socioeconomic status score was derived through factor analysis based on 6 variables representing wealth and level of education (e.g., median housing value), with a higher score indicating higher socioeconomic status (22). Prudent and Western dietary patterns based on the food frequency responses were identified through factor analysis (23).

Both the Web and paper versions of the questionnaire were designed in-house, allowing us to accurately assess the amount of time needed for development of the questionnaires. The first Web questionnaire required 120 hours of a computer programmer's time to design, test, and implement. The time required has subsequently decreased, with design, testing, and implementation taking 100 hours in 2007. The paper questionnaire was designed in-house using TeleForm software (Autonomy Cardiff, Inc., Vista, California). Approximately 40 hours of a programmer's time were required to design and test each new paper questionnaire.

Each Web questionnaire was designed with the same questions and layout as the corresponding paper questionnaire. All primary questions were permanently visible on the Web questionnaire (i.e., they could not be skipped). Both the paper questionnaire and the Web questionnaire directed participants to skip subquestions if they did not apply, with a visible line pointing to the next question and indented subquestions on the paper questionnaire and shaded or invisible subquestions on the Web questionnaire. The Web questionnaire reminded participants if they had skipped a primary question but did not require them to complete this skipped question in order to move on to a subsequent page of the questionnaire.

At the beginning of each of the 3 follow-up cycles under study, participants who had provided us with at least 1 e-mail address and who were known to be alive and had not refused further participation were sent an e-mail message with a link to the latest BWHS Web questionnaire. A reminder e-mail was sent 3 weeks later to all participants who had not yet completed the Web questionnaire. All those who did not respond to the Web questionnaire within the first 6 weeks were mailed a paper questionnaire. In the cover letter of the paper questionnaire, there was a reference to the Web questionnaire, informing participants of the option to complete either questionnaire. This allowed women who did not previously provide us with an e-mail address an opportunity to complete a Web questionnaire. Every 2–3 months, for 2 years, a follow-up mailing was sent to all nonresponders, again listing the Web questionnaire as an option. A reminder e-mail was also sent out to all nonresponders 5–6 times during the follow-up cycle. Additional reminders to complete questionnaires either by mail or by Web were included in the BWHS biannual newsletters. At the end of a 2-year follow-up cycle, persons who did not respond to the paper or Web questionnaire were considered nonrespondents.

Once the paper questionnaires had been returned, staff carried out the following quality control steps: 1) Record receipt of the survey; 2) separate data pages from contact information pages; 3) review all pages for stray marks or text outside of data fields that would not be caught by the scanner; 4) scan the surveys; 5) verify the accuracy of the scanned data; 6) update addresses; 7) code open text responses; 8) review all dates of birth to confirm that the right individual had completed the survey; and 9) review all outliers for particular questions. Web surveys did not require the first 5 quality control steps. Because the participant's month and year of birth were required to access the Web questionnaire, review of birth dates was minimal for the Web questionnaire. In addition, minimal work was required to check outliers on the Web questionnaire, because it included prompts to have participants review their response if a response they had entered would be considered out of range. To calculate the time needed to process each returned questionnaire, 3 staff members each completed the above tasks on 100 records for each of the 3 follow-up cycles. Their time per task/cycle was then averaged to estimate staff processing time for each paper and Web questionnaire received.

RESULTS

The cost of developing and processing a returned paper questionnaire was 4 times that of developing and processing a returned Web questionnaire. The cost difference resulted primarily from the cost of return postage ($0.70) and staff processing time per paper questionnaire received ($3.68), costs that were not incurred by the Web questionnaire. Participants who responded by Web questionnaire prior to the first paper mailing contributed significantly to additional direct cost savings by eliminating the printing and mailing of their questionnaire ($0.85).

Each of the 3 follow-up cycles (2003–2004, 2005–2006, and 2007–2008) resulted in a response rate of approximately 80% of living BWHS participants. As Figure 1 shows, the proportion of women who responded on the Web doubled between 2003 and 2007. Specifically, we received 41,436 paper questionnaires (72.4%) and 4,683 Web questionnaires (8.2%) during the 2003 cycle, 36,041 (63.8%) paper responses and 7,135 (12.6%) Web responses in the 2005 cycle, and 36,203 (65.6%) paper responses and 8,925 (16.2%) Web responses in the 2007 cycle. Age-specific response rates for each questionnaire cycle are given in Table 1. In 2003, the proportion of participants who completed the paper questionnaire was highest (83.5%) in the oldest age group (≥60 years) and declined by age, with the lowest response (63.5%) being observed in persons under age 30 years. The reverse was true for the Web questionnaire, where the best response (11.5%) was from respondents under age 30 years and the poorest response (1.8%) was among respondents aged 60 years or older. Selection of the Web option approximately doubled in each age category during the 6-year period of follow-up, to 20.9% among women under age 30 years and 3.6% among women aged 60 years or older.

Figure 1.

Percentages of participants responding to Web and paper questionnaires during the 2003, 2005, and 2007 follow-up cycles, Black Women's Health Study, 2003–2008.

Table 1.

Response to Web and Paper Questionnaires During the 2003, 2005, and 2007 Follow-up Cycles, by Age Group, Black Women's Health Study, 2003–2008^a

We next compared the baseline characteristics of paper respondents, Web respondents, and nonrespondents (Table 2). Web responders had higher levels of educational attainment and neighborhood socioeconomic status than did paper responders and nonresponders. Prudent diet pattern scores were higher among Web responders. Web responders were less likely to be current smokers, to have children, or to have a chronic disease. The characteristics of Web, paper, and nonrespondents did not vary across the 3 follow-up cycles.

Table 2.

Baseline Characteristics (%^a) of Respondents and Nonrespondents in the 2007 Follow-Up Cycle, Black Women's Health Study, 2007–2008

The number of participants providing an e-mail address increased over the years: 13,398 in 2003, 15,736 in 2005, and 29,875 in 2007. Response to a Web questionnaire was considerably higher if the participant had previously provided us with an e-mail address. As Figure 2 shows, 28.9% of persons with an e-mail address completed the Web version of a follow-up questionnaire, while only 1.4% of those without an e-mail address completed the Web questionnaire. Total response by paper or Web was also higher among persons with an e-mail address: 91.1% among those with an e-mail address and 69.4% among those without one.

Figure 2.

Percentages of participants responding to Web and paper questionnaires during the 2003, 2005, and 2007 follow-up cycles, by e-mail availability, Black Women's Health Study, 2003–2008. Results are presented for all participants who either A) provided (more ...)

Web responders were about twice as likely to complete the entire questionnaire as paper responders (Table 3). In 2003, 82% of Web responders completed all questions, while only 46% of the 2003 paper responders completed all questions. This number increased in 2007 to 97% of Web responders’ completing all required responses, compared with 46% of paper responders. We assessed whether the higher completion rates of the Web responders may have been due to characteristics of those completing the Web questionnaires, such as a higher level of education. After controlling for level of education, the completion rate of Web questionnaires was still almost 2 times greater than that of paper questionnaires. In addition, among women with at least 16 years of education, Web responders were close to 2 times more likely to complete the entire questionnaire than paper responders (2003: 50.8% paper, 87.0% Web; 2005: 43.3% paper, 87.3% Web; 2007: 51.6% paper, 97.7% Web).

Table 3.

Completeness of Response (%^a) to the Paper and Web Questionnaires During the 2003, 2005, and 2007 Follow-Up Cycles, Black Women's Health Study, 2003–2008

There were many advantages to using the Web questionnaire in the BWHS (Table 4). The strongest disadvantage was that the majority of participants, in each age category, preferred completing the paper questionnaire.

Table 4.

Advantages and Disadvantages of Using the Web Questionnaire for Obtaining Follow-Up Response, Black Women's Health Study, 2003–2008

DISCUSSION

BWHS participants’ use of the Web option to complete follow-up health questionnaires doubled over a period of 6 years, from completion of the 2003 cycle through completion of the 2007 cycle, from 10.1% of responses to 19.9% of responses. Although we have not completed the latest follow-up cycle (2009–2010), we have already received over 13,000 Web responses, 4,000 more than we had received by the end of the 2007–2008 follow-up cycle. This increase over time may be due to more participants’ having and providing us with e-mail addresses, increased trust in the study over time, or increased trust in the Internet.

The greatest difference between Web responders and paper responders was age. Women who completed the Web questionnaire were younger than women who selected the paper option. Younger participants have changed addresses much more often than older participants, and inability to contact them may have contributed to a lower response to the paper questionnaire in the past (20). The Web questionnaire appeared to provide an alternative means of completing the questionnaire for these younger, more mobile participants. Most respondents in every category of age, however, still preferred to complete the questionnaire on paper. Concerns have been raised about the feasibility of using Web surveys with older populations (24). While selection of the Web option was much less frequent among older women than among the younger women in the BWHS, the fact that in every age group response to the Web survey doubled over a 6-year interval suggests that the age barrier to use of Web surveys is declining as the population ages.

Web users had higher levels of educational attainment and higher levels of neighborhood socioeconomic status than paper responders. Thus, low levels of education and neighborhood socioeconomic status, which are markers of access to the Web, remain barriers to the use of Web-based questionnaires. In view of limited Web access in some groups, there is a need to allow for multiple options for completing a questionnaire, particularly in studies with a broad range of age groups and socioeconomic levels.

We found that the Web questionnaires had fewer skipped questions than the paper versions, and this was also the case for respondents with at least 16 years of education. Other studies have found similar results (5–7). The fact that there were fewer skipped questions among BWHS Web responders than among paper responders in women with high levels of education suggests that less skipping was probably due to the type of questionnaire rather than the characteristics of the respondents. Web questionnaires have the ability to remind participants when they have skipped a question, encouraging a more thorough response. On the BWHS Web questionnaires, skipped questions triggered a “pop-up” reminder but did not require the participant to answer the missed question.

A previous study indicated that participants would be less likely to answer sensitive questions on the Web than on paper (5). We found the opposite to be true. Sensitive questions, such as those about abuse, were completed as often or more often on the Web as on the paper questionnaire.

The Web questionnaire has resulted in appreciable cost savings for the BWHS. The production and mailing costs and staff processing times have been reduced. Approximately two-thirds of Web responders in the 2007 follow-up cycle completed the questionnaire early in the follow-up cycle, before the paper questionnaire was mailed, which also saved printing, mailing, and processing costs. However, the savings were limited by the relatively small proportion of participants who preferred the Web questionnaire over the paper questionnaire. With a continued increase in Web response, the cost savings should continue to increase.

In summary, the use of a Web questionnaire in the BWHS resulted in cost savings and helped to maintain a high response rate through each follow-up cycle. The Web questionnaires were more complete than paper questionnaires, regardless of the sensitivity of a question. Although there are advantages to using a Web questionnaire, the use of multiple means of soliciting questionnaire response is still needed in view of limited Web access or Web use in some populations.

Acknowledgments

Author affiliation: Slone Epidemiology Center at Boston University, Boston, Massachusetts (Cordelia W. Russell, Deborah A. Boggs, Julie R. Palmer, Lynn Rosenberg).

This work was supported by the National Cancer Institute (grant R01 CA058420).

The authors acknowledge the dedication of the Black Women's Health Study staff.

The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the National Cancer Institute or the National Institutes of Health.

Conflict of interest: none declared.

Glossary

Abbreviation


BWHS	Black Women's Health Study

References

1. Internet World Stats. Internet Usage Statistics for the Americas [database] Bogota, Colombia: Miniwatts Marketing Group; 2009. ( http://www.internetworldstats.com/stats2.htm). (Accessed March 9, 2010)

2. Pew Internet and American Life Project. Pew Internet Project Data Memo. Washington, DC: Pew Research Center; 2009. ( http://www.pewinternet.org/~/media/Files/Reports/2009/PIP_Generations_2009.pdf). (Accessed September 15, 2010)

3. Kroth PJ, McPherson L, Leverence R, et al. Combining Web-based and mail surveys improves response rates: a PBRN study from PRIME Net. Ann Fam Med. 2009;7(3):245–248. [PubMed]

4. Beebe TJ, Locke GR, III, Barnes SA, et al. Mixing web and mail methods in a survey of physicians. Health Serv Res. 2007;42(3):1219–1234. [PMC free article] [PubMed]

5. Smith B, Smith TC, Gray GC, et al. When epidemiology meets the Internet: Web-based surveys in the Millennium Cohort Study. Am J Epidemiol. 2007;166(11):1345–1354. [PubMed]

6. Kongsved SM, Basnov M, Holm-Christensen K, et al. Response rate and completeness of questionnaires: a randomized study of Internet versus paper-and-pencil versions. J Med Internet Res. 2007;9(3):e25. (doi: 10.2196/jmir.9.3.e25) [PMC free article] [PubMed]

7. Bälter KA, Bälter O, Fondell E, et al. Web-based and mailed questionnaires: a comparison of response rates and compliance. Epidemiology. 2005;16(4):577–579. [PubMed]

8. Ekman A, Dickman PW, Klint A, et al. Feasibility of using web-based questionnaires in large population-based epidemiological studies. Eur J Epidemiol. 2006;21(2):103–111. [PubMed]

9. Ritter P, Lorig K, Laurent D, et al. Internet versus mailed questionnaires: a randomized comparison. J Med Internet Res. 2004;6(3):e29. (doi: 10.2196/jmir.6.3.e29) [PMC free article] [PubMed]

10. Holmes S. Methodological and ethical considerations in designing an Internet study of quality of life: a discussion paper. Int J Nurs Stud. 2009;46(3):394–405. [PubMed]

11. Heiervang E, Goodman R. Advantages and limitations of web-based surveys: evidence from a child mental health survey. Soc Psychiatry Psychiatr Epidemiol. 2009 Nov 18 [Epub ahead of print]. (doi: 10.1007/s00127-009-0171-9) [PubMed]

12. Dannetun E, Tegnell A, Giesecke J. Parents’ attitudes towards hepatitis B vaccination for their children. A survey comparing paper and web questionnaires, Sweden 2005. BMC Public Health. 2007;7:86. (doi: 10.1186/1471-2458-7-86) [PMC free article] [PubMed]

13. Basnov M, Kongsved SM, Bech P, et al. Reliability of Short Form-36 in an Internet- and a pen-and-paper version. Inform Health Soc Care. 2009;34(1):53–58. [PubMed]

14. Truell AD, Bartlett JE, II, Alexander MW. Response rate, speed, and completeness: a comparison of Internet-based and mail surveys. Behav Res Methods Instrum Comput. 2002;34(1):46–49. [PubMed]

15. Touvier M, Méjean C, Kesse-Guyot E, et al. Comparison between web-based and paper versions of a self-administered anthropometric questionnaire. Eur J Epidemiol. 2010;25(5):287–296. [PubMed]

16. Leece P, Bhandari M, Sprague S, et al. Internet versus mailed questionnaires: a controlled comparison (2) J Med Internet Res. 2004;6(4):e39. (doi: 10.2196/jmir.6.4.e39) [PMC free article] [PubMed]

17. Dillman DA. Mail and Internet Surveys: The Tailored Design Method. New York, NY: John Wiley and Sons, Inc; 2000.

18. Guise V, Chambers M, Välimäki M, et al. A mixed-mode approach to data collection: combining web and paper questionnaires to examine nurses’ attitudes to mental illness. J Adv Nurs. 2010;66(7):1623–1632. [PubMed]

19. Rosenberg L, Palmer JR, Rao RS, et al. Risk factors for coronary heart disease in African American women. Am J Epidemiol. 1999;150(9):904–909. [PubMed]

20. Russell C, Palmer JR, Adams-Campbell LL, et al. Follow-up of a large cohort of Black women. Am J Epidemiol. 2001;154(9):845–853. [PubMed]

21. Kumanyika SK, Mauger D, Mitchell DC, et al. Relative validity of food frequency questionnaire nutrient estimates in the Black Women's Health Study. Ann Epidemiol. 2003;13(2):111–118. [PubMed]

22. Krishnan S, Cozier YC, Rosenberg L, et al. Socioeconomic status and incidence of type 2 diabetes: results from the Black Women's Health Study. Am J Epidemiol. 2010;171(5):564–570. [PubMed]

23. Agurs-Collins T, Rosenberg L, Makambi K, et al. Dietary patterns and breast cancer risk in women participating in the Black Women's Health Study. Am J Clin Nutr. 2009;90(3):621–628. [PubMed]

24. Klovning A, Sandvik H, Hunskaar S. Web-based survey attracted age-biased sample with more severe illness than paper-based survey. J Clin Epidemiol. 2009;62(10):1068–1074. [PubMed]

Articles from American Journal of Epidemiology are provided here courtesy of
Oxford University Press