In our studies we experienced very low participation rates, despite the provision of monetary incentives. Since potential participants in the first study were members of our research website, CyberIsle, we did not expect such a low participation rate (0.2%) to our email invitation. This figure is close to the lower bound of response rates from email marketing of 0.1% [29
] rather than the average rate of 1%. Several possible explanations for the low participation are discussed below followed by a description of some of the challenges of Internet recruitment.
Authenticity and Legitimacy of Information on the Internet
With the large number of websites youth encounter, it is plausible that the email recipients did not remember their previous involvement with CyberIsle. They may have considered the recruitment message as unsolicited commercial mass email (spam). Our initial recruitment email, which had a response rate of 0%, was formatted in HTML with colors and embedded images (). The graphical layout along with several hyperlinks might have been mistaken for spam by the recipients or by the built-in spam filter in email programs resulting in automatic deletion from the incoming mailbox. However, the response was still low even when the reminder email was formatted as plain text (). The recipients may not have received our second email because of spam filters or because they did not regularly check the email account of the address they provided during the CyberIsle registration (youth often set up separate email accounts used specifically for registration purposes).
Given the low response rate from the first study, where the potential participants were members of our health website, it is not surprising to see a similar low response rate when we extended the recruitment to the general Internet community where there had been no previous connection with our research project.
The level of spam and deceptive email on the Internet has exploded exponentially in the past few years [30
]. The spam to non-spam ratio as of March 2004 was estimated to be 63%. About 12% of spam was estimated to be scams or fraud and many were infected with viruses or worms [31
] that pose a serious threat to online privacy. Since online privacy is one of the major concerns for youth online, it is not surprising that postings or email messages that bear even slight resemblance to spam are ignored.
The context of a message may influence the decision of potential participants to join a study. We expected that postings on University of Toronto-related Usenet forums and discussion boards would enhance the credibility and relevancy of our study. However, only 3 participants were from the University of Toronto discussion boards. It is possible that more individuals would have participated in the study if we had kept the postings online for a longer period of time. However, older messages on discussion boards are rarely browsed once they are not shown on the first page (pushed to later pages by newer postings).
The incentives level might not have been sufficient or gift certificates for an online bookstore may not have been attractive enough for our young potential participants. One of the limitations of electronic gift certificates is that the price of purchase must be lower than the value of the gift certificate. Otherwise, one would need to have access to a credit card in order to purchase online, which is an issue for trials with teenagers. After allowing for taxes and shipping charges, a Can $20-dollar certificate is worth only about Can $14 thereby limiting what can be bought. Despite this limitation with electronic gift certificates, it was chosen as the incentive in the study because of the anonymity it provided. Only a valid email address is required to deliver a certificate to a participant, as opposed to requiring the postal address if other coupons usable in stores are used as incentives. Until electronic cash payments such as PayPal become widely accepted, there are limited options for compensating respondents for their participation in an anonymous way.
Snowball Sampling and Personalization
The explosion of spam on the Internet may explain why our snowball recruitment through email referrals was ineffective. Despite the potential of receiving an additional Can $10 of gift certificate, 4 of the 14 participants in the second study did not provide their friends' email addresses. This is not surprising since they might have wished to preserve their friends' privacy.
Recruitment emails sent to the referrals in both studies were addressed from our study email account. In the body of the email, we indicated that how and from whom (email address of the referrer provided) we had obtained the referrals' email addresses. In the same email, we also sent a copy to the referrer as a way of indicating the legitimacy of the email. Additional personalization to the email was not introduced since we had only the email addresses of the referrals.
A recent study on online shoppers found that compared to basic site improvements such as ease of navigation, the effect of personalization provided little incentive for users to buy from an e-commerce website [32
]. This is in contrast to the general recommendation given to improve response rates in mail surveys [33
]. Again, the weak effect of personalization in email could be the result of widespread personalization in most electronic marketing materials encountered on a daily basis. Better response might be achieved if recruitment emails were sent directly under the referrers' email addresses rather than from the study email address. Instead of sending the referral's email to the study coordinator, the study website could be programmed so that the referrers could send invitation emails using their own email addresses directly to their friends. Spammers have exploited various deceptive techniques such as employing fake sender email addresses from legitimate domains, embedding real logos from legitimate websites onto messages, and using misleading or enticing (such as money or free prizes) subject lines. Therefore it is almost impossible to create a recruitment email message or a Web posting that can easily be distinguished from spam by a casual Internet user. For email to be a viable recruitment medium, more research is needed to explore the factors contributing to a trustable message.
Challenges and Practical Advice
Verification of Participants' Attributes
Because of the anonymous nature of our study design, it was impossible to verify the age of the participants. The eligible age range for our second study was 15 to 24 years. There is no simple online solution for verifying an Internet user's true age. A 2001 study on youth Internet behavior found that 15% of online teens and 25% of older boys when online have lied about their age to gain access to websites which often are pornographic in nature [28
]. On the other hand, in studies where adult participants are required, it is possible to use commercial online age verification services using credit card information as the verifying identifier. However, privacy issues will become a major concern as individually identifiable information is collected by age verification companies.
The information page of our study specified that enrolment was limited to individuals currently living in Canada. There is no simple way to check or enforce the geographical location of a participant, although it is possible, with various free reverse lookup Internet websites, to identify the country of a participant's computer using the Internet Protocol (IP) address. However, it is both difficult and costly to implement this as a real-time check feature on the site. The solution we adopted in this study was to target our postings only to Canadian discussion boards and Usenet forums.
Preventing Multiple Participation
Preventing multiple entries from the same participant is another challenge for Internet recruitment, particularly in studies with monetary incentives [5
One step which can identify multiple participation is the examination of the survey results submitted from same IP addresses for the presence of other indications for multiple participation, such as the lack of internal consistency between items in the survey, and unrealistically short response time to survey questions [5
Simply deleting all entries with duplicate IP addresses is not recommended, because the recent popularity of proxy servers or network address translation (NAT) servers, have made it not uncommon for one public IP address to be shared across many computers within a private local area network [35
]. In addition, for computers connecting to the Internet through dynamic IP addresses (dial-up or broadband), new IP address can be obtained simply by logging in again. Thus, duplicate IP addresses do not necessarily indicate multiple entries from the same person and to delete all such entries would eliminate legitimate data.
] estimates that repeat participations were below 3% in most studies and should not be a threat to the data quality of Internet-based research.
Participation in Internet recruitment may be increased by broadening the dissemination of the recruitment information. For example, one can post the study invitation on those discussion boards or Usenet forums that have higher posting traffic, such as those related to computers. However, it is bad “netiquette” to cross-post in forums with out-of-context messages, such as study recruitment of a health behavior study in a computer-related forum. Such messages will either be ignored or removed. In some cases, the sender will be “flamed” (responded to by overly harsh and often hostile terms). Another possibility is to purchase advertisement space such as in the form of page banner on websites that are popular among target users.
This study is one of the first attempts to investigate the feasibility of Internet recruitment in the “age of spam.” In our specific case our recruitment strategies were not efficient. However, we caution against generalizing our negative results. Internet recruitment may prove viable if studies are conducted on a larger scale, if the right newsgroups are targeted, the right incentives chosen, and the right wording is used. Recruitment announcements in the form of Web page banners can potentially be viewed by tens of thousands, if not more, of online users on high traffic Web portals.
From the researchers' perspective, the validity of study results can be compromised by limitations in verifying participants' attributes such as age. For motivated participants, it is not clear how to differentiate trustable and legitimate messages on the Internet. Researchers using Internet recruitment in their studies should focus on ways to improve the perceived legitimacy of the invitation message. For example, participants should be able to easily identify the study website as belonging to a legitimate organization such as a university.
Success in recruiting participants online depends on many factors, which are similar to those for getting responses in traditional mail and telephone surveys. Studies have investigated various strategies to maximize response rates in offline surveys [36
]. It is clear that there is no single strategy that can guarantee good response rates in all situations, due to variations in study characteristics, target populations, type and amount of incentives, sponsorships, length of questionnaires, text used for recruitment, and follow-up strategies. Future studies on Internet recruitment should focus on investigating ways to convey trust online to Internet users and to find attractive incentive structures for Internet users.