This analysis identified factors that predict retention in an online, prospective follow-up study of MSM. Overall, only 21% (539/2607) of participants who were eligible to participate in the follow-up survey agreed to participate and returned at the 3-month follow-up time point. Hispanic men were less likely to agree to participate in the follow-up survey than white men, but participants reporting high-risk sexual behavior (ie, UAI in the past 12 months) were more likely to agree to follow-up. The latter result perhaps indicates that participants who may benefit the most from an online intervention (ie, are at high-risk for HIV) are willing to participate in longitudinal research. Although offering an incentive did not increase the odds of agreeing to participate in the follow-survey, participants who were offered an incentive were significantly more likely to link into the follow-up survey at the 3-month follow-up time point. Additionally, older men and participants with a higher level of education had increased odds of linking into the follow-up survey, but black men were significantly less likely to link into the follow-up survey compared with white men. Participants overwhelmingly agreed to complete the follow-up survey, with over 90% of respondents providing an email address to be contacted at the 3-month follow-up time point. However, at follow-up only a small proportion (22%) of respondents providing their email address actually returned to complete the survey, indicating that intention to participate in the follow-up survey may not necessarily predict actual participation.
We asked participants who provided their email address to answer a series of questions regarding the type, use, and frequency of use of the email address they were providing. This allowed us, for the first time to our knowledge, to analyze the characteristics of the email addresses provided by participants as possible factors predicting retention in an online study. We found that email addresses provided by participants that were used for online financial management or were addresses that participants checked at least once a day were associated with linking into the survey at the follow-up time point. These data suggest that participants who provide these types of email addresses (ie, an email address that may be important to the participant) are more likely to be retained in the study.
While the overall rate of retention in this study was quite poor, this result is not especially uncommon in online follow-up studies. Retention rates in online longitudinal research have typically been lower than in conventional offline follow-up studies [27
], and the difficulty in retaining study participants in online research spans studies from smoking cessation to diabetes to depression [27
]. In fact, the high number of participants lost to follow-up in Internet research has prompted researchers to title this common phenomenon the law of attrition
, described as the fundamental methodological challenge in longitudinal online research [31
]. Although the difficulty in retaining online study participants is widely recognized, our study furthers knowledge by providing an analysis to compare participants who were retained in the study and those who were lost to follow-up.
This analysis suggests that online studies of MSM similar to this one may experience differential attrition of black and Hispanic men, leading to a discrepant underrepresentation of these groups in the final outcome analysis. There are a number of possible reasons for the decreased retention of racial/ethnic minorities in this and similar studies. First, black and Hispanic Americans have less access to private high speed Internet than white Americans. Data from the US Census 2009 Current Population Survey indicated that 46% of black households and 47% of Hispanic households do not have Internet access in their homes, compared with 27% of white households [32
]. Among households that have access to Internet in the home, 68% of white households have high speed Internet access compared with only 50% and 48% of black and Hispanic households, respectively [32
]. This lack of access to home high speed Internet has resulted in black and Hispanic Americans using the Internet in public locations, such as libraries [33
]. If black and Hispanic participants in our study did not have home access to the Internet, it is possible that they may not have seen the email reminders sent with a link to the follow-up study if they were unable to access the Internet during the time in which the reminder emails were sent or checked their email infrequently during that time. However, black and Hispanic participants did not indicate that they checked their email address less frequently than white participants; therefore, it is also possible that black and Hispanic participants were accessing their email at their place of work or in a public venue where they were not comfortable linking into the follow-up survey, given their knowledge of the sensitive nature of the baseline survey questions. This is difficult to discern, however, since participants were not asked for the location at which they most frequently accessed their email account.
While it has been noted that the historic distrust by racial/ethnic minorities of research studies may extend to the realm of online research [34
], it is unclear if it is a contributing factor in this study. We speculate that because Hispanic participants were less likely to provide an email address for the follow-up survey, they may not have felt comfortable disclosing their email address or did not trust that their confidentiality would be maintained. Given that after having just completed a personal and sensitive online survey black participants were equally as likely to provide their email address compared with white men, it is it is unlikely that black men did not link into the follow-up survey because of a lack of trust.
This study has a number of limitations. Participants in the study are not representative of MSM who use MySpace.com or MSM in the United States, and we have previously characterized the potential selection bias among the same group of study participants [28
]. Because this study relied on self-reported characteristics of participants, we cannot be sure that the reported information from participants such as race or age was correct, and, thus, misclassification bias may have resulted. However, because advertisements were targeted to MySpace user profiles based on selected demographic profile information, respondents would have had to consistently provide incorrect information both in their profile and in the online survey to participate.
Upon completion of the study, technological faults in the survey were discovered. In one question of the baseline survey, if a participant pushed the Enter key instead of clicking on the Next button on the survey screen, they were mistakenly taken to the end of the survey and were not permitted to go back to the previous screen. Therefore, these men, who may have been eligible for the follow-up survey, were never asked if they were willing to participate in the follow-up survey. As previously noted, 21 men were offered incentives at multiple levels and were not included in this analysis. While this represented a small proportion of the sample, it is still a technological error that needs to be addressed in future studies. Additionally, there was a technological difficulty in the emails sent to participants with the link to the follow-up survey. These emails were generated from the non-Emory server (SurveyGizmo) hosting the survey, but the return email address was from the emory.edu domain. Because many email account spam filters filter emails that are generated from a server that is contradictory to the email extension of the sending email address, it is possible that a large portion of the reminder emails were either not delivered to participants or were delivered to a quarantine or spam box, where they were less likely to be recovered by participants. This may have contributed to our low overall retention. However, there is no mechanism with which to track the number of emails that were filtered away from participants’ in-boxes, so the magnitude of the effect of this error on survey retention remains unclear. In future research, requesting participants to add the sending email address to their list of acceptable senders may increase the likelihood of participants receiving study-related emails.
Although this analysis has a number of limitations, the results from this study may provide guidance in the development of strategies to increase retention in online HIV prevention research with MSM. First, we identified that, consistent with similar online studies of MSM, black and Hispanic men are less likely to be retained in the research study. Since a possible explanation for the decreased retention in these groups is access to high speed Internet, participants should be asked about their access to high speed Internet in follow-up studies to further understand this issue. However, we must also explore new avenues to contact these participants at follow-up time points. Recent research suggests that black and Hispanic Americans have higher mobile phone ownership than white Americans, and that 79% of black and 83% of English-speaking Hispanic adults report sending and receiving text messaging on a typical day, compared with 68% of white adults [35
]. Therefore, the use of mobile phones, specifically short messaging system (SMS or text messaging), may be an alternative to the Internet for data collection in future studies, since its accessibility for black and Hispanic MSM may be helpful in increasing the retention of these study participants.
Second, we found that offering an incentive increased the likelihood of participants being retained in the study. Although offering a US $20 incentive was significantly associated with linking into the follow-up survey while US $5 and US $10 were not, we are cautious to conclude that US $20 is the minimum amount to offer participants in order to increase retention. It has been suggested that offering incentives may lead to participants enrolling in studies multiple times in order to claim more than one incentive. This study included two elements to ensure that this would not occur. First, men could only access the survey by clicking on the banner advertisement posted on their profile. Because the advertisements were displayed to random users at varying times of day, it is unlikely that a participant would have seen an advertisement twice. Second, multiple surveys could not be completed from the same Internet protocol (IP) address, so a participant would have had to change his IP address in order to take a second survey. Therefore we believe that we were successful in paying incentives only once to participants and that it is feasible to offer incentives in future online research. Finally, the analysis of email characteristics of participants provides guidance for ways in which to solicit participant email addresses. Future studies may see increased retention by prompting participants to provide an email address that they check at least once per day or to provide an email address that is important to them (ie, one they use for financial management). This may increase the likelihood of participants viewing an email request to complete follow-up surveys, for example. Additionally, although not included as part of this analysis, it may be of interest in future studies to examine whether economic considerations other than providing an incentive (ie, paying money for sex) differentiates groups that participate in online longitudinal research. Further, examining the effect of geographic region (eg, rural versus urban or western versus eastern United States) on participation in longitudinal Internet-based studies may be informative.
As the HIV epidemic among MSM in the United States persists and MSM continue to meet sex partners online, it is essential to create effective methods for the proper creation and testing of online HIV prevention interventions. This study identified factors that may predict enrollment or retention in online research so that mitigation strategies to decrease participant attrition can be developed. Only by creating ways in which to retain participants in online research can critical outcome measures of interventions be appropriately assessed.