Notwithstanding our efforts to describe general trends based on survey results, perhaps the most striking finding to emerge from our analyses was the considerable variation among studies. The mean sample size (collapsing across study conditions for multiple-group designs) was 847.2 (SD = 1,721.7), ranging from 20–15,000. Likewise, the average number of data collection waves was 5.8 (SD = 5.7), ranging from 1–43. Although the survey did not capture time periods between all follow-up points, respondents were asked to provide the time period between the baseline assessment and the first follow-up contact. The average time to first follow-up was 9.2 months (SD = 16.4), ranging from one week to 10 years. When participants were asked to provide an approximate percentage of the subjects that they were able to contact for the first follow-up point (excluding deceased subjects from the base, but including cases in which subjects were contacted but refused to participate), the average follow-up rate was 84% (SD = 11.9), ranging from 36%–100%. The median follow-up rate was 85%.
To learn more about the issues confronting researchers in their efforts to achieve high follow-up rates, respondents were asked to indicate (1) the types of activities for which subjects were paid, (2) how much they paid for each of these activities, and (3) how difficult they found various aspects of subject tracking and locating to be. In addition, data from this survey allowed us to conduct some simple tests of association between study characteristics and follow-up rates.
As shown in , the most common research activity required of subjects in the projects surveyed was a face-to-face interview (80%), followed by telephone interviews (36%), and provision of biological samples, such as blood, urine, or hair (31%). The least commonly reported data collection method was a web-based survey, which was reported by only 4% of the investigators. Based on the subject compensation data, the two most commonly used data collection approaches also brought the highest incentives, with face-to-face interviews being compensated at $44.00, and telephone interviews at approximately $33.00. Somewhat surprisingly, the typical compensation for providing biological samples ($17.20) was substantially lower than that of providing self-report data. Respondents were also asked to provide the full expected compensation for a subject who participated in all study activities for the entirety of the study. The average total possible compensation was $328.70 (SD = 600.7), ranging from $10.00–$4,800.00. To estimate the typical subject payment per data collection session, the total possible compensation was divided by the number of data collection waves for each study.3
This resulted in an estimate of approximately $60.00 (SD = 68.8) per data collection session.
Types of Research Activities for Which Subjects Were Paid, and Amount of Compensation (N=153)
To assess the perceived difficulty of conducting longitudinal research, investigators were asked to rate the level of difficulty of six aspects of tracking and locating substance abusers. Response options for these items ranged from 1, “Very Easy,” to 6, “Very Difficult.” For purposes of the present study, responses were collapsed to indicate percentages of respondents reporting these study elements to be “Difficult” or “Very difficult.” Responses to these six survey items are shown in . By a substantial margin, “tracking and locating” was the most commonly cited difficulty associated with conducting longitudinal research with substance abusers (56%). The element with the second highest difficulty rating was “maintaining current locator information over time” (41%). Fewer problems were associated with issuing subject payments or confirming receipt of payments, with only 5% of respondents rating these as difficult or very difficult. A somewhat encouraging finding was the relatively high participation rate of subjects whom the researchers were able to contact, with 86% of investigators indicating that obtaining data from contacted subjects was not a serious problem. As a general measure of the costs associated with achieving a representative follow-up rate, investigators were asked, “Considering your study overall, how much of a financial burden (including staff costs, etc.) was tracking and locating research subjects in order to collect follow-up data?” Response options and responses were as follows: “Little or No Cost” (2.7%), “Moderately Costly” (48.7%), “Very Costly” (33.3%), and “A Significant Portion of the Budget” (14.7%), with 48% of the respondents endorsing one of the latter two categories.
Perceived Difficulty of Various Aspects of Conducting Follow-Up Research with Substance Abusers (N=153)
To examine possible associations between various study characteristics and follow-up rates—particularly the achievement of the conventional standard of 80%, it was hypothesized that three study elements: sample size, time to first follow-up contact, and compensation amount would be correlated with the follow-up rates in the studies represented in the survey. Specifically, it was hypothesized that larger sample sizes and longer time periods before the first follow-up would be associated with lower follow-up rates, whereas compensation amount would be positively related. These hypotheses found limited support in our data. The Pearson correlation coefficients between follow-up rates and the first two characteristics: sample size (r =−.01, p > .10) and time lapse to follow-up (r = .05, p > .10) failed to show any meaningful association.
If investigators set the values of their subject payments, in part, to incentivize participation, it is important to know the expected returns to subject compensation (i.e., do increases in subject compensation yield a significant response?). To estimate subject responsiveness the compensation elasticity of response was calculated. A high elasticity value would indicate that subjects are responsive to levels of compensation (i.e., that compensation is a useful motivator to improve followup), low values would indicate that subjects are relatively inelastic (unresponsive) to compensation (i.e., that changing the amount of compensation would have little effect on response rates).
To estimate the elasticity the dependent variable (followup percentage) was logit transformed and regressed against compensation, controlling for number of waves, sample size, and time to followup. Elasticity was then calculated as ε = (1 −
is the average followup rate in the sample,
is the average compensation in the sample, and β is the coefficient on compensation. The estimated elasticity was of the predicted sign (followup percentage is positively related to compensation), but subjects were inelastic to the amount of compensation offered. The estimated elasticity was low (ε=0.02), suggesting that doubling the amount of compensation offered from the mean would only increase followup rates by 2%.
An additional analysis comparing mean compensation amounts between studies achieving less than an 80% follow-up rate versus those achieving follow-up rates of 80% or higher revealed significantly higher payment levels for the high follow-up-rate studies (Means = $207.6 (SD = 223.7) versus $376.2 (SD = 690.2), respectively; t (df = 137) = −2.2, p < .03).
Lastly, to assess the impact of low follow-up rates on the generalizability of study findings, investigators were asked whether they compared baseline characteristics of subjects who did and did not participate in follow-up data collection. Nineteen percent of the respondents reported not making these comparisons. Among the 81% who did, more than one third (34.4%) reporting finding meaningful differences between those who did and did not participate in the follow-up.