Sample of Interventions and Controls
We included 194 reports, which provided 354 independent intervention groups and 99 independent control groups. Of the 194 reports, 44 provided a single data set, 91 provided two data sets, 28 provided three data sets, 21 provided four data sets, 3 provided five data sets, 6 provided six data sets, and 1 provided eight data sets. summarizes information about the included reports, as well as their types of interventions, participants, and methods, with separate columns for intervention and control groups. As can be seen from the table, most studies were published around 1996 and the median sample sizes of participant groups was around 100. Most reports were affiliated with the medical sciences, with psychology as the second most frequent affiliation. Although most studies were conducted in the United States, 33 countries were represented. Of the U.S. studies, 33 states were represented, with California providing more groups than any other state.
With respect to intervention strategies, 48% of the interventions contained arguments designed to induce a positive attitude about condom use outcomes, 15% contained normative arguments in support of condom use, 94% contained HIV-relevant information, 20% included arguments designed to verbally promote recipients’ behavioral skills, 47% included persuasive arguments designed to increase perceptions of threat among recipients, an average of 22% trained participants in some type of behavioral skill, and 18% administered an HIV test. Given the different combination of strategies, 51% of groups were exposed to interventions that simply presented arguments (passive interventions), whereas the remaining 49% engaged in activities to promote condom use (active interventions, i.e., HIV counseling and testing or behavioral skills training). Researchers distributed condoms to 22% of the intervention groups and to 7% of the controls.
There was great methodological variability in the studies we examined, in terms of the participants, intervention setup, and research design and implementation. Samples comprised both female and males, and participants were relatively young in age. On average, only 36% of participants were of European descent and only 35% of participants had completed high school. The samples included men who have sex with men, intravenous drug users, partners of intravenous drug users, commercial sex workers, multiple-partner heterosexuals, participants with a history of STIs, and patients with severe mental illness. Some samples included drug rehabilitation patients and general drug users; many included college, middle-school, or high school students; and a small percentage sampled teachers. Most participants for whom a measure of condom use was obtained had low condom use, and only a small percentage of participants were using condoms consistently. The average rate of infection with HIV was 20%, although most studies had no information on this issue.
More communications were presented in school and clinical settings than in any other place, although many of the messages were delivered in community settings, and some through mass-communication media. The communications were generally presented face-to-face, and video- and audiotaped materials were included in many cases. The intervention was applied exclusively to individuals (as opposed to group) in only 20% of the cases and lasted an average of 7.94 hr.
Finally, there was considerable variability in research design and implementation across the studies. For example, although all studies included pre- and posttest measures, some used different samples, whereas the majority were done within subject. The allocation of participants to study groups was done at random in 46% of the cases, and intervention participants were compensated an average of U.S. $18.31. The mean length of time between the intervention and the posttest was slightly over 3 months, although the median was about 1 month. Half of the intervention groups in our sample were explicitly based on theory, and 33% were designed from formative research with the target population. Most of the studies targeted a specific population. Quite frequently, samples were self-selected; attrition was around 12% across intervention and control groups.
Effects of Intervention Strategies Across Participant Populations and Intervention Setups
Given the need to assess intervention outcomes across different populations and intervention setups, we entered dummy-coded variables describing the nature of the groups under study in an analysis of the effectiveness of the different types of strategies. These analyses appear in – and were conducted on passive and active interventions considered simultaneously.
Influence of Participants’ Characteristics and Intervention Setup on Behavior Change
Change as a Function of Strategy Across Intervention Setups
To analyze the generalizability of different interventions across populations, we performed analyses with gender, age, and ethnicity; the inclusion of men who have sex with men, intravenous drug users, partners of intravenous drug users, and multiple-partner heterosexuals; and past condom use. (Other groups in were not sufficiently represented to perform these analyses.) The analyses with gender, age, and ethnicity were replicated using continuous variables in addition to the breakdowns presented here: gender = predominantly male when more than 50% of the sample was male; ethnicity = predominantly European background when more than 50% of the sample had that background; age = under 21 years when the mean or median age was under 21. The analyses using dichotomous and continuous predictors were very similar, which led to presenting the ones with dichotomous predictors for interpretational purposes. The analyses with past condom use required collapsing moderate and high condom use owing to the low number of conditions with high condom use (see ).
To estimate the effects of the setup of the intervention, we first considered whether the intervention was presented in a school, a clinic, or a community setting. Most of the interventions in our meta-analysis were delivered face-to-face (see ), which made it impossible to analyze interactions between face-to-face presentation and type of strategy. However, we considered the inclusion of video- or audiotapes, which may increase the impact of certain strategies but can also detract from the interaction with real-life facilitators, as well as the use of group or individual formats for the intervention sessions.
presents the QB statistics for the main effects of the population and intervention factors. It also includes the control means for different populations to permit comparison with the mean change in different intervention groups when applied to the same population. and present the QBs for the interaction between a given population or setting variable and a specific argument or behavioral strategy, as well as the QB for the simple effects of a strategy in a particular group. In the following sections, we summarize the significant interactions and highlight simple effects only when the statistical interaction was significant. When the interaction was not significant, one should rely on the main effects reported in and to reach conclusions.
Change as a Function of Strategy Across Different Participants
Independent influence of population participant characteristics and intervention setup
Not surprisingly, population and intervention factors influenced the amount of behavior change in the studies we summarized. Male, older, and minority recipients showed greater increases in condom use than female, younger, and majority recipients. Whereas groups including men who have sex with men changed more than groups not including them, the inclusion of partners of intravenous drug users and multiple-partner heterosexuals was associated with less behavior change. The inclusion of intravenous drug users and initial condom use had no significant main effects on the amount of behavior change observed.
It is important to note that even when different groups had different rates of behavior change overall, as shown in the first two sections of , the means for interventions were greater than control means in most cases. The only exception was the mean change for intervention recipients under 21, which did not differ significantly from the change in control condition. (As shall be seen from further analyses reported below, however, our meta-analysis later identified effective interventions for people under 21.)
With respect to the intervention setup, we examined the effects of presenting the intervention in a school, a clinic, or the community, as well as playing video- or audiotaped materials and performing group sessions. Of all these, only playing video- or audiotaped materials had a significant main effect on behavior change. Specifically, the use of these materials was associated with decreased behavior change.
Analysis of interactions between intervention strategies and characteristics of the populations
As suggested by the statistics in , there were manysignificant interactions. For example, an examination of the first panel, which is relevant to gender effects, reveals that the negative effect of presenting threat-inducing arguments and interpersonal skills training was stronger for predominantly male groups, whereas the negative effect of presenting normative arguments was stronger for predominantly female groups. In addition, the presentation of behavioral skills arguments as well as condom use skills training had positive effects among males but null or negative effects among females, whereas attitudinal arguments, information, self-management skills training, and HIV counseling and testing exerted more positive impact among females than among males. Actually, attitudinal arguments and information had nonsignificant effects among males.
Age also moderated which strategies were successful, with greater age generally amplifying effects that were observed across the board. Groups over 21 years of age responded more negatively to normative appeals and threat-inducing arguments than did groups under 21, which were positively affected by normative arguments and unaffected by threat-inducing arguments. At the same time, groups over 21 showed significant positive effects of behavioral skills arguments, self-management skills training, and HIV counseling and testing, whereas groups under 21 showed a nonsignificant effect of behavioral skills arguments, a positive but weaker effect of self-management skills training, and a significant negative effect of HIV counseling and testing. In addition, the provision of condoms had a positive effect for audiences under 21 but a negative effect for audiences over 21.
The ethnicity findings also suggested various ways in which the background of the sample moderated the effectiveness of the different intervention strategies. Samples of predominantly European backgrounds were less negatively affected by normative and threat-inducing arguments than those with a predominantly African background. In addition, as shown by the simple effects in , whereas condom provision benefited only samples with predominantly European backgrounds, behavioral skills arguments and HIV counseling and testing benefited only samples with predominantly African backgrounds. Finally, interpersonal skills training had stronger negative effects when the predominant background was European, and self-management skills training had stronger positive effects when the predominant background was African.
The middle set of panels of summarizes the outcomes of different strategies for different HIV risk groups, including men who have sex with men, intravenous drug users, partners of intravenous drug users, and multiple-partner heterosexuals. One notable finding that appears to characterize all these groups is that compared with lower risk populations, most strategies had weaker effects for these high-risk populations. For instance, groups explicitly including men who have sex with men, intravenous drug users, partners of intravenous drug users, and multiple-partner heterosexuals generally showed weaker negative effects of normative arguments (three out of four interactions were statistically significant), threat-inducing arguments (three out of four interactions were statistically significant), and interpersonal skills training (two out of three available interactions were statistically significant). Actually, interpersonal skills training had a significant positive effect when the condition included partners of intravenous drug users. The positive effects of attitudinal arguments, self-management skills training, and HIV counseling and testing were also weaker in these high-risk groups, with the exception of men who have sex with men. Further, attitudinal arguments were less effective when the samples included men who have sex with men but more effective when the samples included partners of intravenous drug users, and information was more effective when intravenous drug users were excluded rather than included. Of importance, the only strategy consistently associated with more positive effects when conditions included high-risk participants was the provision of condoms as part of the intervention.
The last section of presents the effects of each strategy on change in condom use as a function of the level of past condom use. As suggested by most of the analyses of risk factors, low condom use as a risk factor moderated the impact of some of the strategies (see ). Although consistent with Prochaska et al.’s (1992)
predictions, the beneficial effects of self-management skills training were smaller among higher condom users than among low users; contrary to their predictions, the influence of attitudinal arguments and information did not vary significantly as a function of condom use. In addition, there were significant negative effects of interpersonal skills and condom provision when condom use was either moderate or high.
Analysis of interactions between intervention strategies and intervention setups
We were also interested in evaluating potential interactions between the strategies used in an intervention and characteristics of the intervention setup. The relevant fixed-effects analyses are summarized in , organized by (a) setting (clinical, school, or community), (b) use of audiovisual media, and (c) presentation to groups (vs. individuals). Again, apparent differences in simple effects were interpreted only when accompanied by a significant interaction.
As can be seen from the first three panels, all intervention strategies but condom use skills training had stronger effects in clinical than other settings. The stronger effects included lesser change in response to normative arguments, threat-inducing arguments, and interpersonal skills training, as well as greater change in response to information, behavioral skills arguments, condom provision, self-management strategies, and HIV counseling and testing. In addition, attitudinal arguments, which had favorable effects in nonclinical settings, had a reverse effect in clinical contexts.
We next compared intervention strategies for school and nonschool settings. As judged by the significant interactions in the last column of , behavioral skills arguments and threat-inducing arguments both had less impact in schools than in other places. Notably, however, normative arguments and condom use skills training had significant positive effects only in schools. When the setting was not a school, normative arguments continued to have the previously reported reverse effect and condom use skills training had a nonsignificant effect.
With respect to community settings, the effects of information, behavioral skills arguments, threat-inducing arguments, interpersonal skills training, and self-management skills training, which were significant in noncommunity settings, were nonsignificant when the intervention was conducted in the community. Normative arguments had a significant negative effect in community settings, although the effect was weaker than the one in noncommunity settings. HIV counseling and testing continued to have a positive effect in community settings, although it was smaller in size relative to the one in noncommunity settings.
The second to last panel in presents the effects of playing a video- or audiotape. As can be seen, playing a tape was associated with an increased positive impact of attitudinal and behavioral skills arguments and HIV counseling and testing, as well as with increased negative effects of normative arguments and interpersonal skills training. In contrast, the favorable effects of self-management skills training were stronger when the intervention did not include a tape, and the provision of condoms had a positive effect when no video was used but a negative effect when a video was used.
Finally, we analyzed whether the use of group sessions as part of the intervention coincided with increases or decreases in the effects of different intervention strategies. As seen from the last panel of , attitudinal arguments, information, self-management skills training, and HIV counseling and testing were more effective when the intervention included group sessions, whereas behavioral skills arguments and condom use skills training were more effective when the intervention did not include group sessions. Finally, normative arguments had stronger negative effects during group than individual sessions.
We also examined the possibility that other participant and intervention factors could moderate behavior change and also be responsible for the outcomes. First, we regressed d. for behavior on the participant and intervention variables in that we had not previously analyzed. As could be observed from the fixed-effects simple regressions, change in condom use was positively associated with percentage of high school graduates (β = .13, p < .001, k = 83); city population (β = .37, p < .001, k = 180); rate of HIV infection at pretest (β .42, p < .001, k = 50); and face-to-face presentation of the intervention (β = .12, p < .001, k = 200). Also, change in condom use correlated negatively with inclusion of participants with a history of STIs (β = −.16, p < .001, k = 200); inclusion of college students (β = −.12, p < .001, k = 200); and inclusion of middle and high school students (β = −.33 and −.12, respectively,p < .001 and k = 200 in both cases).
We also analyzed other associations with methodological features of the studies. These analyses revealed significant positive associations of behavior change with (a) the use of within-subject designs (β = .18, p < .001, k = 200); (b) random assignment of participants to conditions (β = .29, p < .001, k = 200); (c) amount of payment (β = .05, p < .05, k = 200); (d) number of days between the intervention and the posttest (β = .09, p < .001, k = 191); (e) the use of a theory-based intervention (β = .10, p < .001, k = 200); (f) targeting interventions to specific genders (β = .11, p < .001, k = 200); and (g) self-selection bias (β = .13, p < .001, k = 200). Moreover, change in condom use correlated negatively with (h) the use of formative research (β = −.12, p < .001, k = 200) and (i) attrition (β = −.07, p < .001, k = 111, k = 200). However, the negative effect of using formative research became nonsignificant (β = −.12, ns) when we reran that predictor in a multiple regression including all the methodological and population predictors entered simultaneously.
Because these supplementary analyses identified a number of factors that influence behavior change, we reran the analyses in – to ensure that the described effects were not due to the association of the population and intervention characteristics we analyzed with other methodological features of this study. Education, pretest HIV infection rates, and attrition could not be introduced owing to low report of these factors. However, introducing the other methodological variables in did not alter the patterns of findings we discussed.
The analyses in suggest that arguments designed to improve attitudes and behavioral skills in favor of condom use increase condom use across passive and active interventions. However, these analyses cannot confirm that these strategies have an impact because they affect the mediator they are supposed to affect. For example, it is unclear thus far whether the interventions designed to improve attitudes and behavioral skills actually managed to do so. In addition, attitudinal arguments convey not only that “using condoms is good” but also that “the communicator thinks that using condoms is good.” Consequently, the impact of attitudinal arguments on condom use could be mediated by changes in norms instead of changes in attitudes. Similarly, hearing a message about protection from a disease could spontaneously arouse anxiety, in which case perceived threat could be the mediator as well.
Two caveats are necessary when considering the use of path analyses in meta-analysis. There is pressure both to maximize the inclusion of effect sizes and to maintain the included effect sizes across analyses (avoiding pairwise deletion procedures). For example, because we concluded that attitudinal arguments were effective on the basis of an analysis of 200 conditions, the mediational analyses should include those 200 effects. This strategy, however, is complicated by the fact that not all studies measured the same variables, and data on potential mediators are much less frequent than data on condom use itself (see ). Therefore, to maintain the original 200 units while including the available data on a particular mediator, one must resort to pairwise deletion procedures, which often produce nonpositive definite matrices (Shadish, 1996
In light of the complications involved with the study of mediation in meta-analysis, several approaches were explored. First, we attempted to fit models to a matrix that included, in addition to condom use, the indicators for all the intervention strategies in and and all psychological variables in . These models yielded impossible solutions and were therefore discarded. Next, we proceeded to fit models to smaller matrices. Of the various possibilities, we chose to report models that would parallel the analyses in and . These models included the indicator variable for the strategy being considered, the likely mediator for that strategy, and change in condom use, plus the indicators for all other strategies in and . However, the matrices involving normative arguments and change in norms as well as threat-inducing arguments and either perceived risk or threat were non-positive definite, which led us to analyze the mediation of only the strategies that had favorable effects on condom use. The analyses we report were estimated using maximum likelihood methods and the lowest N
in pairwise deletion matrix. Sobel (1982)
tests were calculated and are presented along with the path diagrams in –. For the sake of simplicity, these path diagrams show only the paths relevant to the strategy that is the focus of each panel, even when all the models included the predictors in and , depending on whether passive or active strategies were analyzed.
Figure 2 Path analyses to determine the mediating effects of change in specific psychological variables on changes in condom use among passive interventions. A: Effects of attitudinal arguments. B: Effects of behavioral skills arguments. Both models also included (more ...)
Figure 4 Path analyses to determine the mediating effects of change in specific psychological variables on changes in condom use as a function of active strategies among active interventions. A: Effects of self-management behavioral skills training. B: Effects (more ...)
summarizes the findings from the path analysis for the effects of attitudinal and behavioral skills arguments, which had significant, positive main effects across passive and active interventions. As shown in Panel A, the positive effects of attitudinal arguments on behavior change were mediated by changes in attitudes. The influence of attitudinal arguments, however, was also mediated by norms and perceived threat, which suggests various ways in which this type of strategy has an influence. In addition, the analyses in Panel B indicate that the possible influence of behavioral skills arguments on condom use change was mediated by control perceptions. However, as can be seen, the direct effect of behavioral skills arguments on behavior became nonsignificant only once we introduced changes in behavioral skills, and the mediation test suggested that behavioral skills was in fact a plausible mediator.
summarizes the effects of information, which was significant only in the context of active interventions (see ). As can be seen, the favorable effects of information on condom use were in fact mediated by increases in knowledge about HIV. The path model shows that the positive direct effect of information on behavior change became slightly negative once changes in knowledge were included.
Figure 3 Path analyses to determine the mediating effects of change in specific psychological variables on changes in condom use as a function of information among active interventions. This model also included all the strategies used in active interventions (see (more ...)
presents the effects of self-management skills training and HIV counseling and testing, which had significant effects in the sample of active interventions (see also ). As one might expect, the effects of self-management behavior skills training strategies were mediated by changes in both control perceptions and behavioral skills. The effects of HIV counseling and testing were less clear, which led us to conduct analyses with various potential mediators. These analyses (see , Panel B) indicated that HIV counseling and testing contributed to changes in skills. Changes in skills, in turn, correlated with changes in condom use, and their inclusion reduced the size of the direct effect from HIV counseling and testing to condom use.8
Assessment of Publication and Eligibility Biases
Of course, publication practices and eligibility criteria shape the sample of reports that are included in a meta-analysis. For instance, 12 of the examined reports contained insufficient statistics to derive the necessary effect sizes (see footnote 1). In addition, although we closely examined 15 unpublished reports, only one was ultimately included. To estimate potential biases in the report of findings and study inclusion, we examined the funnel plot of behavior change effect sizes (see ) and the normality of the distribution under examination (see ). If no bias is present, the plot takes the form of a funnel centered on the mean effect size, with smaller variability as the sample size increases. In the presence of publication bias, there is a distortion in the shape of the funnel. If the true effect size is zero and there is bias, the plot has a hollow in the middle. If the true effect size is not zero, the plot tends to be asymmetrical, having a large and empty section where the estimates from studies with small sample sizes and small effect sizes would otherwise be located. Following these guidelines, a subjective examination of the plot in thus suggests no publication or selection bias in our meta-analysis.
Funnel plot. Two effects with extremely large sample sizes were excluded to make the shape of the plot more apparent. These large sample groups had average effect sizes.
Normal quantile plot. The line on the diagonal indicates normality; the lines around the diagonal represent the 95% confidence interval around the normality line.
In addition to examining the funnel plot, we used the normal quantile plot method to uncover evidence of bias (Wang & Bushman, 1999
). In a normal quantile plot, the observed values of a variable are plotted against the expected values given normality. If the sample of effect sizes is from a normal distribution, data points cluster around the diagonal; if the sample of effect sizes is biased by publication practices or eligibility criteria, data points deviate from the diagonal (Wang & Bushman, 1999
). As can be seen from , the standardized behavior effect sizes followed a straight line and generally fell within the 95% confidence intervals of the normality line. This conclusion was supported by the fact that our findings remained unaltered after excluding the most extreme outliers from the sample of conditions (see the seven extreme observations in ). In sum, there was convincing evidence that even if one determined that a large number of studies have been kept in researchers’ file cabinets, inclusion of these studies would be unlikely to alter our conclusions about the effectiveness of HIV-prevention interventions.