The hegemony of the randomized trial design has dampened enthusiasm for alternative designs that are sometimes more appropriate and definitely less expensive for evaluating STI/HIV prevention programs. As the statistical methods described in this manuscript become increasingly accessible, observational designs should experience a revived interest for evaluating STI/HIV prevention, especially as renewed calls for an evidence base in multi-faceted social and structural prevention initiatives gain momentum.11, 35, 36
The methods presented herein to re-weight the study population provide a means to utilize longitudinal, observational study designs for causal estimates of programmatic effects, correcting for the biases common in observational studies. Weighting has not been used widely in prevention research to date; we believe that evaluations of behavioral, social, and multi-level interventions can benefit from use of these methods.
In the Encontros study, despite the likelihood that self-selection and loss-to-follow-up would skew results, there was less bias than expected. There were some differences between participants who remained in the intervention and those who were lost and between those who participated and those who did not, but overall those discrepancies did not substantially alter the estimates of intervention effect. Because we corrected the potential biases through application of weights and used various model selection approaches, we can infer that our findings are not likely due to uncontrolled confounding and that the intervention reduced incident infections among actively participating sex workers. Even so, the possibility of residual confounding, particularly from unmeasured factors, cannot be dismissed, despite utilization of some of the best weighting tools available.
By using IPW we generated marginal estimates of effect: the average intervention effect across the entire study population, as if all participants were in the high exposure group compared to the counterfactual of having all participants in the low exposure group.†
This estimate is more useful in terms of planning for a public health impact than conditional estimates, or those generated controlling for covariates. We ran a conditional multivariate analysis for purposes of comparison and found a very different odds ratio (OR: 0.68). It is not surprising that this fundamentally different effect estimate, which describes the intervention effect in only a subset of the study population, was not very close to the marginal estimates in . Conditional estimates are often reported without acknowledgement that they are estimates of association or effect in a small subset of the study population. Had we run a traditional analysis (non-weighted) we may have interpreted our findings to indicate that the intervention was not successful. Care needs to be taken in reporting conditional estimates and interpreting them as global effect estimates. We also note that this traditional approach relies on the dubious assumption that an a priori specified model is correct. Methods using machine learning to describe the treatment mechanism, such as the DSA described herein, make no such (extremely important) assumptions.
Additional estimation approaches are available to balance observational data, including the G-computation algorithm, which can be used for continuous exposures and is based on imputing the outcomes for individuals keeping all covariates fixed at their original values but modifying the treatment variable of interest.38, 39
Another iteration of the weighting approach is the use of propensity scores. With propensity scores the probability of treatment is modeled for each participant, much the same as it is with IPW; however, the resulting score for treatment probability is used to stratify populations or match participants to allow for estimation of association or effect within comparable strata or matched pairs.40,41
These estimation approaches can also be applied in other observational and in trial designs. For example, IPW can re-balance study populations at different time points in serial cross sectional designs in order to estimate intervention effects while adjusting for secular changes in population. In trials, estimation approaches such as IPW and G-computation can be applied to account for post-randomization differences in distribution of important covariates (i.e. empirical confounding).
Causal inference methods, including marginal structural models, have rarely been presented in forums specific to STI prevention. IPW is one approach that can help improve the quality of and maximize the use of data generated in observational studies and program evaluations of STI/HIV prevention. Given limited resources, rigorous evaluations of community-based interventions using these methods may provide more information regarding successful intervention approaches than funding a limited number of RCTs. Application of IPW is not beyond prevention researchers; however, users should be attentive about meeting the assumptions necessary to apply the methods correctly.