We used clinical and behavioral data collected from an STD clinic to develop two risk scores predictive of HIV infection, and validated those scores using data from a large multi-center behavioral intervention trial. While the scores we developed would optimally undergo additional validation in other populations,50, 51
we believe our simple score can be useful to clinicians and others in counseling MSM about their risk of HIV infection, and might be used to identify persons who require intensified interventions or more frequent HIV testing. The resources necessary to conduct biomedical and behavioral HIV prevention trials depend on HIV incidence.46, 52, 53
Thus, this score might also help researchers identify MSM at high risk of HIV infection for prevention trials with HIV acquisition as the primary outcome.
Despite the model’s simplicity and excellent calibration, its discriminatory accuracy was only modest (AUC, 0.66). However, this finding should be interpreted with the understanding that extremely high relative risks (>100) are required to generate risk predictor models with high AUC estimates,12, 54–56
and that such models are extremely uncommon in clinical practice. The AUC estimates that we report are similar to those of other risk models, such as the Framingham risk score for coronary heart disease (AUC, 0.63 to 0.83),57
that are commonly used to guide clinical decisions.
While the calculation of the risk scores may seem daunting, the scores we present are not more complicated than the Framingham risk score9
or the CHADS score14
, tools that clinicians commonly use to estimate patients’ risk of coronary heart disease and the risk of stroke among persons with atrial fibrillation, respectively. Our MSM risk score should not require the use of a computer for calculation, and may be suitable for use in non-clinical settings or on the Internet. We are currently developing a public health website that will allow MSM to calculate their score and explore their risk of HIV acquisition. An approach to calculating the simple risk score and estimating a man’s 4-year risk of acquiring HIV is presented in the .
Estimating a patient’s 4-year risk of HIV acquisition according to the simple model.
The ideal risk score cut-off may be one that leads to the follow-up of a number of patients that the resources of a clinic, provider, or community-based organization can accommodate. We found that using a score that identifies men with any risk predictor (i.e., risk score ≥1), our simple model would identify 83% of STD clinic patients and 86% of Explore participants who acquired HIV over 4 years, but would require follow-up of approximately 70% of the two populations. Using a higher risk score identifies a population at greater risk, but a smaller proportion of all men who acquire HIV. Currently, our STD clinic employs the simple model to identify men with ≥1 risk predictor for more frequent HIV testing, follow-up counseling by telephone, and reminder notices to return for repeat HIV testing.
The strongest predictors of HIV acquisition in our models were use of methamphetamine or inhaled nitrites and a history, or current diagnosis, of bacterial STD, findings that highlight the importance of routinely asking MSM about these drugs and of concentrating prevention efforts in persons with bacterial STD. While non-concordant UAI has typically been the strongest risk factor for testing HIV-positive in cross-sectional studies,33
it was a weaker predictor of HIV acquisition in our models. This difference may reflect the distinction between behaviors or characteristics that operate at the individual-level and those that operate at a network-level. For example, substance use and STD may indicate that an individual circulates within a sexual network more conducive to HIV transmission, while the report of non-concordant UAI does not always indicate this network-level risk. Additionally, we found that the agreement in reports of non-concordant UAI over four years of follow-up was only fair. This fair agreement likely diminishes our predictive models’ discriminatory ability. We overestimate the risk of HIV acquisition for men who report non-concordant UAI at baseline, but do not practice non-concordant UAI during the follow-up period. Conversely, we may underestimate the risk of HIV acquisition for men who do not report non-concordant UAI at baseline, but engage in non-concordant UAI during the follow-up period.
Our study is subject to important limitations. First, our risk score was derived from a population of MSM who repeatedly tested in a single U.S. STD clinic. Not all MSM in our clinic tested serially during the study period, and studies suggest that risk behavior may influence HIV testing behavior.58, 59
On the other hand, our models performed reasonably well among MSM from six U.S. cities who tested at 6-month intervals as part of Project Explore, suggesting that the prediction models can be applied to a more diverse population of MSM. However, as our development and validation samples were composed mostly of white MSM, we cannot be sure of our models’ performance in racial and ethnic minority MSM. Additionally, we cannot be sure of our models’ applicability to populations and geographic areas with lower rates of methamphetamine and inhaled nitrites use.
Second, data collected from the PHSKC STD Clinic and Project Explore were dissimilar in important ways. Project Explore employed ACASI and collected behavioral and STD diagnosis data for 6-month periods, while clinicians at the PHSKC STD Clinic collected behavioral data through face-to-face interviews, and sexual behavior questions focused on the prior 12 months. Also, data concerning bacterial STD were based on self-report and diagnoses on the day of HIV testing in the STD clinic, but reflected only self-reported history of bacterial STD in the prior 6 months in Project Explore. Misclassification would diminish model discrimination. For example, men in the STD clinic who practiced non-concordant UAI might not acknowledge this behavior because of social desirability bias associated with face-to-face interviews. Therefore, the predictive models would overestimate the risk of HIV acquisition in men who did not report non-concordant UAI in both the development and validation samples.
Prediction models are improved through repeated validation, augmentation, and examination in research and practice.9, 15, 17
We present our prediction models as a step in the process of developing consistent guidelines for HIV prevention practice. Optimally, future work should augment and validate these prediction models in diverse settings and measure the acceptability and utility of prevention efforts guided by prediction models.50, 51
However, even without such additional research, we believe our prediction model can be useful in counseling MSM and in prioritizing intensified prevention efforts to the MSM at highest risk for HIV.