|Home | About | Journals | Submit | Contact Us | Français|
There is a strong push to conduct large-scale randomized controlled study designs in HIV prevention studies. In these randomized controlled studies, the primary research objective is typically to determine the treatment effect based on some biological outcome (e.g. HIV infection). But many unutilized self-reported outcomes are also being collected. We illustrate the extent of this problem using the EXPLORE data as an example.
There is a strong push to conduct large-scale randomized controlled study designs in HIV prevention studies. In these randomized controlled studies, the primary research objective is typically to determine the treatment effect based on some biological outcome (e.g. HIV infection). Millions of dollars are spent on these studies and lots of resources are spent towards collecting data on that biological outcome but also on many self-reported outcomes. However, many of these self-reported outcomes, while of great interest, are rarely used in subsequent analyses. Does the extent they are used justify the cost? Does it justify the researcher’s and subject’s burden? What are the alternatives?
Extensive data collection is a general problem and the EXPLORE study is just one example that can be used to compare the volume of data collected against what actually ends up getting used. The EXPLORE study was a multi-site randomized control trial to test the efficacy of a behavioral intervention to prevent acquisition of HIV among men who have sex with men (MSM). It was the first study of its kind to use HIV infection as the primary endpoint. The behavioral intervention consisted of 10 one-on-one counseling sessions followed by maintenance sessions every 3 months. The standard condition was twice-yearly Project RESPECT individual counseling. Participants under both conditions were tested twice each year for the HIV antibody.
At baseline, participants were asked to fill out a survey consisting of 229 questions regarding their risk behaviors. At each follow-up visit, participants were asked 219 questions regarding their risk behaviors. Over the course of the study, on average, participants in the treatment group spent 9 hours answering these risk assessment questions. But this is only a fraction of the data that was collected. Even more time was spent for participants to answer questions pertaining to counseling, STD’s, use of post exposure prophylaxis, etc. By the end of the 4 years, approximately 5000 data points for each participant was collected.
When assessments are lengthy, there is risk that respondents may tire of a particular survey and not answer certain questions, they may change their responses in order to prompt fewer continuing questions, or may even drop out of the study entirely. Recruitment for these studies is already difficult, so keeping participants interested is an important concern. If all of the data being collected was used for analysis, then the lengths of these assessments may be justified. But in reality, it is not clear how much of the data are actually used.
Our review was performed using Google scholar. It enabled us to search across many sources (articles, theses, books, online repositories, universities, academic publishes, and professional societies). The study period ended in July, 2003. The article that presented the main findings of the EXPLORE study was used as our starting point1. On November 26, 2014 all citations that met the following criteria were retrieved:
The search resulted in (n=13) citations that met the inclusion criteria. A reviewer assessed the studies and marked off what questions from the EXPLORE study had been used for analysis. In particular, the focus was to examine the risk assessment questions that were used, since those are the focus of most analyses of HIV prevention research.
The results of the reviewing process are summarized in Table 1.
The primary analysis1 did not utilize any risk assessment behaviors (just HIV status and group assignment). In the secondary analyses, serodiscordant unprotected receptive anal intercourse, serodiscordant unprotected anal intercourse, and unprotected anal intercourse were examined across all time-points in the study. These analyses also utilized baseline risk assessment behaviors regarding injection and non-injection drug use, sex with HIV-positive males/females, unprotected receptive/insertive anal sex, alcohol use, and depression. Looking at the baseline survey alone, we estimate that data from between 33 to 66 of the 229 questions were utilized in these analyses. Looking collectively across all risk assessment surveys given over the 4 year study on risk behaviors, we estimate that data from 185 to 482 of the 1,981 questions were utilized in the secondary analyses.
While the variables utilized in each of the analyses were always clearly stated, it was less obvious what particular questions from the survey were used in data management to define each of the variables. In particular, information regarding serodiscordant unprotected receptive anal sex was mentioned (in some form) across 24 questions on the baseline survey and there was (at times) uncertainty from the reviewers whether all 24 instances were used in any given analysis. To account for this uncertainty, we give a range of our estimates and anticipate that the true value of questions used lies somewhere within the range provided.
At baseline, we estimate that between 75 (38%) and 99 (43%) of the 229 variables collected on risk assessment were used in at least one of the papers that met the inclusion criteria. Furthermore, we estimate that between 435 (22%) and 747 (38%) of the 1,981 variables collected across the 4-year study on risk assessment were used in at least one of the included papers.
One reason we focused on risk assessment questions was to highlight that even the most prominently used questionnaire of the study is not being fully utilized. The EXPLORE team collected dozens of other surveys with thousands of other data points that are even more sparsely used than the risk assessment questions.
We realize that data is collected for a variety of reasons – at times it is collected with a specific analysis in mind, and at times it is collected for the purpose of discovering new relationships to be confirmed in future studies. Although it seems harmless to collect as much data as possible, the extraneous data requires extra work to extract and manage. These other variables could cause oversight on the more critical variables of interest. While the cost of an additional question may seem marginal during the survey design phase, the true cost comes from the impact on data quality.2 As data collection procedures move into more electronic formats, decline in attention span and its impact on data quality becomes even more problematic. This stresses the need to make data collection as efficient as possible.
It is common today to collect biological (blood) specimen, use part of it and freeze the rest for future use. Some would argue the same reasoning for the self-report questionnaires. However, the HIV epidemic is changing drastically all the time and most of the information from these studies will not be applicable for more than several years. There is definitely an expiration date on self-report and biological outcomes data. For example, while the EXPLORE data was collected, HIV drugs were very different than what is available now, treatment as prevention was not considered and community viral load was not an issue.
When surveys are constructed collaboratively, it is possible that researchers all come in with their own favorite set of questions. Perhaps as a way of appeasing all collaborators, all questions become part of the survey. However, at some point, only a specific set of variables are used for analysis, making inclusion of all questions superfluous.
Even if individuals agree to participate, researchers must keep in mind the value of a participant’s time. Since HIV studies are at times conducted on marginalized and socioeconomically disadvantaged populations3, time spent on these surveys could be critical time away from earning money, seeking shelter, or caring for children.
From a statistician’s perspective, there seems to be room for improvement in terms of data collection and study design. In particular, it is important that the whole research group will attempt to reduce patient burden and cost by using more appropriate and innovative study designs and procedures. It is possible to introduce “designed missingness”, where each participant is randomly assigned a subset of questions or assigned to only participate on a subset of time points. With this approach, there will still be flexibility to explore new relationships, but will hopefully be less-burdensome for participants. There is evidence that suggests that this can be achieved without sacrificing the integrity of the inferences to be made 4. These ideas are used by the education research community 5. It may be time to start rethinking how these large-scale studies are designed to adequately justify how resources are spent.
This project was partially supported by Award Number K01MH087219 from the National Institute of Mental Health.
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health or the National Institutes of Health.