|Home | About | Journals | Submit | Contact Us | Français|
The reliability and validity of self-reports of vaginal microbicide use are questionable given the explicit understanding that participants are expected to comply with study protocols. Our objective was to optimize the Population Council's previously validated dye stain assay (DSA) and related procedures, and establish predictive values for the DSA's ability to identify vaginally inserted single-use, low-density polyethylene microbicide applicators filled with hydroxyethylcellulose gel.
Applicators, inserted by 252 female sex workers enrolled in a microbicide feasibility study in Southern India, served as positive controls for optimization and validation experiments. Prior to validation, optimal dye concentration and staining time were ascertained. Three validation experiments were conducted to determine sensitivity, specificity, negative predictive values and positive predictive values.
The dye concentration of 0.05% (w/v) FD&C Blue No. 1 Granular Food Dye and staining time of five seconds were determined to be optimal and were used for the three validation experiments. There were a total of 1,848 possible applicator readings across validation experiments; 1,703 (92.2%) applicator readings were correct. On average, the DSA performed with 90.6% sensitivity, 93.9% specificity, and had a negative predictive value of 93.8% and a positive predictive value of 91.0%. No statistically significant differences between experiments were noted.
The DSA was optimized and successfully validated for use with single-use, low-density polyethylene applicators filled with hydroxyethylcellulose (HEC) gel. We recommend including the DSA in future microbicide trials involving vaginal gels in order to identify participants who have low adherence to dosing regimens. In doing so, we can develop strategies to improve adherence as well as investigate the association between product use and efficacy.
Microbicide clinical trials seek to test products designed to reduce or prevent transmission of HIV and/or other sexually transmitted infections (STIs).(1) As administration of vaginal and/or rectal products is participant dependent, one of the major challenges confronting these trials is the measurement of product adherence.(2-6) The reliability and validity of self-reports of product use are questionable given the explicit understanding that participants are expected to comply with study protocols.(7) Social desirability, fear of being excluded from the trial, and/or embarrassment about not using the product may result in participants' over-reporting of product use. Accurate measurement of adherence is particularly important for trials that fail to demonstrate efficacy as primary analyses on the intent-to-treat population may be less likely to demonstrate efficacy if participants fail to use the product.(7-9)
Concerned with the quality of self-reported data, and in advance of the Carraguard® Phase 3 trial, Population Council scientists sought to develop an assay to objectively measure vaginal insertion of applicators. While tests for the growth of lactobacilli on inserted applicators were 100% predictive of vaginal insertion, this method was not considered desirable because it was time consuming, and the percentage of women with culturable lactobacilli is low.(10) Council scientists therefore developed an assay where the tips of applicators are sprayed with an aqueous blue dye solution that adheres to vaginal mucus present on applicators after insertion, producing a grainy, streaky turquoise pattern. This is distinct from the smooth and shiny pattern that is present on noninserted applicators when the dye adheres to residual gel only. The pattern is especially visible in and around the ‘seam’ or grooves of the applicator tip.(10) [See Figure 1]
This dye stain assay (DSA) was validated twice using single dose Microlax®-type low-density polyethylene applicators (LDPE) filled with 2.5% methyl cellulose (MC) gel; positive controls were provided by female study participants who were instructed to both insert the applicator and expel the gel into their vaginas.(10, 11) The first validation study used a 1% (w/v) Trypan Blue dye solution.(10) The second study used a 0.05% (w/v) FD&C Blue No. 1 Granular Food Dye solution due to concerns about safety of the staff performing the DSA, since Trypan Blue is an irritant and potential carcinogen.(11, 12) For both studies, applicator tips were sprayed with dye solution to thoroughly coat the applicator (3 sprays within approximately 5 seconds); sprayed applicators were rinsed after 10 seconds, and left to dry overnight. Stained applicators were visually inspected to determine whether they had been vaginally inserted. The first study reported that the DSA had 97.5% sensitivity (the probability that an inserted applicator was read as positive); specificity (the probability that an applicator that had not been inserted was read as negative) was not reported.(10, 13) The second study reported 97.5% sensitivity and 96% specificity.(11) Similar results were reported when applicators were stored for up to four months and/or when inserted applicators were stored in the same plastic bag along with noninserted applicators.(10, 11)
Based on these validation studies, the DSA was deemed to be an effective tool to validate self-reported vaginal applicator use for the Carraguard® Phase 3 trial.(10, 11, 14) Participants were instructed to use the gel before each act of sex, and the DSA was used to determine how product use corresponded to sex acts; the number of inserted applicators according to the DSA was compared to the number of sex acts reported by participants. According to self-reports study product was used during 96% of sex acts, but DSA results indicated that participants used the study product during only 42% of sex acts. Further, only 61% of returned, opened applicators were identified as vaginally inserted.(14) In a three-month follow-up placebo study in South Africa assessing the effect of interview mode on adherence self-reports, the DSA indicated that 52% of opened applicators had been vaginally inserted, resulting in a mean difference ranging between 2.3 and 4.1 inserted applicators per month when comparing self-reports to the DSA.(15) Depending on month of participation and interview mode, a range of 31-60% of follow-up study participants reported ever having sex without inserting the gel.(15) In both the Carraguard® Phase 3 trial and interview mode study, the number of sex acts reported was greater than the number of applicators that tested positive for insertion; participants appear to have had sex without inserting the gel before-hand, as instructed.(14, 15)
Council researchers contended that over-reporting of gel use was due to social desirability bias; the presumption was that participants squeezed the contents of unused applicators out ex-vivo before returning them to the clinic so as to appear adherent.(14, 16) Other researchers in the field speculated that the DSA was inadequately validated (16, 17) and that it would work only with the LDPE applicators with which it was originally validated.(18) Consequently, two DSA studies were conducted by other researchers in the field.(18, 19) The first study tested two modified DSAs on single-use LDPE applicators filled with 0.5% PRO2000 or matched placebo gel.(19) Applicators were either sprayed with a 0.4% (w/v) Trypan Blue solution or immersed in a 0.4% (w/v) Trypan Blue water bath for one minute; researchers waited 5-10 minutes prior to rinsing. The study concluded that the DSA was accurate and useful for future microbicide trials.(19) The second study tested single-use high density polypropylene (HDPP) applicators that were either filled with UC-781 (used once daily) or VivaGel® (used twice daily).(18) A 0.05% (w/v) FD&C Blue Dye No. 1 solution was used. It is unclear, however, how long the dye remained on the applicators between spraying and rinsing; in previous studies, dye remained on applicators for 10 seconds before rinsing in order to ensure that dye solution that adheres to vaginal mucus remains visible after washing.(10,11, 18) Variability among evaluators was reported, with the sensitivity of the assay ranging between 81-95% for single-use applicators, and between 44-71% for applicators used twice daily. Researchers speculated that the DSA performed with lower sensitivity with applicators used twice daily due to gel remaining in the vagina from the previous insertion.(18) Results focused on the lower than optimal performance with applicators used twice daily, ignoring the high sensitivity of the DSA with applicators used once daily.(18) A recent study of 1% tenofovir gel utilized the DSA with single-use polypropylene applicators; high interobserver variability and a relatively high false positive rate lead researchers to conclude that the DSA is not effective for determining polypropylene applicator insertion.(20,29) See Table 3 for a summary of all DSA validation experiments published to date.
Results from the recent CAPRISA 004 trial of tenofovir gel, bolstered by those from the iPrEX PrEP trial of Truvada® (Emtricitabine combined with tenofovir), provided proof of concept that ARV-based products used before and after sex reduce HIV incidence.(9, 21) Adherence to ARV-based products is particularly crucial because inconsistent use may lead to drug resistance.(22) Both studies reported an association between product adherence and HIV protection, as well as variability between self-reports and more objective measures such as pill and applicator counts, and drug levels in blood.(9) As it is difficult to detect drug levels in blood with topically applied products due to low systemic absorption, biomarkers of gel adherence, such as cervicovaginal lavage (CVL) and measurement of drug levels in tissue, are currently being investigated.(23) In the meantime, researchers continue to rely on self-reports and counts to measure product use. Considerable effort has been devoted to improving the reporting of adherence through counseling and/or innovative reporting tools such as audio computer-assisted self-interviews (ACASI), interactive voice response surveys (IVRS), and short message service (SMS).(8, 9, 15, 24, 25)
The DSA is currently being employed by Population Council researchers to validate the self-reported use of the ‘universal placebo’ hydroxyethylcellulose (HEC) gel (26, 27) in a study among sex workers in India. While training site staff on the implementation of the DSA, the pattern characteristic of inserted applicators was not clearly visible on applicators known to have been inserted. HEC that remained inside the applicator after insertion had leaked out of the applicators during storage; other publications have suggested that gel may interfere with the DSA.(18, 19) Prior to implementing the DSA in India, and in order to respond to criticism of the assay's accuracy, Population Council researchers sought to systematically and comprehensively validate the DSA by: (i) optimizing the assay and related procedures, and (ii) establishing predictive values for the DSA's ability to identify vaginally inserted applicators. This paper presents the results of these experiments and is the first to report all four predictive values for the DSA: sensitivity, specificity, and positive and negative predictive values.(13)
Study applicators consisted of single-use, twist-off top, low density polyethylene (LDPE) Microlax®-type applicators (Tectubes, Åstorp, Sweden). Applicators were filled with 7 mL of 2.7% Hydroxyethylcellulose (HEC) placebo gel (Clean Chemical Sweden AB, Borlänge, Sweden) and delivered approximately 3.5 mL of gel.
Female sex workers (FSWs) of reproductive age (18 to 45 years) were enrolled in a placebo microbicide study in Southern India. All participants gave informed consent in accordance with established guidelines and ethical standards for experimentation with human subjects. At enrollment, participants were trained to insert HEC-filled applicators and inserted their first applicator in the presence of the study clinician. Applicators inserted at the clinic by 252 participants served as positive controls for the experiments; each participant provided a single applicator. Inserted applicators were plugged with the stick-end of a cotton swab to prevent leakage and stored at ambient room temperature (no higher than 25°C) in separate, labelled plastic bags for up to 5 months. Lab technicians created negative controls by squeezing HEC gel out of applicators, ex vivo. Negative controls were plugged and stored under the same conditions as positive controls, but were stored for only a few hours prior to testing. Blinded test kits were prepared by randomly coding and labelling positive and negative controls which were equally distributed in each kit.
The DSA involves spraying applicator tips with an aqueous solution of FD&C Blue No. 1 Granular Food Dye. Prior to DSA validation, a 3 × 3 factorial optimization experiment was performed involving 54 positive controls stored for no more than one month prior to testing and 54 negative controls, to ascertain optimal dye concentration and staining time (i.e. amount of time dye remains on the applicator prior to rinsing). Spray bottles were filled with 0.01%, 0.05% and 0.25 % (w/v) dye solutions. Applicators were sprayed 3 times within approximately 5 seconds; lab technicians waited 5 seconds, 5 minutes, or 10 minutes before rinsing. After applicators dried, readers visually examined them for the presence of the characteristic pattern indicating that applicators were “positive” (vaginally inserted), or the absence of this pattern indicating that applicators were “negative” (not vaginally inserted). [See Figure 1]
Once the optimization experiment was complete, three validation experiments were conducted; each validation experiment involved 66 positive controls and 66 negative controls. Readers' observations were scored and analyzed to determine sensitivity, specificity, negative predictive values (NPVs), and positive predictive values (PPVs).
Four to six laboratory and clinical staff members involved in the India study served as readers. Readers were trained in how to identify positive applicators by a Population Council staff member, who also served as a reader.
Each reader independently examined and evaluated the applicators. Readers reviewed results from the optimization experiment to improve identification of inserted applicators for the subsequent validation experiments. Readers' observations and scores from the validation experiments were not disclosed until all experiments were completed. One individual, who did not serve as a reader, created the blinded test kits, recorded and scored all observations. Observations were scored as “True Positive,” “True Negative,” “False Positive,” or “False Negative.”
Sensitivity and positive predictive values (PPVs) tested for the probability that an inserted applicator would be read as positive. Specificity and negative predictive values (NPVs) tested for the probability that an applicator that was not inserted would be read as negative.(13)
Sample size calculations performed with SAS 9.2's Proc Power were based on an exact test of binomial proportions. They indicated that the analysis of 66 positive controls and 66 negative controls would provide evidence sufficient to reject the DSA if the sensitivity (or specificity) was no more than 90%. Friedman's tests were performed with SAS 9.2's Proc Freq to detect between experiment differences in sensitivity, specificity, NPV, and PPV.
Based on the optimization experiment, the dye concentration of 0.05% (w/v) and staining time of five seconds was determined to be optimal (100% sensitivity and 91.7% specificity, on average). This combination of dye concentration and staining time was used for the three validation experiments. [See Table 1]
There were a total of 1,848 possible applicator readings across all three validation experiments; 1,703 (92.2%) applicator readings were correct. On average, the DSA performed with 90.6% sensitivity, 93.9% specificity, and had an NPV of 91.0% and a PPV of 93.8%. [See Table 2]
To validate the DSA (experiment 1), five readers evaluated 132 applicators, for a total of 660 possible readings. Of these, 607 (92.0%) applicator readings were correct. On average, the DSA performed with 92.4% sensitivity and 91.5% specificity, and with an NPV of 92.4% and a PPV of 91.7%.
To validate the DSA for use with “older” applicators (experiment 2), four readers evaluated 132 applicators, for a total of 528 possible readings. Of these, 493 (93.4%) applicator readings were correct, and on average, the DSA performed with 91.7% sensitivity and 95.1% specificity, and had an NPV of 92.1% and a PPV of 95.0%. To determine repeatability, applicators were sprayed, rinsed and dried by the same individual who performed these steps for the optimization and first validation experiments.
To validate the reproducibility of the DSA (experiment 3), applicators were sprayed, rinsed and dried by an individual who did not perform these steps in the previous experiments, and five readers evaluated 132 applicators, for a total of 660 possible readings. Of these, 603 (91.4%) applicator readings were correct and, on average, the DSA performed with 87.6% sensitivity and 95.2% specificity, and had an NPV of 88.5% and a PPV of 94.8%.
No statistically significant differences between the three validation experiments were noted.
The DSA was optimized and successfully validated for use with single-use, LDPE applicators filled with HEC placebo gel. The combination of dye concentration and staining time determined to be optimal was 0.05% (w/v) and 5 seconds. Across these experiments, averages of sensitivity, specificity, NPV, and PPV were at least 90%, and no significant differences were found between the results of the 3 experiments. As there were no significant differences between the results of experiments 1 and 2, the DSA performed well with applicators that were stored for up to 4 months and is repeatable. As there were no significant differences between the results of experiments 2 and 3, the DSA can be replicated.
In order to compare the results of previous experiments with the results of our experiments, sensitivity, specificity, PPVs and NPVs were derived from data presented in other DSA validation publications, when available [See Table 3]. Based on these data the DSA is reliable when used with a number of different gel and applicator combinations: low-density polyethylene applicators filled with either MC or HEC, PRO2000 (or matched placebo) and high-density polypropylene applicators filled with UC-781, if used once daily.
All DSA experiments conducted to date have sought to identify applicators that were vaginally inserted; researchers have not yet conducted similar DSA experiments with gel products/applicators designed for rectal use or with ARV-containing products. While the DSA can easily be replicated, as the evaluation of sprayed applicators relies on visual inspection, hands-on training and practice is needed to reduce subjectivity identified by the high degree of observer variability. The DSA cannot be defined as a biomarker,(28) and while the DSA cannot provide information about whether the product was actually expelled into the vagina, the timing of gel use, nor the amount of product used, a biomarker might not be able to provide that information either.(2) Future studies should seek to validate the DSA for use with applicators inserted after sex in order to determine whether the presence of semen interferes with the DSA.
Previous studies have highlighted the simplicity of the DSA as a surrogate marker to measure gel adherence and the relatively low cost of implementing it within a large-scale trial. Accordingly, even if the use of a true biomarker becomes feasible, the DSA's low cost and relative ease might make it a useful tool for the ongoing monitoring of trials.(11, 29) Recent publications suggest that the DSA might be used as part of a triangulated approach to measuring adherence, in order both to capture accurate adherence information and reduce the error introduced by any one particular method.(1, 17) We recommend including the DSA in future microbicide trials involving vaginal gels in order to identify participants who have low adherence to dosing regimens. In doing so, we can develop strategies to improve adherence(1, 9) as well as explore the association between adherence and efficacy, which is especially important in trials with inconclusive results.(10)
The authors would like to thank Meredith Paddock and Barbara Miller for their brilliant assistance in the preparation of this manuscript, and Sharon Abbott, Barbara Friedland, Sarah Littlefield, Deborah Tolenaar and Waimar Tun for their insightful comments, guidance and support. We would also like to thank C. Chellamani, C. Shalini, C. Sivanesan, Madhan and V. Vijaya for their dedicated work on the India study.
Sources of support: This manuscript is made possible by the generous support of the American people through the United States Agency for International Development (USAID), Bureau for Global Health, Office of Population and Reproductive Health, under the terms of Award No. GPO-A-00-04-00019 and by Grant Number R21 HD060270 from Eunice Kennedy Shriver National Institute of Child Health and Human Development. The contents of this manuscript are the sole responsibility of the Population Council and do not necessarily reflect the views of USAID, NIH, or the United States Government.
Conflict of interest statement: No conflict of interest exists.
Lauren L. Katzen, Population Council.
José A. Fernández-Romero, Population Council.
Avina Sarna, Population Council.
Kailapuri G. Murugavel, YRG CARE.
Daniel Gawarecki, Population Council.
Thomas M. Zydowsky, Population Council.
Barbara S. Mensch, Population Council.