|Home | About | Journals | Submit | Contact Us | Français|
Drs. Arbelaez and Walls: Department of Emergency Medicine, Brigham and Women's Hospital, 75 Francis Street, Boston, MA 02115.
Mr. Reichmann and Drs. Katz and Losina: Orthopaedic and Arthritis Center for Outcomes Research, Brigham and Women's Hospital, 75 Francis Street, BC 4-4016, Boston, MA 02115.
Mr. Block: Partners AIDS Research Center, 149 13th Street, Room 5234k, Charlestown, MA 02129.
Mr. Dooley, Mr. Hetland, Mr. Kimmel, and Ms. Solomon: Department of Emergency Medicine, USHER Trial Research Assistant, Brigham and Women's Hospital, 75 Francis Street, Boston, MA 02115.
Expanded HIV screening efforts in the United States have increased the use of rapid HIV tests in emergency departments. The reported sensitivity and specificity of rapid HIV tests exceed 99%.
To assess whether a reactive rapid oral HIV test result correctly identifies adults with HIV infection in the emergency department.
Diagnostic test performance assessment within the framework of a randomized, clinical trial.
Brigham and Women's Hospital emergency department (Boston, Massachusetts) from 7 February to 1 October 2007.
849 adults with valid rapid oral HIV test results.
Rapid HIV testing with the OraQuick ADVANCE Rapid HIV-1/2 Antibody Test (OraSure Technologies, Bethlehem, Pennsylvania). Patients with reactive rapid test results were offered enzyme-linked immunoassay, Western blot, and plasma HIV-1 RNA testing for confirmation.
Specificity and positive likelihood ratio.
39 patients had reactive results (4.6% [95% CI, 3.2% to 6.0%]). On confirmation, 5 patients were HIV-infected (prevalence, 0.6% [CI, 0.1% to 1.1%]) and 26 were non–HIV-infected (8 patients declined confirmation). The estimated rapid test specificity was 96.9% (CI, 95.7% to 98.1%). Sensitivity analyses of the true HIV status of unconfirmed cases and test sensitivity resulted in a positive likelihood ratio of 8 to 32. Western blot alone as a confirmation test provided conclusive HIV status in only 50.0% (CI, 30.8% to 69.2%) of patients at first follow-up. The addition of HIV-1 RNA testing to the confirmation protocol improved this rate to 96.2% (CI, 88.8% to 100.0%).
Test sensitivity cannot be assessed because nonreactive OraQuick test results were not confirmed.
Although patients with a reactive oral OraQuick HIV screening test in the emergency department had an 8- to 32-fold increased odds of HIV infection compared with the pretest odds, the specificity of the test was lower than anticipated.
In 2006, the Centers for Disease Control and Prevention (CDC) revised guidelines for HIV screening to recommend that all adults and adolescents (age 13 to 64 years) be offered an HIV screening test in health care settings, including emergency departments (1). The availability of rapid HIV tests has facilitated the implementation of such universal screening practices (2–4). Screening guidelines have become more aggressive because, in the era of potent antiretroviral therapy, timely diagnosis of HIV infection is critical to ensure maximal treatment benefits. Favorable treatment outcomes have transformed HIV infection from a fatal condition to a chronic disease with a life expectancy that exceeds 20 years from treatment initiation (5). During the past 3 years, the approval of rapid HIV tests, the newly recommended HIV screening policy, and advances in HIV management have motivated the transition from targeted HIV testing to universal HIV screening.
The performance of screening tests and confirmatory algorithms is critical to feasible implementation of HIV screening in emergency departments. Even highly accurate tests may be reactive in the absence of disease, especially when prevalence of the disease is low. An excessive number of false-positive results could create psychological and financial problems for patients and providers. It is critical that health care providers be appropriately trained to assist patients in interpreting test results and facilitating appropriate follow-up. This is particularly important in the case of a low-prevalence but highly stigmatized disease, such as HIV infection. In the context of a clinical trial on routine HIV testing in the emergency department of a tertiary care hospital, we sought to evaluate the specificity and positive likelihood ratio of the oral sampling method of the Ora-Quick Advance Rapid HIV-1/2 Antibody Test (OraSure Technologies, Bethlehem, Pennsylvania).
The HIV testing program was conceived and implemented within the USHER (Universal Screening for HIV infection in the Emergency Room) Trial, a clinical trial funded by the National Institutes of Health. Patients who enroll in the USHER Trial are randomly assigned to be offered rapid HIV testing by a dedicated HIV counselor or by an emergency department staff member. Eligible patients who consent to trial enrollment are asked to complete a data collection instrument. Consent to trial enrollment does not imply consent to rapid HIV testing. In accordance with recent CDC guidelines (1), enrolled patients undergo limited counseling, which emphasizes that reactive results require confirmation and that rapid results will not be part of the medical record. The HIV counselor or emergency department provider then asks patients for informed consent to rapid oral HIV testing. Because rapid HIV test results become part of the research database, but not the medical record, patients with reactive rapid test results are asked for a third consent to confirmatory testing. Confirmatory HIV test results become part of the patients' medical record. The study is approved by the Partners Human Research Committee of Brigham and Women's Hospital (Boston, Massachusetts) and is overseen by a data safety and monitoring board.
Brigham and Women's Hospital is a tertiary care center in Boston, Massachusetts. The emergency department sees more than 56 000 patients annually, of whom 25% are black, 48% are white, and 20% are Hispanic. The median age of patients who visit the emergency department is 44 years, and approximately 60% are women.
To be eligible for participation in the HIV screening study, patients must be 18 to 75 years of age; have an emergency severity index score of 3, 4, or 5 on a scale of 1 (most severe) to 5 (least severe) (6–8); speak English or Spanish fluently; not be receiving prenatal care; and not knowingly be HIV-infected. Designated recruitment times vary weekly—ranging from 8 a.m. to 12 a.m. for a minimum of 60 hours per week—to adequately sample the full spectrum of “day of week” and “time of day” variation. Research assistants recruit patients for enrollment who are eligible for the study on the basis of age and emergency severity index scores by circulating room to room in the emergency department. For each patient approached, the research assistant describes the study, confirms eligibility, and then requests consent for participation.
To maximize screening acceptability, an oral rapid test is used (OraQuick ADVANCE Rapid HIV-1/2 Antibody Test). Oral samples are collected at the patients' bedside in the emergency department, and tests are run and developed at the on-site emergency department laboratory. All tests are read, as recommended by the product insert, within the 20- to 40-minute reading window (9).
The CDC recommends performing an HIV Western blot to confirm a reactive rapid test result. If the confirmatory HIV Western blot is negative or indeterminate, the CDC suggests a repeated Western blot 4 weeks later to rule out the possibility of acute infection or specimen mix-up (10). Within the trial, all patients with reactive results are offered a comprehensive confirmatory test panel in the emergency department at the same visit as their reactive test. The panel includes serum enzyme-linked immunoassay (EIA [ADVIA Centaur HIV 1/0/2, Bayer HealthCare, Tarrytown, New York]), serum Western blot (Genetic Systems HIV-1 WB, Bio-Rad Laboratories, Redmond, Washington), CD4 count, and plasma HIV-1 RNA testing. In the trial, patients with negative or indeterminate Western blot results are not asked to return in 4 weeks because undetectable plasma HIV-1 RNA generally rules out the possibility of acute HIV infection.
The OraQuick test is waived by the Clinical Laboratory Improvement Amendments and approved for use with careful instruction from the package insert alone (9). All USHER Trial staff doing the test completed the 1-day OraQuick training session conducted by the Massachusetts Department of Public Health and passed the accompanying competency test.
Screening test specificity is calculated as the proportion of screened patients who had a negative OraQuick result among non–HIV-infected patients. Patients with a negative OraQuick screen are not retested in the context of the study. Positive likelihood ratio is calculated, initially assuming perfect sensitivity, as sensitivity divided by (1–specificity). Positive predictive value of HIV infection is calculated as the proportion of patients with an OraQuick reactive result who are HIV-infected. Ninety-five percent CIs for specificity and positive predictive value are calculated on the basis of the normal approximation of the binomial distribution (11).
We also developed a measure of conclusiveness of the confirmatory algorithm. The rate of conclusiveness is defined as the frequency with which a confirmation protocol ascertained the patient's true infection status at the 1-week follow-up visit without requiring further testing. We calculated this measure for the single Western blot confirmation. We also calculated the conclusiveness of the full battery of confirmatory tests, including EIA, Western blot, CD4, and HIV RNA measurement.
In the base-case analysis, all 8 unconfirmed reactive cases were eliminated from the analysis. Therefore, we did sensitivity analyses to examine the impact of alternative assumptions about unconfirmed reactive test results (for example, persons excluded, persons included as truly HIV-infected, and persons included as truly non–HIV-infected). Because we did not confirm nonreactive OraQuick test results, we also examine how the results might be influenced by at least 10 (1.2%) false-negative results.
On receipt of the first 2 false-positive test results, the USHER Trial temporarily ceased enrollment to investigate why these events might be occurring more frequently than expected, according to the product insert. This evaluation included a site visit from the Massachusetts Department of Public Health OraQuick Trainer and 2 representatives from OraSure Technologies. During their assessments, visitors conducted a laboratory audit and witnessed a random subset of our operators conduct the test. We reported the false-positive results to the manufacturer and requested a replacement test kit lot; however, according to the manufacturer, no previous or subsequent problems with the lot were reported. We filed a detailed report to the Partners Human Research Committee and the Data Safety and Monitoring Board. The audit uncovered no flaws in our testing conduct or a reason for the false-positive results. The only consequential protocol change was a revision of the informed consents, which eliminated the claim that the test is 99% accurate.
The National Institute of Mental Health and Doris Duke Charitable Foundation funded this study. The funding sources had no role in the design, analysis, or interpretation of the study or in the decision to submit the manuscript for publication.
From 7 February to 1 October 2007, 2356 patients who visited the emergency department were offered enrollment in the USHER Trial, and 1397 (59.3%) agreed to participate (Figure 1). Of the 1397 enrolled patients, 854 (61.1%) were tested for HIV. Of the 544 patients who were not tested, 144 (26.5%) declined, 326 (60.0%) were not offered testing because either their emergency department stay was too short or the staff was too busy, and 74 (13.6%) were not offered testing for other reasons (for example, the patient was judged to be too sick, not at risk, or unreceptive to testing).
Among patients who were tested, 849 had reportable test results and 5 had invalid results and declined retesting. According to the package insert, invalid results are results that cannot be interpreted because no line appears in the control zone, a red background in the results zone makes the test too difficult to read, or there are lines outside of the C or T test area (9).
Thirty-nine patients had a reactive OraQuick test result (4.6% [95% CI, 3.2% to 6.0%]). On average, patients with a reactive test result were older than those with a nonreactive result (46 years vs. 37 years); other demographic characteris tics did not differ between patients with reactive and nonre-active results (Table 1).
Of the 39 patients with reactive test results, 31 agreed to the confirmatory testing panel (Table 2) and 8 declined. On confirmatory tests, 5 of the 841 patients with reportable test results were confirmed to be HIV-infected, yielding an estimated HIV prevalence of 0.6% (CI, 0.1% to 1.1%) (Table 2).
Twenty-six patients with reactive rapid HIV test results had conclusive confirmatory testing indicating that they were not HIV-infected. Thirteen of these 26 non–HIV-infected patients (50.0% [CI, 30.8% to 69.2%]) had indeterminate Western blot results. Twenty-five of 26 patients initially had an HIV RNA level less than the limit of detection (<75 copies/mL). One patient (patient 13 in Table 2) had an initial HIV RNA level of 86 copies/mL but an undetectable level on a repeated HIV RNA measurement 1 week later. All 26 of these uninfected patients had a nonreactive serum EIA result. Thus, the addition of EIA and HIV RNA measurement to a single Western blot increased the conclusive result rate at 1 week of follow-up to 25 of 26 patients (96.2% [CI, 88.8% to 100.0%)].
The OraQuick false-positive rate was consistent with an estimated screening test specificity of 96.9% (CI, 95.7% to 98.1%) (Table 3). Assuming perfect test sensitivity, the positive likelihood ratio is 32.2 and the posterior probability of infection is 16.2%, at the estimated study prevalence of 0.6%.
Assuming that unconfirmed cases are truly negative, the test specificity is 96.0% and the positive likelihood ratio is 24.8. Alternatively, if all unconfirmed reactive results are truly positive, the test specificity is 96.9% and the positive likelihood ratio is 32.2 (Table 4).
In the case of just 1 false-negative result, test sensitivity decreases from 83.3% to 92.9%, depending on assumptions about unconfirmed cases. Under this scenario, the positive likelihood ratio ranged from 20.7 to 29.8. Assuming 10 false-negative results, test sensitivity ranges from 33.3% to 56.5% and the positive likelihood ratio ranges from 8.2 to 18.0. Because false-negative results affect both sensitivity and likelihood ratio in a complementary manner, the posterior probability of infection ranges from 12.8% to 33.3%, regardless of the assumptions about false negativity (Figure 2).
The USHER Trial screened 849 patients in the emergency department with an oral OraQuick HIV test. The estimated prevalence of newly identified HIV infection was 0.6%, which supports continued screening in the emergency department in compliance with the CDC guidelines (1). During this study, 39 patients (4.6%) had reactive results on rapid oral HIV tests; the estimated specificity of these tests was 96.9% (CI, 95.7% to 98.1%). Depending on assumptions about test sensitivity (33.3% to 100%), the positive likelihood ratio ranged from 8.2 to 32.2. Thus, patients with a reactive oral OraQuick HIV screening test in the emergency department have an 8- to 32-fold increased odds of HIV infection relative to the pretest odds.
The manufacturer reported 99.8% specificity for the OraQuick test (CI, 99.6% to 99.9%) (9). Our results not only suggest that the specificity of the oral OraQuick test is statistically significantly less than that reported by the manufacturer (9) but also demonstrate a false-positive rate (3.1%) more than 15 times greater than the anticipated specificity of 0.2% (based on the observed prevalence and the manufacturer's specificity).
Our results are consistent with other published data that have suggested lower-than-expected specificities when the oral test was implemented in real clinical settings. In 2005, high false-positive OraQuick test rates associated with oral sampling were reported in New York City, Los Angeles, and San Francisco; a cause was never found (12). A CDC study in Minneapolis reported an oral OraQuick test specificity of 99.0% and a positive predictive value of 28.1% (estimated prevalence, 0.4%) (13). More recently, a study in Washington, DC, found that only 9 of 13 patients with reactive oral OraQuick test results were actually HIV-infected (estimated prevalence, 0.4%) (14).
Unfortunately, many studies about routine HIV screening do not report details of confirmation. A PubMed search of studies published in English in 2007 using the keywords “HIV screening” and “emergency department” revealed 7 emergency department–based HIV screening programs in the United States that reported using OraQuick tests (oral or fingerstick). Only 3 of the 7 studies documented whether reactive results occurred in HIV-negative patients (14–18). In all studies that commented on OraQuick false positivity, at least 2 cases of false-positive results occurred. There is no standardized way of reporting rates of false positivity, and often the provided data are inadequate to discern whether lack of reporting is due to incomplete confirmation, follow-up, or linkage to care. Our results and those of others highlight a need for standardized reporting that should include, at a minimum, the type of test (and means of specimen collection), the confirmation algorithm, and the rate of false positivity based on the confirmation protocol.
Substantial rates of false positivity and low specificity highlight the need for a quick and reliable method of confirmation. Twenty-six patients with reactive tests who were not HIV-infected (that is, had false-positive results) consented to confirmation procedures. Among them, 13 (50%) had indeterminate confirmatory Western blot results; however, Western blot is the initial method of confirmation suggested by CDC (10). Taken together in the absence of HIV RNA data, these patients' screening and confirmatory results might mistakenly suggest early HIV infection. We propose the addition of an HIV RNA test, which in combination with Western blot, offered a conclusive confirmation in 96.2% of cases. Although HIV RNA testing is more expensive than Western blot ($120 versus $40 per test) (19, 20), HIV RNA confirmation is unlikely to place a substantially increased financial burden on a testing program because few samples require confirmation.
Our study has several limitations. First, it represents a relatively small sample from a single institution. The literature, however, suggests that other sites are also encountering increased rates of false-positive results (12–14). Second, although our study design allows evaluation of test specificity, it does not allow examination of test sensitivity. Because these tests often have different results when implemented clinically compared with preapproval studies, further studies should be conducted to examine the rate of false-negative results. Third, we did not strictly follow the CDC guidelines, which recommend that patients with a negative or indeterminate Western blot result have repeated testing 4 weeks later. Although the simultaneous results of undetectable HIV RNA testing probably rule out early infection, we cannot exclude the possibility of specimen mix-up. However, samples were handled by usual protocols, which suggests that the likelihood of specimen mix-up is very small. Finally, because confirmation results were available on the medical record, the 8 patients without confirmation of HIV status may have systematically elected not to confirm, introducing verification bias. Our sensitivity analyses indicate that if verification bias exists, it does not substantially change our conclusions.
Our experience shows continued enthusiasm for HIV testing in the Brigham and Women's Hospital emergency department (21). Results indicate that for every 100 patients tested, 95 leave knowing they are negative, and 5 leave with pending confirmatory studies. One of these 5 patients will have HIV infection; the other 4 generally know they are most likely negative within 24 hours (on the basis of a negative EIA result and a normal CD4 cell count) and are confirmed to be HIV negative when HIV RNA results are returned within 1 week. The real efficacy of screening should be measured in true cases identified—cases that, in the absence of screening, would otherwise remain undetected and untreated. Even at the likelihood ratios and posterior probabilities reported in this study, rapid HIV screening functions better than many other screening tests commonly used in U.S. clinical practice. For example, mammography for breast cancer in women age 50 to 59 years has a positive predictive value of 9% (22).
The manner in which providers respond to and support patients with reactive test results—regardless of the frequency at which they occur—is critical to ensure the success of HIV screening programs. It is crucial, for example, that “HIV positive” is not documented in the medical record until appropriate follow-up testing confirms preliminary results of rapid HIV screening. Patients should be notified, as part of the counseling process, that a reactive result in the absence of disease is plausible. Furthermore, providers and patients should be equipped with this information ahead of time so that results can be appropriately managed and discussed.
This study supports the original intent of the HIV screening guidelines: to identify undiagnosed HIV infection. Our findings lead to 2 additional conclusions. First, reactive screening results on the OraQuick test are statistically significantly more likely to occur than the manufacturer suggested in individuals who test negative for HIV. Therefore, greater guidance is required to equip both providers and patients to understand and manage preliminary results most effectively. Also, addition of HIV RNA testing to the confirmation algorithm for rapid HIV screening tests in the United States should be considered.
Few studies have examined performance characteristics of rapid oral HIV testing.
In this study, 39 of 849 adults who visited an urban emergency department had a reactive rapid oral HIV test. Confirmatory tests showed that 26 of the 39 patients were not HIV-infected (specificity, 96.9% [95% CI, 95.7% to 98.1%]). Patients with reactive tests had an 8- to 32-fold greater odds of HIV infection relative to the pretest odds.
Test sensitivity was not assessed because patients with nonreactive results did not receive confirmatory tests.
Rapid oral HIV testing can help identify patients with increased odds of HIV infection.
The authors thank the co-investigators and staff of the USHER Trial, including Carrie Braverman, Susan Larrabee, Regina Mikulinsky, Paul Sax, and Elizabeth Wright. They also thank the faculty and staff of Brigham and Women's Hospital emergency department for their participation in the USHER Trial and for their dedication to the identification of undiagnosed HIV infection, especially the emergency service assistants and focus group members. Finally, they thank Kenneth A. Freedberg, MD, MSc, and A. David Paltiel, PhD, for critical review of the manuscript; Doug Owens, MD, for advice and study critique; and Mariam Fofana for technical assistance.
Grant Support: By the National Institute of Mental Health (R01 MH073445, R01 MH65869) and the Doris Duke Charitable Foundation, Clinical Scientist Development Award (Dr. Walensky).
Author Contributions: Conception and design: R.P. Walensky, E. Losina. Analysis and interpretation of the data: R.P. Walensky, W.M. Reichmann, J.N. Katz.
Drafting of the article: R.P. Walensky, C. Arbelaez, E. Losina.
Critical revision of the article for important intellectual content: R.P. Walensky, C. Arbelaez, W.M. Reichmann, R.M. Walls, J.N. Katz, B.L. Block, M. Dooley, A. Hetland, S. Kimmel, J.D. Solomon, E. Losina. Final approval of the article: R.P. Walensky, C. Arbelaez, W.M. Reichmann, J.N. Katz, B.L. Block, M. Dooley, A. Hetland, S. Kimmel, J.D. Solomon, E. Losina.
Provision of study materials or patients: R.P. Walensky, R.M. Walls. Statistical expertise: W.M. Reichmann, J.N. Katz, E. Losina.
Obtaining of funding: R.P. Walensky.
Administrative, technical, or logistic support: R.M. Walls, B.L. Block, M. Dooley, A. Hetland, S. Kimmel, J.D. Solomon.
Collection and assembly of data: R.P. Walensky, W.M. Reichmann, B.L. Block, M. Dooley, A. Hetland, S. Kimmel, J.D. Solomon.
ClinicalTrials.gov registration number: NCT00502944.
Potential Financial Conflicts of Interest: None disclosed.