|Home | About | Journals | Submit | Contact Us | Français|
In 2005, the New York City (NYC) Department of Health and Mental Hygiene implemented a standardized human immunodeficiency virus (HIV) incidence surveillance protocol based on the serologic testing algorithm for recent HIV seroconversion deployed nationwide by the Centers for Disease Control and Prevention (CDC). We evaluated four key attributes of NYC's HIV incidence surveillance system—simplicity, data quality, timeliness, and acceptability—using CDC's guidelines for surveillance system evaluation. The evaluation revealed that the system could potentially provide HIV incidence estimates stratified by borough and major demographic groups at about nine months after the period of interest. The system strengths include its relative simplicity and integration with routine HIV/acquired immunodeficiency syndrome surveillance. Weaknesses include lack of completeness of testing history information, a critical component of incidence estimation. Continued improvements in data completeness and timeliness will improve the currently available information to inform personnel who develop HIV-prevention programs and policy initiatives in NYC and nationally.
In New York City (NYC), effective human immunodeficiency virus (HIV) prevention strategies require estimating citywide HIV incidence rates and identifying high-risk populations. However, incidence estimation is challenging because the majority of new diagnoses represent people who were infected one to 10 years before diagnosis.1,2 Routine HIV reporting only captures incident diagnoses and cannot distinguish the new vs. established infections among all new diagnoses. Following longitudinal cohorts from enrollment to seroconversion is expensive and logistically impractical,3 while back-calculating incidence from acquired immunodeficiency syndrome (AIDS) case rates is now challenging because highly active antiretroviral therapy has altered the natural history of AIDS progression.
In 1998, the Centers for Disease Control and Prevention (CDC) introduced a new incidence estimation methodology based on a laboratory algorithm that uses antibody levels to estimate the length of time between infection and diagnosis,4 known as the serologic testing algorithm for recent HIV seroconversion (STARHS).5 When applied to serum specimens from new HIV diagnoses among a target population, the STARHS method estimates incidence from the proportion of specimens in this transient early state.6
STARHS was first used in NYC to assess changes in incidence rates among intravenous drug users (IDUs) during 1990–2002 after the large-scale expansion of needle-exchange programs in the city.7 It has been applied in numerous other settings among different populations.8–18 Statistical methods have now been developed to estimate new infections among untested people, as well those who have been tested. This approach was recently used to calculate widely publicized nationwide HIV incidence estimates.13 It was also recently used by the NYC Department of Health and Mental Hygiene (DOHMH) to publish its first citywide estimate.19
In 2005, NYC and 34 other jurisdictions across the United States began to incorporate HIV incidence surveillance, including HIV subtype B, E, and D (BED) testing, into routine HIV/AIDS surveillance. Periodic evaluation of public health surveillance systems ensures efficient and effective monitoring of public health problems; yet, to date, there has been no systematic evaluation of the incidence surveillance system. This article summarizes the findings of an evaluation conducted of the NYC HIV incidence surveillance system by using CDC guidelines for evaluating public health surveillance systems.20 The evaluation addressed four attributes: simplicity, data quality, timeliness, and acceptability. Based on this information, we discuss the usefulness of HIV incidence surveillance in the ongoing effort to reduce HIV transmission in NYC. We evaluated system attributes for 2006.
The HIV incidence surve illance system is an extension of routine HIV surveillance. Broadly, routine surveillance receives laboratory and provider reports of new HIV diagnoses, matches reports to the HIV registry, updates laboratory information regarding previously reported cases, and investigates nonmatching reports to confirm fact and diagnosis date and collect other surveillance data needed to establish a newly reported diagnosis.
After routine surveillance has identified new diagnoses and verified their eligibility (i.e., Western blot confirmation within three months of HIV diagnosis), incidence surveillance staff performs the following: (1) salvages remnant Western blot-positive (WB-positive) serum; (2) aliquots, selects, and ships eligible serum to the BED testing laboratory; (3) merges BED results to case records; (4) collects information regarding last negative test and treatment history for all cases; (5) creates a monthly dataset for transmission to CDC; and (6) calculates incidence rates.
As illustrated in Figure 1, the process of diagnosis and reporting begins when diagnostic providers first submit a blood sample to a laboratory to be tested for HIV antibody. Results are returned to the submitting provider and must also be reported directly to the New York State Department of Health (NYSDOH), which forwards the results to NYC DOHMH. DOHMH then matches incoming reports to the HIV registry and initiates case investigations for nonmatching diagnostic reports.
During this time, the DOHMH public health laboratory receives most remnant diagnostic specimens and processes them for BED testing. A nominal proportion (3%) is sent directly to the STARHS laboratory from commercial laboratories or providers. The specimens are matched against the HIV registry to verify whether the submitted specimen represents a WB-positive sample that was drawn within three months of the initial HIV diagnosis date. All such specimens are queued to be sent to the STARHS laboratory for BED testing.
STARHS requires use of the Calypte BED™ capture assay (Calypte Bomedical Corp., Portland, Oregon), an enzyme immunoassay that detects B, E, and D HIV-1 subtypes and uses a capture-based enzyme immunoassay (BED capture enzyme immunoassay, or BED CEIA) to identify recent infections on the basis of their HIV-immunoglobulin-G (IgG) to total IgG ratio. The ratio and optical density value corresponding to recent seroconversion have been determined to occur, on average, within 153 days (i.e., about five months) postinfection.5 This period is commonly referred to as the test's window period3,4 (Figure 2). In the spring of 2005, BED CEIA received a surveillance-use-only designation for public health authorities, enabling authorities to link BED-based STARHS results to HIV surveillance data and allowing 34 funded jurisdictions to incorporate incidence surveillance into routine surveillance. The test is not Food and Drug Administration (FDA)-approved for diagnostic testing, and results cannot be reported back to physicians or patients.
Incidence surveillance staff tracks actual submissions against expected submissions weekly and contacts laboratories when shipments are late or incomplete. Specimens are shipped biweekly to the STARHS testing laboratory. After BED results are received, the data are merged with the patients' preexisting case records. Routine surveillance activities include collecting the negative HIV testing and treatment history information needed for incidence estimation of each case. The incidence dataset consists of routine surveillance variables and BED test results. The dataset is transmitted monthly to CDC through a secure data network. Incidence is estimated according to the new CDC stratified extrapolation approach (SEA).21
Incidence is estimated by measuring the number of new infections in a given period in the NYC population at risk (i.e., population not already infected with HIV/AIDS). The number of new infections is estimated from observed new infections, because not all people test regularly for HIV and not all new diagnoses will have serum available for BED testing. Hence, estimating incidence among the NYC population as a whole requires estimating the number of recently infected people among the untested and tested population by using SEA.21 The method requires that each diagnosis has an actual or imputed BED test result and actual or imputed HIV testing and antiretroviral treatment history (TTH). Accurately imputing values depends on completeness of both BED test results and TTH.
SEA is analogous to estimating total population from a sample survey.21 Each new HIV diagnosis is assigned a weight that is based on probability of (1) testing within one year, (2) having a BED result, and (3) the BED test indicating a recent infection. These probabilities are based on TTH obtained from providers. Missing BED results or TTH are imputed by using a 20-fold multiple imputation procedure.13 Each person is assigned a weight that is the inverse of the probability that a person with similar demographics and testing behavior is in the sample. Population total, which includes untested people, is the sum of the estimated weights.
Accurate and precise incidence estimation requires that specimens from as many new HIV diagnoses as possible undergo BED testing and have information concerning previous TTH21 (i.e., date of last negative HIV test and information concerning previous history of antiretroviral treatment). Provider report forms are the main source for the majority of these data, but certain data are also obtained from chart review and by partner notification interviews, which are performed on a subset of new diagnoses (14% as of 2007).
Evaluation and validation of the SEA method itself is ongoing at CDC. Therefore, this evaluation focused on system logistical aspects and whether they provide timely data that meet standards for the SEA method, and simplicity of the system. The attributes selected were simplicity, data quality, timeliness, and acceptability among key stakeholders.
Simplicity refers to ease of operation, structure, and integration of incidence surveillance with routine HIV surveillance. To evaluate simplicity, we assessed all aspects of operation and data flow within incidence surveillance and how it is integrated with routine surveillance by conducting a series of interviews with HIV incidence surveillance personnel. The operational burden, activities, number of steps added to routine surveillance, personnel required, and costs for testing and shipping were summarized for calendar year 2006.
Data quality refers to the completeness of data collected for incidence surveillance and whether they are sufficient for accurate incidence estimation according to established CDC standards. To evaluate data quality, we calculated the percentage of new, eligible HIV diagnoses for 2006 with a BED test result and TTH and compared these values with CDC HIV incidence estimation outcome standards for data completeness. According to these standards, ≥85% of eligible new HIV diagnoses for the diagnosis year should have a BED test and TTH within 12 months of initial diagnosis and measured 12 months after the diagnosis year. To estimate incidence by using SEA for a particular stratum, CDC standards require a minimum of 200 total HIV diagnoses, with ≥40 BED tested and 10 classified as recent infections per cell for SEA. With these criteria, we also determined the degree to which HIV incidence estimates can be stratified by sex, race/ethnicity, age, and borough for 2006 in NYC.
Timeliness refers to the speed with which surveillance information is made available to inform health-care personnel of programmatic initiatives and policymaking. We evaluated timeliness by determining time between steps in the system and lag time to obtain an incidence estimate. We assessed the distribution of overall lag times between diagnosis date and completion of BED testing and time between steps in the incidence surveillance system through an analysis of receipt and shipping dates (obtained from surveillance records) for serum specimens, BED-testing results, and TTH information, as well as from interviews with surveillance personnel.
Acceptability reflects the willingness of people and organizations to participate in HIV incidence surveillance. We assessed acceptability through interviews with key stakeholders at CDC and with participants in incidence surveillance at NYC DOHMH. Because incidence surveillance relies on clinical laboratories to submit remnant serum specimens from newly diagnosed HIV-infected people, we also developed an online survey instrument that assessed clinical laboratory managers' understanding of the submission requirement, staff hours required to fulfill submission requirements, and problems they encounter in achieving the requirement. We hypothesized that such factors as staffing, time, specimen quality and quantity, competing priorities to allocate remnant samples for other research or internal validation studies, or financial concerns might affect compliance (i.e., acceptability).
Further, the goals of HIV incidence surveillance might be unfamiliar to participating clinical laboratories. We sent the instrument, developed using Survey Monkey®,22 to all 22 clinical laboratory managers in NYC who manage specimen salvage and submission to DOHMH. We compiled and analyzed responses using Microsoft® Excel and SAS®.23
The questionnaire consisted of nine questions assessing respondents' understanding of New York State requirements for submitting remnant HIV-positive serum samples, average number of HIV tests performed, and resource availability and limitations.
Implementation of incidence surveillance in NYC requires six personnel dedicated specifically to this system (a principal investigator, epidemiologist, coordinator, data manager, and two laboratory staff). In addition, a staff of nine scientists and other personnel is funded through the HIV incidence estimation grant to support portions of the routine surveillance infrastructure upon which incidence surveillance is built, including supporting collection and management of the required data related to history of negative HIV tests of each case.
Table 1 depicts the following: total number of new diagnoses; number of eligible new diagnoses; eligible diagnoses with a BED test; eligible diagnoses with partial (only date of last negative diagnosis available) or complete (both date of last negative diagnosis and any history of antiretroviral treatment history) TTH; and number of BED-tested new diagnoses with partial or complete TTH. For 2006, a total of 76% of eligible new diagnoses were BED-tested. A total of 50% of eligible new diagnoses had partial testing history, and 39% had complete testing history. Among BED-tested new diagnoses, 51% had partial testing history, and 39% had complete testing history.
The number of new HIV/AIDS diagnoses reported to routine surveillance, the number that were BED-tested, and the number classified as recent infections, stratified by sex, race/ethnicity, age, and borough for 2006, are presented in Table 2. Some data met CDC criteria for minimum cell size for incidence estimation. HIV incidence surveillance gathered sufficient numbers to estimate incidence for all boroughs (except Staten Island), sex, the majority of racial/ethnic groups, and groups aged 20–49 years.
Overall median lag time between diagnosis and receiving a BED test by incidence surveillance was 133 days (25th quantile [Q25] = 104, 75th quantile [Q75] = 169). Median lag time between diagnosis date and receipt of a remnant serum sample from clinical and public health laboratories was 19 days (Q25=6, Q75=39). Median lag time between receipt of remnant samples and shipment for BED testing was 76 days (Q25=50, Q75=112). Approximately 55 of the 76 days were required by routine surveillance to verify new diagnoses. After shipment for BED testing, median lag time for receiving BED results was 24 days (Q25=16, Q75=36).
As indicated in Table 3, from diagnosis date, 90% of BED results from specimens successfully salvaged were received within nine months after initial diagnosis, with the largest proportion of these arriving within four to six months. Approximately 94% of corresponding TTH information for those with any TTH was received by routine surveillance within nine months of diagnosis.
Acceptability of incidence surveillance varied by stakeholders and components of the system. Virtually all respondents indicated that the system and purpose were acceptable. All agreed that the SEA method requires further evaluation and refinement. Incidence surveillance personnel indicated that remnant sample salvage and provider reporting needs improvement.
Of 22 clinical laboratory managers surveyed, 20 (91%) responded and completed the questionnaire. Only nine of 20 (45%) respondents correctly identified the purpose of the clinical laboratory submission requirement as either “testing for new infections” or “incidence estimation” or both. Nine (45%) of 20 laboratorians reported submitting specimens on a predetermined schedule. The reported median number of laboratory staff that laboratories required for complying with submission requirements was two (range: 1–8). The median number of laboratory staff hours per month devoted to preparing and shipping remnant samples was eight (range: 1–60). Thirteen (65%) of 20 laboratorians indicated they had problems with compliance.
Laboratory managers reported a median response of three for staffing and specimen quality and quantity, indicating that they were often a problem. Time and financial concerns received median responses of two, indicating they were sometimes a problem. Needing serum for research projects or internal validation studies had a median response of one, indicating they were never a problem.
Among respondents who reported other problems (n=8), four indicated that packaging specimens was time-consuming, their staffs were improperly trained, pickup service was complicated, and they lacked knowledge about the specimen pickup process. Other requests included that DOHMH provide shipping packages and training or instructions regarding packaging and shipping specimens.
We evaluated four key attributes of the NYC HIV incidence surveillance system—simplicity, data quality, timeliness, and acceptability—using CDC evaluation guidelines. The evaluation revealed that the incidence surveillance system can potentially provide an HIV incidence estimate stratified by borough and major demographic groups at approximately nine months after the period of interest. The strengths of the system include its relative simplicity and integration with routine HIV/AIDS surveillance. The primary weakness of the system is the incomplete reporting of TTH, a critical component of incidence estimation. Improvements in data completeness have been made during the time of this evaluation, and we expect that further improvements in data completeness and timeliness will improve the incidence information available to public health professionals and policy makers in NYC and nationally.
With respect to personnel required at NYC DOHMH, incidence surveillance is a relatively simple system, integrated within routine HIV surveillance and requiring only six additional staff for major tasks (e.g., salvaging remnant serum, shipping samples for BED testing, and calculating incidence). Obtaining TTH requires additional routine HIV/AIDS case surveillance staffing, which is provided for in the HIV incidence grant. Specimen salvage is the least simple component of incidence surveillance, requiring substantial time and effort on the part of incidence surveillance staff to coordinate with commercial laboratories. This issue might be attributable in part to the absence of a predetermined schedule for specimen shipment at most commercial laboratories, despite longstanding state requirements for submission of positive specimens for other communicable diseases. Although the process only requires about eight staff hours/month, clinical laboratories still assign two to three staff for the task. Some laboratory managers indicated lack of familiarity with packaging and shipping processes, despite being provided written instructions by incidence surveillance regarding these tasks.
Nevertheless, the NYC incidence surveillance unit achieved a high completion rate for BED testing—77%—among the highest of all the participating jurisdictions throughout the U.S. Since the beginning of this evaluation, substantial improvements in salvage completeness in NYC have been achieved by sending formal letters of noncompliance to laboratory personnel at the management level, bringing high-level attention to the issue, and improving salvage rates immediately. Incidence staff continue to evaluate whether these improvements have increased completion rates to the 85% criteria recommended by CDC (Unpublished data. Technical guidance for HIV/AIDS surveillance programs, volume 1. CDC, Division of HIV/AIDS Prevention, 2005).
Completion rates for TTH were lower compared with BED testing. Only 39% of new diagnoses have this information, which does not achieve CDC's recommended 85%. Local decreased rates of TTH are attributable to decreased rates of HIV provider case-report form completion. Noncompliance with the HIV report form likely reflects underreporting, a frequent problem for public health surveillance systems,24–27 rather than a specific problem related to routine HIV or incidence surveillance. Routine surveillance personnel regularly conduct active surveillance for these forms, including routine visits to providers and requesting report forms of newly reported cases, yet compliance remains limited.
According to our finding that 90% of BED results on salvaged specimens are received within nine months of diagnosis and 87% of TTH on provider report forms containing this information arrive within six months of diagnosis, incidence surveillance personnel can expect to calculate incidence approximately one year after the last calendar month of the year for which an estimate is sought. Calculating incidence earlier with fewer data is possible, but reduced precision and potentially biased estimates are expected. Proportionately, the majority of this time is related to establishing STARHS eligibility by routine HIV surveillance.
No substantial problems with acceptability were indicated by key personnel at CDC, NYC DOHMH, or NYSDOH. However, commercial clinical laboratorians indicated that packaging and shipping processes were not fully satisfactory. This issue might be an internal problem to individual laboratories, but one that is worth addressing more closely by DOHMH and CDC. Underreporting of testing and TTH information indicates that provider report form acceptability varies among providers.
In its current form, incidence surveillance in NYC has been useful for contributing to the national estimate of HIV incidence21 and for providing NYC with an incidence estimate that can potentially be stratified by sex, racial/ethnic categories (e.g., black, Hispanic, and white), certain age categories, certain risk factors, and four of the five NYC boroughs.28,29 Detailed examination of infection rates (e.g., comparisons of demographic groups by borough or such smaller geographic areas as zip codes) is not yet possible, and rates cannot be calculated for populations that are not enumerated (e.g., men who have sex with men and IDUs). We do not know how the inadequate provider report form completion rates that translated into missing TTH might affect accuracy and precision of the incidence estimate. Until a more thorough evaluation of the impact of this data issue is conducted, incidence estimates should be interpreted with caution, especially when making future comparisons between years or jurisdictions.
Recent studies have shown that pooled-ribonucleic acid testing for acute HIV infection can be used for incidence surveillance.30 NYC DOHMH recently began pooled nucleic acid amplification test (p-NAAT) screening to routine HIV testing in city-run sexually transmitted disease clinics.31 Currently, pooled p-NAAT screening is only routinely conducted in this setting, making citywide incidence estimates with these results not possible. Further, one important challenge of this approach is that all HIV-negative specimens must be tested, rather than retesting HIV-positive specimens as in BED testing. In many settings currently, because of the widespread implementation of oral fluid and blood fingerstick-based rapid testing, no blood sample is taken on people with a negative rapid HIV test. This is one operational issue that makes the widespread implementation of this approach for incidence estimation more challenging than the current BED-based approach.
This evaluation had several limitations. It provided no information on sensitivity or positive predictive value of incidence surveillance, primarily because we did not have an external measure of the true frequency of recent infections in the population. We did not survey providers to assess barriers to reporting TTH; a survey might have provided useful information on ways to improve reporting. We also did not survey nonmanagerial clinical personnel at commercial laboratories, which might have provided additional insights into improving specimen salvage as well as overall acceptability of the system.
An approach to improving disease reporting by providers might be to make them more aware of both the public health benefit and their legal obligation to report, which in a previous study has been shown to improve reporting rates.25 Both, however, are stated on the report form. Another approach that has been successful in NYC is asking patients directly for this information during patient interviews. NYC has been expanding the number of newly diagnosed patients interviewed for partner notification purposes, and 92.7% of the completed interviews result in obtainment of TTH. Further expansion of such interviews from the 20% of newly diagnosed cases will likely improve overall TTH ascertainment.
In the interim, incidence surveillance should carefully characterize the demographic characteristics of diagnoses with complete information because imputation of TTH and BED values among the missing or untested population are based on these characteristics. Incidence surveillance should also ensure that time trends in incidence estimates and differences between local estimates and national or other jurisdictions do not result from missing testing and treatment information.
As a result of this evaluation, DOHMH personnel made clinical laboratorians more aware of their legal obligation to submit remnant samples, which has improved salvage rates. We also recommend that DOHMH and CDC work more closely with clinical laboratorians to improve their standardized protocol for remnant specimen packaging and shipping. Changes to routine surveillance are necessary to improve timeliness because the majority of time between diagnosis and receipt of BED test results is related to investigating new diagnoses and establishing eligibility for STARHS. Evaluation of routine surveillance should be conducted and is planned, but was beyond the scope of this evaluation.
Another possibility is to allow clinical laboratories to perform the BED test themselves, perhaps simultaneously with WB testing at the clinical laboratory, thereby eliminating the need to ship specimens and improving timeliness. This will require FDA approval of BED testing for clinical use, which reportedly has not occurred because of concerns regarding the validity of results on an individual level.5,21
Incidence surveillance provides a new method that avoids the logistical problems and expense of a longitudinal cohort study approach for calculating HIV incidence. As implemented in NYC, its strengths are in its integration with routine HIV surveillance and its ability to capture incident infection among the majority of the newly diagnosed population. It is hampered, however, by provider noncompliance with reporting requirements, a problem also affecting other surveillance systems. Continued improvements in data completeness and timeliness will improve the currently available information to inform personnel who develop HIV-prevention programs and policy initiatives in NYC and nationally.
The authors thank Drs. Lorna Thorpe and Thomas Matte at the New York City Department of Health and Mental Hygiene (NYC DOHMH) and Dr. Julie Magri at the Centers for Disease Control and Prevention (CDC) for their valuable comments on this article.
This study was supported through a cooperative agreement between the NYC DOHMH Human Immunodeficiency Virus Epidemiology and Field Services Program and CDC (U62/CCU223595).
The findings and conclusions in this article are those of the authors and do not necessarily represent the views of CDC.