In support of the Global HIV/AIDS Vaccine Enterprise (GHAVE), the Bill & Melinda Gates Foundation funded the Collaboration for AIDS Vaccine Discovery (CAVD), an international network of 17 Vaccine Discovery Consortia with five Central Service Facilities (CSF) that provide immunology and statistical support 
. As one of the CSF of the CAVD, the overall goal of the Comprehensive T Cell Vaccine Immune Monitoring Consortium (CTC-VIMC) is to provide standardized immunogenicity monitoring services in CAVD and GHAVE sponsored clinical trials of HIV vaccine candidates. To this end, the CTC-VIMC established a core of four cellular clinical immunogenicity testing laboratories, all of which are accredited to good clinical laboratory practice (GCLP) certification 
. Core laboratories include the International AIDS Vaccine Initiative (IAVI) Human Immunology Laboratory (London, UK), the Uganda Virus Research Institute (UVRI; Entebbe, Uganda), the HIV Vaccine Trials Network Laboratory (HVTN; Seattle, US) and NVITAL, core laboratory for the Vaccine Research Center (Gaithersburg, MD).
The Enzyme-linked immunosorbent spot (ELISpot) assay is a commonly used bioanalytical method for monitoring cellular immune responses in humans and animals. While being a relatively simple assay, the ELISpot has been shown to be highly specific, sensitive with good precision and stable over time 
. ELISpot assays were originally developed to enumerate B-cells secreting antigen-specific antibodies 
, and have since been widely used as a screening tool to assess the T- cell immunogenicity of, among others, candidate HIV vaccines 
. IFN-g secretion, as assessed by the ELISpot, occurs as a result of the recognition of cognate peptides or mitogenic stimuli by CD4 and/or CD8 T -cells. Secreted IFN-g is captured on IFN-g antibody-coated membranes and detected through subsequent recognition by further biotinylated IFN-g-specific antibodies, which in turn complex with streptavidin-conjugated enzymes that react with chromogenic substrates. The chromogenic reaction causes a spot to form where the reacting cells released their IFN-g; these spot forming units (SFUs) are then enumerated per number of stimulated Peripheral Blood Mononuclear Cells (PBMC). Typical stimulants used in such an assay are pools of overlapping synthetic peptides that correspond to sequences incorporated into vaccines. These pools consist of 8 to 15meric peptides overlapping in sequence to ensure maximal coverage of potential CD4 and CD8 epitopes.
Although the principal techniques underlying the assay remain constant, the use of differing SOPs for the ELISpot assay may result in variability of enumerated data between laboratories 
. Within the CTC-VIMC, both IAVI and UVRI core laboratories use the IAVI IFN-g ELISpot SOP, whereas the HVTN uses the HVTN IFN-g ELISpot SOP; both SOPs have been qualified in-house and across collaborating sites and are now routinely used to assess HIV vaccine candidates 
. Early plans for the CTC-VIMC were to utilize a commercially available ELISpot kit for all CAVD ELISpot tests. Unfortunately, concerns regarding reagent stability mitigated against use of these kits by the CAVD. Because significant time, effort and financial resources had been invested by IAVI and HVTN to qualify and propagate the use of SOPs across their respective laboratory networks, there was an understandable reluctance from either laboratory to use an alternative SOP when running specimens for the CAVD initiative. This study was therefore designed to generate sufficient statistical evidence to rigorously evaluate the results of the IAVI and HVTN SOPs for concordance and to guide the prospective comparison or pooling of ELISpot immunogenicity assessments across laboratories within the CAVD initiative. The CAVD Vaccine Immunology Statistical Center (VISC, Seattle, WA) assisted in the design of this comparison study and provided unbiased data management and analysis to assess this objective. The over-arching strategy for the CTC-VIMC assay comparison study, along with delineation of appropriate follow-up procedures (e.g., assay transfer and ongoing performance monitoring) is summarized in the flow chart presented in . Use of this systematic approach resulted in the adoption of an appropriate study design. With a common set of specimens and centrally prepared stimuli and controls, the findings from this evaluation have justified the continued use of the two different SOPs within the core laboratories of the CTC-VIMC.
CTC-VIMC/VISC ELISpot Standardization Methodology.
Because the ELISpot assay readout is often dichotomized into positive or negative responses based on pre-specified positivity criteria, this study focuses on the comparison of the two IFN-γ ELISpot SOPs with respect to the percentage of positive responses (i.e., response rates) from the tested samples and positivity call of each individual sample. Acceptance criteria on the equivalence of response rates and positivity calls were pre-specified prior to data analysis, thereby avoiding subjectivity of study results and inferred conclusions.
Furthermore, it was crucial to assess the false positive rates of the ELISpot assay, in addition to comparing the distribution of both background (i.e., responses from negative control wells with no antigen stimulation) and background-subtracted responses from antigen-stimulated wells. Responses to a Gag peptide pool from HIV-negative samples were used to assess the false positive rate of each assay. These assessments not only characterize the properties of the assay, but can also be used to identify sources of disagreement, if any, in the assay results.
To inform the design of the inter-laboratory comparison study, a pilot study using a small set of common samples and peptides was conducted at the IAVI and HVTN laboratories. Encouraged by evidence of concordance from data collected from the pilot study, the inter-laboratory comparison study was then designed and conducted with the appropriate sample size required to achieve statistical power in establishing equivalence between the dichotomized outcomes of the two assays.