|Home | About | Journals | Submit | Contact Us | Français|
Communication between research laboratories within a given field is often an important key to rapid successes within that field. We propose that consensus standards may be a useful tool to help facilitate such communication by providing a “common language” for laboratories that utilize similar methodologies within a field. The existence of consensus standards is well known in other fields, and through this commentary we hope to (i) introduce the concept of consensus standards to investigators in the field of microbial pathogenesis/host response who may not be familiar with them and (ii) provoke thought and discussion by others in the field regarding the possible usefulness of additional consensus standards for their own work.
A consensus standard is a published collection of standardized nomenclature, descriptions, assays, and/or methodologies, with the distinguishing feature being that the recommendations are not the product of a single researcher, laboratory, or institution, but instead represent a collective experience and expertise that lead to the refinement and consolidation of multiple methods/protocols. Use of a consensus standard is not mandated or required, but instead is intended to serve as a guide or tool for investigators. A consensus standard makes available useful information if, for example, an investigator wishes to directly compare data generated in their laboratory to those generated in other laboratories. Consensus standards can appear under many guises in the literature and can be somewhat difficult to locate in the course of a routine search. A Pubmed or Web of Science search using key words such as “consensus standard” or “consensus” or “standard” or “interlaboratory” or “harmonization” plus another key term relevant to the area of interest can help locate useful literature citations. Some consensus standards are published as a more formal document that has undergone a predetermined series of oversight steps and public review process laid out by the institution that initiated or sponsored the call for the consensus standard; some examples of these institutions in the United States include the American National Standards Institute (ANSI), the American Type Culture Collection Standards Development Organization (ATCC-SDO), and the National Institute of Standards and Technology (NIST), U.S. Department of Commerce. Thus, it is important for anyone searching for a consensus standard to not only perform a search using their usual literature database, but also search the websites of the organizations that generate standards. The consensus standard on anthrax toxin in vitro activities discussed in more detail later in this commentary was the product of a formal call for a consensus standard by ATCC-SDO.
Successful and well-known examples of grassroots, community-led, volunteer consensus standards have arisen following the introduction of “omics” technologies in the life sciences. A prime example of science-driven standardization is the Gene Ontology Consortium (2, 15). This consortium was formed by leaders in the fields of Drosophila, Saccharomyces, and mouse genetics with a goal to systematically describe genes from a functional standpoint in order to facilitate comparative analysis of their research across various model systems (2, 7, 8, 15, 23). In fact, there has been such a remarkable proliferation of community-driven international data standardization activities related to the “omics” fields that an entire issue of the journal “OMICS” was dedicated to data standards, including those for genomics, proteomics, metabolomics, transcriptomics, flow cytometry, and phylogenetic analyses (8, 13). Other pertinent examples of experimental method standardization include those for the derivation of cell lines (27). The field of microbial pathogenesis benefits directly from these consensus standards, given the interdisciplinary nature of this field and its use of these tools and resources.
There are a number of specific and important examples in the area of microbial pathogenesis, where a need for standardization has been recognized due to variability in data obtained by different groups. In some cases, the need has been addressed by (i) development of standard protocols (18, 20, 28); (ii) endorsement of a standard reference material for use in assay comparisons (4, 28); (iii) interlaboratory comparisons of existing or new protocols (18, 20, 21); (iv) evaluation of factors that contribute to assay variability (19); and/or (v) addressing the needs for standardized assays through international conferences and/or committees (29). Botulinum neurotoxin type A (BoNT/A) is a good example given the exponential increase in the use of this toxin over the past 2 decades for basic research as well as for clinical purposes. The “gold standard” assay for estimating the potency of BoNT/A is the mouse 50% lethal dose assay that provides the in vivo toxicity of a given BoNT/A sample; however, this bioassay is inherently variable (19, 26). Factors that affect the potency of BoNT/A activity include differences in toxin formulations, choice of diluent, choice of mouse strain for the bioassay, and the assay design (19). The results of an international collaborative study underscored the importance of using standard reference material in the BoNT/A bioassay to reduce interlaboratory variations in toxin potency data (26). Another example arises from a recognized need for standardization of Bordetella pertussis assays, which prompted the formation of an international conference to discuss the harmonization of immunoassays for pertussis diagnostics and vaccine evaluation (29). Yet another example of the need for consensus standards in the study of microbial pathogenesis stems from the increasing global incidence of methicillin (meticillin)-resistant Staphylococcus aureus and the need to type the strains in a reliable and reproducible fashion in different laboratories in widespread geographic locations for epidemiologic purposes (28). Accordingly, a group of investigators in Europe recognized the need to standardize the pulsed-field gel electrophoresis protocols for the molecular typing of strains of methicillin-resistant Staphylococcus aureus and worked together to develop by consensus a single approach that was subsequently evaluated and validated in 10 European laboratories (20, 28). Similarly, the lack of consensus standards for PCR-based identification of other microbes provides an example of a specific need for development of standard PCR primers and assay design. For example, the lack of standardized PCR primers in addition to different nonvalidated assays for identification of Chlamydia pneumoniae in vascular tissue and other clinical specimens has been responsible for some of the controversies in ascribing a role for this pathogen in human disease (4). These are but a few examples extracted from the literature to highlight the potential need for standardized protocols, especially when generating critically important reagents for experiments or when even subtle variations in study design may be enough to create significant variability in results within or between laboratories.
We saw a need for a consensus standard within the anthrax field because of the marked increase in research on Bacillus anthracis and its toxins over the past several years. Approximately a decade ago, the field of study of B. anthracis toxins began to expand as the threat of terrorist or criminal use of pathogenic microorganisms became of heightened concern both in the United States and internationally. Dissemination of B. anthracis spores through the U.S. postal system in 2001 further underscored both the potential for and ramifications of the use of this microorganism as a bioweapon. This event served to illuminate gaps in our knowledge on the pathogenesis of anthrax and quickly led to additional efforts to elucidate mechanisms of virulence of B. anthracis and to develop new strategies for prophylaxis and treatment of infection. These efforts resulted in an influx of investigators into the field, followed by an increase in the number of publications on B. anthracis and its virulence factors, notably the two bipartite toxins known as lethal toxin (LT) and edema toxin (ET) (reviewed in reference 30). To illustrate this increase, a Web of Science citation search by topic using the terms “anthrax” and “toxin” (or “anthrax toxin”) revealed that between the years 1970 and 2001, only 260 papers (ca. 8 papers/year) were published on B. anthracis toxins. In contrast, since 2001, there have been 914 papers published (ca. 130 papers/year). Overall, this represents a 16-fold increase in the number of publications/year involving B. anthracis toxins.
This increase in research activity on the toxins of B. anthracis has been associated with a concomitant increase in the number of research laboratories using these toxins. The anthrax toxins are each comprised of two separate components, a binding protein and a catalytically active enzyme, that bind to the target cell in a 7:3 ratio of the toxin components, respectively. LT consists of protective antigen (PA) that binds to cell surface receptors and lethal factor (LF), a zinc metalloprotease that is active in the cytosol. ET is comprised of PA combined with edema factor (EF), an adenylate cyclase that is active in the cytosol (reviewed in reference 30). The source of the toxin proteins used in research studies has ranged from private (individual laboratories) to commercial sources of either native or recombinant (B. anthracis-derived or Escherichia coli-derived) PA, LF, and EF (e.g., see references 3, 6, 10, 14, and 34). The quality of the toxin components from any given source has proved difficult to evaluate in individual laboratories because of the absence of standardized assays to use for the comparisons. Some of the critical variables that can impact experimental outcomes include the quality of the toxin preparation, presence of bacterial cell wall contaminants, choice of host cell type and handling of the cells, toxin concentrations, ratio of toxin components, duration of exposure of cells to the toxins, and choice of in vitro assay system and its readout measurement. For example, in two separate published studies of microarray analyses of cellular gene expression responses to LT, the PA and LF were used in different ratios and concentrations with a different duration of toxin exposure to RAW264.7 macrophages. In those studies, different gene profiles were reported (3, 10). Another study reported distinct cell type-specific protein expression profiles for RAW264.7 macrophages and J774A.1 macrophages exposed to LT, under the same experimental conditions and in the same laboratory (25). Another potentially confounding factor is that PA binding to host cells can be variable and dependent on cell type, cell differentiation, and which receptors for PA are present or predominantly expressed on the host cell membrane (9, 12, 24). Thus, because individual investigators have used a broad range of toxin concentrations (ng/ml to μg/ml), a variety of PA/LF or PA/EF ratios ranging from 0.5:1 up to 1,000:1, and various cell lines and exposure times of the cells to the toxin components for in vitro assays to assess toxin action (e.g., see references 3, 6, 10, and 31), it can be difficult to assess how the toxin activity compares to other toxin preparations that another investigator may use in their laboratory. We do not mean to imply that data generated from an individual laboratory are incorrect or that the design and conduct of laboratory experiments involving toxins should be dictated in any way; instead, we propose that the availability of protocols that investigators can utilize in order to compare toxin activities or address troubleshooting issues regarding toxin activities provides a valuable resource to the field.
The authors of this commentary are members of a volunteer work group of investigators assembled under the auspices of ATCC-SDO. Our work group was formed with the purpose of addressing the above-mentioned issue of variability and comparability related to use of the toxins of B. anthracis by generating a consensus standard entitled “Standardization of In Vitro Assays to Determine Anthrax Toxin Activities.” This volunteer work group is comprised of investigators in the fields of biodefense, B. anthracis, and bacterial toxins, representing academia, government, and industry. The goal of the work group was to write a consensus standard describing specific assay methodologies and protocols for the characterization of B. anthracis toxin components and assessment of their activities, in order to provide a compilation of protocols to be used as an optional tool for investigators. The standardized protocols include sections on individual toxin components (PA, LF, and EF) as well as the bipartite toxins, LT and ET. For individual toxin components, the consensus standard focuses on (i) determination of purity and quantity of each toxin component; (ii) in vitro measurement of the enzymatic activity of LF or EF; and (iii) appropriate storage and handling conditions. For the bipartite toxins, LT or ET, the consensus standard focuses on (i) cell culture and handling, which are critical aspects to optimizing the in vitro cell-based toxin activity determination; (ii) determination of optimal toxin activities and ratios (e.g., PA/LF ratio), using a checkerboard assay design for in vitro assays; and (iii) recommended assays to determine in vitro cell-based activities of LT and ET. Consensus recommendations for the determination of optimal PA/LF ratios were perhaps one of the most difficult to address, since such a broad range of toxin ratios has been used in published methods, as noted above. The problem was compounded by the absence of an available reference standard, which represents yet another need in the anthrax field. Thus, after much discussion and review, the work group reached consensus by recommending that investigators test each lot of their PA and LF toxin components by using a straightforward checkerboard assay design in which the PA or LF concentration is varied, while the concentration of the other component (PA or LF) is kept constant at an excess concentration, and that a murine macrophage cytotoxicity assay be used to quantify the lethal effect of the PA and LF. In this way, the optimal concentration for PA and for LF can be determined to achieve 90% cytotoxicity of the cells. Thus, in the consensus standard, no fixed ratio was recommended; however, the approach allows for an assessment of the quality of individual lots of toxin components.
ATCC-SDO consensus standards, such as this one, are recognized by ANSI and are compatible with International Standards Organization (ISO) guidelines for standards development. The draft consensus standard was reviewed initially by a committee of investigators assembled by ATCC-SDO and then during a 45-day public comment period. Importantly, in addition to the formal public peer review process, this consensus standard will remain a “living document,” subject to revisions over time that may be recommended by other investigators to reflect changes in the field and new experimental methodologies. Investigators in the field are encouraged to contribute to future revisions of the standard.
Some researchers may object to the development of consensus standards since they may perceive that a standardized approach may hinder individual thought and design of experiments and that a small group of scientists could be viewed as imposing a particular experimental design among a larger group of scientists. Some investigators may argue that it is the uniqueness of the experimental design and their creativity that allow for progress in a field and that standard protocols remove the freedom of creative thought. One could also raise the question about how data should be interpreted when they are generated from experiments performed without these standards. We would respond by pointing out that there are already examples, many cited above, that illustrate how groups of scientists have come together to adopt, by choice, a consensus approach to achieve significantly improved interlaboratory reproducibility. Importantly, voluntary use of standard protocols by some groups should not be to the detriment of other groups undergoing peer review of their work that do not follow the same protocols; a consensus standard is not meant to stifle research but to facilitate progress for those who choose to use it. A consensus standard could be especially useful to investigators new to the field who are trying to sort through an array of published protocols to determine and bring together the essentials of experimental design for setting up a new and unfamiliar assay. A consensus standard can also aid in troubleshooting toxin activity data since many factors can affect each assay and potentially alter reproducibility of results. In all of these ways, a living consensus document can provide a platform to be used by investigators to promote discussion and catalyze further revisions and improvements, especially if investigators participate and contribute to future updates of the standard. A consensus standard can elicit changes (by the investigators themselves) that promote progress, productivity, and generation of meaningful results that can be compared with those from other laboratories. This latter point is especially important considering that efficiency, reliability, and reproducibility in science are ever more critical factors when funding and resources are limited.
It should be noted that a wide variety of standards exist in the public and private domains that have been generated and disseminated for use within a particular organization or that target industry, regulatory, or clinical laboratories (e.g., see references 22 and 26) to serve as standard operating procedures/protocols for that laboratory. Additionally, many clinical consensus-based recommendations have been published, including consensus statements for physicians on the management of anthrax (16), plague (17), tularemia (11), viral hemorrhagic fever (5), and botulism (1) resulting from the potential use of these agents as biological weapons. However, consensus standards targeted specifically to the basic researcher are relatively less common, although some examples have been provided above. To our knowledge, the anthrax toxin consensus standard that we have discussed in this commentary represents a first of its type in the field of B. anthracis research and may be unique to the larger field of microbial toxins and pathogenesis. The major goal was to provide a set of standardized protocols generated by consensus of a group of investigators in the field that are focused on the needs of basic researchers and which would facilitate the direct comparison of data generated in different research laboratories.
We propose that the development of consensus standards is a powerful option available to the research community that will foster reproducibility and comparability of results generated in different laboratories, thereby facilitating communication and, ultimately, progress in the field. In this commentary, we have discussed a consensus standard that has been developed for B. anthracis toxins as one example. We hope that investigators will consider and debate the use of consensus standards as an effective strategy by which members of the research community can come together to potentially accelerate progress within their own field.
We thank Liz Kerrigan (ATCC-SDO), Christine Alston-Roberts (ATCC-SDO), Trish Hugunin (ATCC-SDO), and Steven Permison (Standards Based Programs, Inc.) for their dedication in assisting and advising the members of the ASN-0001 Consensus Standards Workgroup.
Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the U.S. Army.
The views expressed in this Commentary do not necessarily reflect the views of the journal or of ASM.
Editor: S. R. Blanke
Published ahead of print on 3 August 2009.