While the ultimate hope of MND biomarker investigations is to impact patient care, any biomarker must overcome a series of hurdles in order to obtain clinical practice utility. One important question to keep in mind during biomarker assay development is how the assay will be used in the clinic and how it compares to the current diagnostic standard. This is more problematic for MNDs that currently have no biochemical diagnostic test. For MND diagnostic tests, clinicians certainly desire a very low false-positive rate for a diagnostic biomarker given the implications of an ALS diagnosis. Therefore, a useful diagnostic biomarker assay must have high specificity, and clinicians would likely sacrifice some sensitivity for it. As more effective treatments become available, this balance could shift.
Given the prolific pace of protein biomarker discoveries, there is a need for regulatory oversight of analytic validation and qualification data to translate biomarkers to the clinic. Prognostic and predictive tests could identify patient populations that benefit from treatment, or equally important, those who would respond poorly to a specific treatment. As described below, the clinical utility of a protein biomarker-based diagnostic requires testing in prospective randomized clinical trial designs and subsequent validation in follow-up clinical trials. This is quite similar to the traditional regulatory approval pathway for therapeutic agents.
Four phases of clinical biomarker assay development for diagnostics can be conceptualized (). These include the exploratory, discovery phase (I), the clinical assay development and verification phase (II), the retrospective validation phase (III), and the prospective screening and qualification phase (IV). The initial discovery phase typically uses retrospective samples to discover potential biomarkers. Often samples used for biomarker discovery experiments are collected at one site, although the use of samples obtained from multiple sites during this early phase may be beneficial for the next clinical assay development phase. This initial phase may also include a separate test group of samples to confirm the initial biomarker discovery results from the training set, but this is rare. The second phase further modifies the assay and verifies the protein so that it can be used in continued candidate biomarker validation studies. This phase may include optimizing antibodies used for protein detection, using another proteomic technology that increases sensitivity or specificity of the assay, or modifying the sample processing sets to enrich for the biomarker of interest. During phase II, one must establish assay performance parameters and generate assay reference standard calibrators, possibly a recombinant form of the biomarker or synthetic peptide recognized by antibodies used in the assay. The final assay plan should include standard calibrators, quality control (QC) samples, pooled sample controls from actual samples, and the true test samples. The candidate biomarker protein must be extensively verified with respect to tissue and cell type expression, physiologic functions, and potential biologic mechanisms related to the MND of interest. Thus, the verification stage will likely include multiple independent studies and may occur over multiple years. It is also imperative that standard operating procedures be described for the collection and storage of samples for use in all future studies. Assay development costs associated with verification studies can be extensive and therefore limit the total number of candidate biomarkers that will successfully complete this phase of development.
FIGURE 1 Biomarker assay development phases from discovery to validation and qualification. The flow diagram depicts the process of biomarker discovery to regulatory approval as a diagnostic or surrogate biomarker. Few biomarkers discovered will ultimately pass (more ...)
The third phase (III) involves large validation studies typically that use retrospective samples. A small confirmatory study of the final assay using additional samples from a single site may be performed prior to the large validation study. During this large validation study, it is necessary to include samples from multiple sites to determine the assay variability induced by sample collection and storage procedures. Proper power analysis must be performed to design a research protocol that will obtain statistically significant results concerning the sensitivity and specificity of the assay with large numbers of disease mimics. Many hundreds to thousands of total samples will be required to complete this phase of the biomarker assay development.36
Appropriate disease mimics include polyradiculopathy, myeloradiculopathy, multifocal motor neuropathy, other motor neuropathies, focal axon-loss motor predominant mononeuropathies, the “spinal form” of multiple sclerosis, myelopathy, and possibly inclusion body myositis. Other MND subtypes, such as SMA, Kennedy's syndrome, and HSP, should also be included.
Whether it is important to include a broad range of ALS presentations and phenotypes is not certain, but it seems prudent to stratify the analyses based at least on features that vary among genetic forms of ALS. Although there is considerable overlap among sporadic ALS and genetic forms, especially ALS 1, 6, 7, and X,37
there are noteworthy outliers. The best examples are in ALS 1, in which the A4V SOD1 mutation is associated with short survival of 12–18 months and limited upper motor neuron involvement,37
while the H46R mutation is associated with an 18-year life expectancy.38
There are also three juvenile onset forms, ALS 2, 4, and 5, that tend to progress slowly. Interestingly, ALS 2 (ALSIN) causes primary upper motor neuron or upper and lower motor neuron ALS. In contrast, dynactin mutations result in early adult onset, slowly progressive motor neuron disease with vocal cord paralysis as a primary manifestation.37
The location of involvement at onset also appears to be a distinctive characteristic of a newly discovered mutation in the fused in sarcoma/translated in liposarcoma (FUS/TLS) gene. In one kindred, affected patients presented with proximal upper extremity weakness without bulbar involvement39
; in another, cervical was twice as common as lumbar onset, and bulbar onset was uncommon.40
There are also examples of unexplained phenotypic variability from the same mutation as seen in vesicle-associated membrane protein (synaptobrevin-associated protein) B (VAPB) mutations. They range from typical ALS to late-onset spinal muscular atrophy and slowly progressive ALS with tremor, all from the same mutations. Finally, there are inherited forms of ALS with frontotemporal dementia and parkinsonism.37,38,41
At least partly based on the above variations, it would seem useful to stratify patients based on age of onset, rate of progression, type of motor neuron involvement (upper or lower, or both), associated features such as dementia and extrapyramidal signs, and site of initial involvement (bulbar vs. upper or lower limb in proximal or distal distributions).
However, any subgroup analysis will require sufficient numbers in each ALS subgroup to power the study and generate statistically significant results. The number of patients will determine how refined the stratifications may be and will limit the sensitivity and specificity capabilities of the testing conditions. Eventually, inherited as well as sporadic forms require subgroup analyses. The phenotypic variations that occur in the inherited forms could be due to environmental factors, and biomarker studies could provide insight into their origins.
Careful consideration of inclusion/exclusion criteria is necessary to properly address validation questions regarding the biomarker. Additional prospective validation studies are required when new standard operating procedures are incorporated into the biomarker assay development process. One must also remember that validation methods may differ for diagnostic (distinguish disease from controls) and prognostic (predict clinical outcome) MND biomarkers versus biomarkers for drug development (safety, efficacy, and exposure-effect relationships).42
The final phase (IV) will apply the diagnostic biomarker assay in a prospective manner across multiple sites to monitor the predictive value of the test when compared to the final clinical diagnosis for each subject. This is the phase in which the ability of the biomarker assay to impact patient care is answered and actively qualified as a biomarker-based test for the clinical indication.43
Long-term outcome measures are required to determine the final sensitivity and specificity of the assay. Substantial costs and time will be required to complete a biomarker validation study. This phase may also include incorporation of the biomarker assay into clinical trials to evaluate its use as a surrogate marker of drug efficacy. To determine the biomarker's utility as a surrogate marker, the natural history of the biomarker has to be assessed in controls via serial analyses. Surrogate markers to monitor disease progression are not equivalent to diagnostic biomarkers; therefore, surrogate markers of disease progression may not have diagnostic utility. When incorporated into drug development-based clinical trials, biomarkers should be an exploratory substudy within the clinical trial and not a component of the regulatory submission for the drug. However, the results may provide critical validation data such that the biomarker could be used as a useful endpoint in future registration studies for the therapeutic agent. Again, biomarkers of drug efficacy may not have diagnostic utility and should be selected based on the proposed mechanism of action of the drug candidate.
Thus far, protein biomarker studies for MND remain predominately in the discovery phase of the development process. One common problem with the published biomarker studies is that they are generally underpowered, use limited numbers of MND patients, and typically lack the appropriate disease controls. Very few have used greater than 100 total subjects during biomarker discovery experiments.11,17,27,44
These studies are all retrospective in nature, using subjects with clinically defined MND and control subjects. A few words of caution are also in order regarding the methods used for data analysis when discovering and validating biomarkers. In general, multivariate statistics and machine-learning algorithms are prone to experimental overfitting and generation of false-positive biomarkers.45
Multivariate analysis may perform well on the original sample set (i.e., the training set) but often fail when applied to an independent test set. Therefore, careful biomarker validation studies must be formulated, especially for panels of candidate biomarkers.
While most acknowledge that standard operating procedures for the collection and storage of clinical samples are required for clinical proteomic biomarker discovery and validation efforts,46
few individual studies have carefully defined such standard operating procedures.35,47
Additional standard operating procedures must be developed for any sample processing step necessary to enrich the biomarker prior to analysis as well as the methods for data capture, normalization, and statistical analysis. The Human Proteome Organization (HUPO) has developed an HUPO-Proteomics Standard Initiative and recently published guidelines for standardizing the collection, analysis, and reporting of proteomics data.48
Careful definition of protocols and procedures early in the biomarker development process will not only reduce the chance of generating false-positive biomarkers in the discovery phase, but it will also enhance all validation studies and will ultimately be required for regulatory approval.
The task of taking biomarker discovery results and generating a validated and accepted clinical assay requires extensive experimentation that must be carefully formulated and documented. There are two general paths to get a diagnostic assay to the clinic in the United States.49
The first is FDA approval of the diagnostic test, whereas the second is via the Center for Medicare & Medicaid Services (CMS) that regulates all laboratory testing though the Clinical Laboratory Improvement Amendments (CLIA). The FDA is charged with regulating all in vitro diagnostic devices (IVDs) to ensure safety and effectiveness as defined in the Code of Federal Regulations. Safety and effectiveness refers to the consequences of reliance on the IVD to make clinically significant diagnostic or clinical care decisions. Unless the candidate biomarker is already established to detect disease or monitor therapeutic efficacy, all biomarkers will require extensive clinical studies to support a claim of effectiveness and safety before receiving FDA approval. Meetings with the FDA via a preinvestigational device exemption (pre-IDE) are required prior to the initiation of trials designed to evaluate a new IVD. In addition to this expensive clinical validation process, thorough documentation and validation of all reagents and equipment used in the IVD is required by the FDA. Therefore, considerable time and collaborative efforts will be required to generate an FDA-approved diagnostic test for an MND.
A less cumbersome path for translation from biomarker discovery to the clinic is the CLIA approved in-house or “home brew” test assay for “research only” purposes. The laboratory developed test (LDT) is generated within the CLIA-approved clinical laboratory using analyte-specific reagents (ASR) that typically are antibodies manufactured elsewhere and used in the LDT. The FDA exempted most ASRs from 510(k) clearance requirements and does not regulate the in-house laboratory tests that are generated from these ASRs. It is critical to adhere to FDA guidelines describing ASRs so that the final assay conforms to all federal regulations (CLIA, 42 U.S.C. 263a 62 FR 62252). CMS oversight with state regulatory agencies ensures that biomarker assays generated with ASRs are accurate, safe, and available for use while their potential clinical utility is being examined before submission to the FDA for approval.
Regardless of the pathway used to translate the biomarker discovery efforts to the clinic, large validation studies are required to properly test the diagnostic accuracy of the proposed assay. As noted above, no proper validation studies have been performed to date for MND diagnostic assays. By contrast, clinical trials for MND therapies typically include hundreds to thousands of subjects, and the results are often replicated in separate studies prior to approval of the therapy. Ideally, diagnostic biomarker assays should also be validated in analogous prospective, well-controlled clinical studies of diverse patients across multiple institutions. Standard operating procedures for sample collection, processing, storage, and shipping must be incorporated in these clinical studies.