|Home | About | Journals | Submit | Contact Us | Français|
The immunogenicity immunoassay validation process ensures development of a robust, reproducible method. However, no matter how well developed, validated, and maintained a method is, in the course of running a large number of samples over time, it is not uncommon to see bad reagents, poorly calibrated equipment, personnel errors, or other unknown and unpredictable factors that have an impact in the performance of the method and quality of the sample results. The immunogenicity immunoassay thus needs to be closely monitored with an internal statistical quality control process overtime to ensure a consistent and reliable output. The statistical process control has been widely applied to monitor manufacturing processes and in clinical laboratories. Its application to immunogenicity immunoassays is relatively novel. Limited guidance is available to implement the process to monitor semiquantitative immunogenicity immunoassay performance. Here, we have performed a suitability evaluation for process control charts with actual laboratory data from three immunogenicity immunoassay methods each utilizing a different technology platform. Additionally, a panel of prepared samples designed to assess long-term method performance were periodically evaluated for over a year. Finally, we make recommendations for an internal quality control process based on the results of these evaluations.
Analytical testing of clinical samples is a central part of the clinical development process. These clinical results are used for the establishment of label claims for the product. A good quality control process will help to ensure the validity of the test results. Additionally, no matter how solid the development, validation, and maintenance of an assay, some degree of natural variability is unavoidable. The assay is considered in statistical control when it is operating with such natural variability. Sometimes, sources such as pipetting error, a bad reagent, analyst error, etc. can lead to large variability and unacceptable results. In such cases, the assay is considered out of statistical control. A good quality control process will help identify these assays as out of trend (OOT). However, several quality control processes exist, and there is limited guidance available to implement a process to monitor semiquantitative immunogenicity immunoassay performance. Shewhart-type (1) and Levey–Jennings (2) control charts are common quality control tools to monitor performance of all types of processes. Westgard developed statistical multirules that optimized the detection of process error and minimized incorrect process rejection (3). These quality control processes have been widely used by clinical laboratories. Exponentially weighted moving average (EWMA) charts, which monitor a weighted average of the current observation and all previous observations (4), have also been evaluated for clinical laboratories (5,6). In diagnostic assays, quality assessment with respect to proficiency testing has been utilized as a means to determine the quality of the results generated by the laboratory in Clinical Laboratory Improvement Amendments laboratories (7).
Implementation of an appropriate control process to an immunogenicity immunoassay can be complex since the number and type of controls between methods can vary; in addition, there are many control charts and run rules from which to choose. Selection of a process control chart must minimize detection of false alarms and maximize the detection of true alarms. Excessive false alarms or failure to detect a true alarm quickly can lead to unnecessary process adjustment when an assay is perfectly in control or delayed process adjustment when it is needed, loss of confidence in the control charts as monitoring tools, and ultimately damaged productivity.
Three elements of quality control during every assay should include the following attributes. Each assay run must (1) follow a standard operating procedure, (2) include a set of controls, and (3) the controls for each test run must yield results within the limits of acceptability and validity of the run. Typically, each test has a set of positive and negative controls (internal controls) and may also include additional set of assay-specific controls (method performance controls). Internal controls are essential for quality control measures for each run and are intended for use only with the lot number of the corresponding test. Method performance controls can be included in a run to monitor consistent performance, lot-to-lot variation between kits, and to serve as an indicator of assay performance on samples that are borderline reactors.
We have used statistical approaches to determine the fit-for-purpose of these process control charts in the performance of validated immunogenicity immunoassays supporting clinical development of therapeutic proteins across various analytical platforms. Using predefined acceptance criteria, we have evaluated a robust statistical method and developed functional requirements for a Laboratory Information Management System tool which can be used for monitoring performance of immunogenicity immunoassays in real time.
A process control procedure which can identify different magnitudes of shifts as well as systematic shifts is ideal to monitor assay performance. X chart, which monitors an average of measurements (can be individual or multiple measurements for each sampling point), is a good candidate for detecting large shift, but it is not sensitive in detecting small to moderate shift. When there are multiple measurements per sampling point, X chart can theoretically detect small to moderate shift if the number of measurements per sampling point can be increased without much constraint; however, it is often not practical. Two commonly used approaches to increase the sensitivity of X chart in detecting small to moderate shift are supplementing the X chart with run rules and combining X chart with EWMA chart.
The average run length (ARL) to false alarm (average number of sampling intervals, the interval between two consecutive sampling points, it takes to detect an OOT when the process is in control) and ARL to detection (average number of sampling intervals it takes to detect an OOT when the process is not in control) are often the criteria used in evaluating process-monitoring procedures. A good procedure has long ARL to false detection and short ARL to true detection.
Supplementing X chart with run rules is not the preferred approach given the increased possibility of detecting false alarms (having an OOT signal when an assay is in control) and also the complication in applying multiple rules on a control chart. It is also well known that cumulative sum (CUSUM) chart and EWMA chart are good candidates for detecting systematic small shifts from the process mean. A commonly recommended approach is to combine X chart with either CUSUM or EWMA for the capability of detecting small to moderate shifts from process mean (4). We chose EWMA over CUSUM for the ease of application and interpretation and X chart with sample size of one to avoid within assay run correlation.
The criteria for an appropriate set of control charts were determined as follows: one false alarm in approximately 200 assay runs, detection of large shifts from the process mean within four assay runs, detection of moderate shifts from the process mean within five assay runs, and detection of small shifts from the process mean within 20 assay runs. The control charts to monitor the performance of immunogenicity immunoassays were evaluated using these predefined acceptance criteria.
Assay control data were generated by three validated semiquantitative immunogenicity immunoassays each using a different previously described detection technology—electrochemiluminescence (ECL) (8), ELISA (9), and surface plasmon resonance (SPR) (10). Each immunogenicity immunoassay was used in the routine analysis of clinical samples. Assay controls from each assay were first evaluated against method acceptance criteria. Assays with defined causes of failure (i.e., method deviations or high replicate percentage of coefficient of variation) were excluded from the control charts. All other assays were included in the control charts.
The assay controls were tested each time an assay was performed and were plotted on the control chart against time. Controls entered into control charts were grouped by assay runs. An assay run for this work included results related to the single performance of the immunogenicity immunoassay from start to finish. Often, this resulted in controls from multiple plates being grouped into one assay run. The mean result of each assay control for all plates in a run was calculated and entered as a single point into the control charts. Each assay run consisted of plates from the same lot to minimize within-run variability. Information of assay parameters which potentially contribute to the OOT such as critical reagent lot numbers, instrument number, and analyst were captured for each run in a companion spreadsheet for diagnostic purposes.
In order to identify when an assay was out of statistical control on a control chart, values of assay controls were evaluated relative to the control limits. The set of control limits define the bandwidth of the variation due to natural variability. The monitored immunoassay was out of control when an assay run was outside of the control limits or identified by EWMA chart or by run rules.
The control limits were generated using a collection of measurements from independent and normally distributed assays. An initial collection of data was analyzed, and the measurements that were outside control limits were excluded. A final collection of in-control measurements were used to calculate the new set of revised control limits to use for monitoring the immunoassay. Out-of-control signals were excluded from the control charts to satisfy the assumption that the measurements used to generate control chart limits are independent and identically distributed normal variants. Special statistical techniques, such as data transformation, need to be applied if there is severe deviation from either independence or normality.
Process control charts were evaluated for detection of a known change to a method. The SPR method data set included assay runs in which a reagent lot change was made that impacted the process mean. Process control charts were applied to the data set to observe sensitivity to changes made in the method.
Beyond process control chart analysis, we evaluated performance of assay runs with method performance controls that consisted of normal donors and antidrug antibody (ADA)-spiked samples that spanned the method detection range. The method performance controls were tested over a year to determine the variability of a SPR method. JMP version 7.0.2 software was used to generate process control charts.
All individual plate results from April 2007 to June 2008 for the ECL method were plotted in a Levey–Jennings chart (Fig. 1). The results within an assay run are more similar to each other than between assay runs. Since the variables that drive an immunoassay to be out-of-control normally change between assay runs, assay run seems to be a reasonable sampling interval. It is also a sensible choice in consideration of the operational practicality and convenience. As a result, the immunogenicity immunoassay methods were evaluated using the mean results of plate controls according to assay run so that each run can be determined to be in or out of control.
The two process-monitoring procedures were designed to (1) have similar ARL to false alarm, (2) have similar ARL to detection, and (3) meet the predefined functional requirements for the control charts. The control limits of X chart were determined to be the assay process mean ± 3 standard deviations. Based on the ARL to false detection and ARL to true detection as shown in Table I, the run rules for X chart were chosen to be 22s (signaling an alarm if a shift of more than 2 standard deviations is observed in two continuous sampling interval) and 41S (signaling an alarm if a shift of more than 1 standard deviation is observed in four continuous sampling intervals). EWMA chart monitors a weighted average of current observation and all previous observations. The weight was chosen as 0.25, and the control limits were calculated as the assay process mean ± 3 standard deviations of weighted averages.
The X + EWMA procedure, when applied to the SPR method (Fig. 2), indicated a total of four alarms. The X chart indicated three alarms for large shifts from the process mean. In addition, the EWMA chart indicated one systematic small to moderate shift from the process mean. For clarification, the first alarm of a series in EWMA chart informs that there is a systematic shift from the process mean. If it is not addressed, the shift is announced as an alarm until the process is back to normal. Therefore, the first alarm of a series is counted as a single alarm for the EWMA chart.
The X + Run Rules procedure, when applied to the SPR method (Fig. 3), indicated 12 alarms. Small systematic shifts were identified by five 41s alarms, moderate systematic shifts were detected by four 22s alarms, and large shifts were captured by three 13s alarms. Although there were discrepancies in the number of alarms, the X + EWMA and X + Run Rule procedures identified alarms in the same assay runs. The two procedures were also applied to the ECL and ELISA methods, and although the process control charts are not shown, the number of alarms indicated between the two methods was consistent (Table II).
The two process control procedures were evaluated to detect a known change to an immunogenicity immunoassay. Both process control procedures detected a biosensor chip lot change to the SPR method in the data set. The change, known to occur in run 68 (June 12, 2007), was first signaled in the X + EWMA and X + Run Rule procedures at run 68 by the X chart. Both process control procedures continued to announce the change in subsequent runs until the chip lot was replaced.
In consideration of the impact of using different number of runs to calculate the control limits on the X chart and EWMA chart, control limits were calculated using the first 20, 25, and 30 runs. The number of detections was recorded, only for comparison purposes since it was not known whether detections were true or false, for both process monitoring procedures when applied to different immunogenicity immunoassay methods: ECL, ELISA, and SPR (Table II). There was consistency in the number and type of alarms between X + EWMA and the X + Run Rules procedure for the ECL and ELISA methods. There were differences in the number of alarms among the two procedures in the SPR method; however, closer examination of the charts (Figs. 2 and and3)3) revealed that the assay runs being alarmed were consistent for both procedures.
Results of the method performance controls are illustrated in Fig. 4. All replicates of subjects not spiked with ADA were found to be negative and were similar in variability range (~14 RU). The lowest value for subject 10408 (known to be spiked with ADA) was determined to be falsely negative; however, all assay controls for the assay run passed method acceptance criteria and did not generate an OOT alarm using either process control procedure (Fig. 5). Upon repeat analysis, the misdiagnosed sample was correctly determined to be positive. Again all assay controls for the repeat analysis passed method criteria, and the assay was in control. All other replicates of subject 10408 and all other spiked ADA subjects were consistently determined to be positive. Subjects 10406 and 10415 were both spiked at the same ADA concentration; however, the lowest point of subject 10415 caused the variability to be approximately twice that of subject 10406. Such deviations in variability would be cause for sample reanalysis now that the normal method variability is understood.
An appropriate internal quality control process to monitor semiquantitative immunogenicity immunoassays can help ensure consistent and reliable assay performance. The process should be able to detect assay drift over time to avoid potential sources of bias in determining positive and negative values. An immunogenicity immunoassay that has drifted downward has the potential for decreased sensitivity for determining a true positive, and conversely, if the method has drifted upward, it is more likely to get a false positive result.
We have utilized a systematic statistical approach to evaluate the performance of process control charts in accurate assignment of OOT alarms to the performance of immunogenicity immunoassays. We have demonstrated that the two process monitoring procedures, X chart in combination with EWMA charts and X chart with carefully selected run rules, are capable of detecting small, moderate, and large shifts from the process mean, and made a recommendation for use of such procedures to monitor the performance of immunogenicity immunoassays.
To date, there has been no systematic evaluation of the application of process control charts in monitoring the performance of semiquantitative immunogenicity immunoassay methods. Using a large body of data from assay controls in three different immunogenicity immunoassays, we have conducted an analysis of the ability of the process control charts to identify deviation of the assay from an in-control state. All analysis indicates that performances of the two procedures are comparable as expected. Furthermore, both charts detected a known assay reagent change equivalently. Our suggestion is to use X chart together with EWMA chart as its capability in detecting process shift of small to large scale is well studied and documented, and also, the combined procedure is easy to apply. The weight and the multiple of standard deviations of EWMA chart can also be chosen to achieve the ARL to false alarm and the ARL to detection the management would like to have. The X chart with multiple run rules can be used if the run rules can be selected such that the ARL to false alarm and ARL to detection are acceptable, and also, the pattern identified by selected run rules are characteristic of the assay when it is in an out-of-control state.
The assay runs used to calculate control limits should (1) be independent, (2) not deviate from a Gaussian distribution, and (3) numerically be large enough to capture the inherent variability of the assay. Industry guidelines recommend 20 experiments to evaluate precision (11). The data in this analysis indicate that the accumulation of approximately 30 runs may be required to calculate the control limits. If any out-of-control points need to be excluded to achieve independence and normality, 20 to 30 runs should still be available to determine the control limits.
Limits calculated without taking into account the within-run correlation can lead to an underestimation of the method variability within an assay run and consequently result in high frequency of false OOT alarms. The strong within-run correlation observed in our immunogenicity immunoassays has been observed in other clinical labs and was attributed to the nonstationary and nonergodic nature of analytical processes in the clinical laboratory (12). As stated, our approach was to chart the mean of all within-run plates against assay run. There are more statistically driven ways to deal with the correlation. For example, a time series model can be used to remove the correlation, and then, one can apply control charts to the residuals from the model. Our approach was chosen for its simplicity since it suitably removed the correlation effect and is reasonably sensitive in detecting out of control assay runs.
Another consideration of appropriate control limit calculation is whether to chart assay runs with control values that fail acceptance criteria described in the immunogenicity immunoassay. The specific criteria for a positive control in semiquantitative immunogenicity assays is to be above the assay cut point. These conditions make it challenging to set lower control chart limits which could be below the assay cut point (i.e., failed). Assay runs that fail due to a known reason such as a method deviation should not be included in control charts since the assay controls may be influenced by the deviation. However, if the assay controls fail for unknown reasons, it cannot be accurately determined if the event is a part of the variability of the assay or if the assay is out of control. We make recommendations that assay controls that fail for unknown reasons should be included in control charts for the purposes of control limit calculation so long as the controls maintain the normal distribution and independence requirements. Furthermore, we make recommendations for the inclusion of assay controls from runs that fail for unknown reasons after control limits are determined so that all OOT events are detected.
Monitoring performance of a semiquantitative immunogenicity immunoassay process relies heavily on manual supervision of assay parameters. Development of an electronic tool in a laboratory information management system or a spreadsheet should provide the users to automate the internal quality control process. Application of process control charts to monitor the performance of assay across time using positive and/or negative controls ensures robust assay performance over time. In addition, although validation of immunogenicity immunoassays provides a thorough understanding of the assay variability and the sources that contribute to this variability, long-term performance of an immunogenicity immunoassay can be assessed by control chart analysis and method performance controls. Figure 6 illustrates a flow chart of an internal quality control process that can be implemented in the laboratory.
The OOT alarms identified by process control procedures can be minimized if reagents are qualified for use in the method prior to analysis of clinical samples. Critical reagents should be qualified by at least one independent process and assay runs that consist of controls and a sample set spiked at the lower limit of reliable determination. For example, a biotin-conjugated reagent can be independently qualified using biophysical methods to determine specific reagent characteristic profiles before testing it in assay qualification runs.
In addition to assay controls, method performance controls to periodically evaluate assay performance can be a valuable supplement to a quality control process. Semiquantitative immunogenicity immunoassays utilize positive and negative controls. These controls are typically either single-serum samples or pooled samples from multiple donors. Although these controls provide necessary guidance on assay performance, one major contributor to variability of the immunogenicity immunoassays is subject-to-subject diversity. A panel of method performance samples can provide further information about method variability over extended periods of time and can potentially indicate deviations not captured by assay controls or process control charts. Furthermore, the sample panel can be used as a tool to assess impact for methods that have gone OOT.
The ultimate goal of an internal quality control process is to ensure data integrity through long-term monitoring of the assay performance. Upon identification of an OOT event, a plan to investigate the root cause of the assay drift should be established quickly. The plan should include, but not be limited to, review of reagent qualifications, potential impact of critical reagents, equipment, and environmental conditions. In several instances, analysis of raw signal (i.e., OD or ECL units) should be evaluated, since normalization against background can influence bias.
Antibody detection assays are routinely validated over a limited time frame which does not consider any long-term variation of the method. We have developed a statistical process for monitoring semiquantitative immunogenicity immunoassays over time that can be a valuable supplement to the validation data. The X + EWMA and X + Run Rule procedures provide appropriate tools to evaluate the performance of assay controls. Implementation of internal quality control and investigation processes will ensure long-term performance and will be valuable for regulatory agencies to evaluate impact of clinical parameters which may be dependent of results from these methods.
We thank PPD and the scientists at Amgen (Suzanna Tatarewicz, Geoff Houghton, Dohan Weeraratne, and Andrew Kuck) for performing the assays. This work was done at Amgen, and all authors hold company stock.