This study evaluated non-risk adjusted and risk adjusted statistical process control methods for detecting elevated adverse event rates among randomized controlled trial data. Both SPC and LR-SPC performed well and were comparable to each other. However, BUS was significantly over-specific and did not alert in any month in either trial.
The proportional difference test was validated by comparison with Fisher’s exact test and the results were concordant in both trials analyzed. The SPC method alerted properly in each of the months identified by Fisher’s method for the Trial A data, and did alert in 4 months in the Trial B data in which the Fisher’s method did not find a significant difference. The LR-SPC method alerted appropriately with the exception of one false positive in Trial A (for the 0.20 model threshold) and one false positive alert in Trial B (both thresholds). The BUS method did not alert any during either trial, resulting in false negatives in Trial A.
Substantially different performance was found between SPC and BUS. The SPC method was consistently more sensitive and the BUS method was more specific. The performance of both methods is sensitive to the data used in establishing the expected event rates and alerting thresholds. Theoretically, as the n
(number of subjects) of the baseline data increases, SPC becomes more sensitive and BUS becomes less sensitive (and more specific) to event rate deviations in the monitored data. Conversely, BUS should more rapidly detect an event rate difference than SPC for sparse or low volumes of baseline data, and this has been shown in other monitoring applications.27, 28
The clinical trials evaluated here had large numbers of control patients in order to appropriately evaluate the primary outcomes, and this could have favored the SPC method in this analysis. A sensitivity analysis between the performance of SPC and BUS for large ranges of n
is ongoing in order to determine relative performance between the methods, and we are currently evaluating an alternate BUS alerting threshold using the percentage overlap of the area under the probability density function.
LR-SPC, unlike SPC and BUS, is not directly sensitive to the n of the baseline data because the alerting threshold is generated from the predicted event rate of the observed data, which results in the n used to generate the alerting threshold being equal to n of the observed data. LR-SPC is sensitive to the performance of the logistic regression model used, and such models are more robust when generated from larger data sets.
In this evaluation, LR-SPC performed in a comparable manner to the SPC method. However, this result should be interpreted as LR-SPC performing in a non-inferior way to SPC in the absence of confounding in the evaluated data. This is a useful finding because it supports the use of the methodology, but it does not provide an evaluation of the method’s risk adjustment efficacy. Further work needs to be performed to establish the relative performance between SPC and LR-SPC using observational cohort data.
There are a number of limitations to this study. The control group data in both trials were accumulated concurrently and in a randomized fashion with the intervention data. However, in order to evaluate the system, the control data was assumed to be collected prior to the intervention data. This allows direct comparison of methods, but may result in over-optimistic performance measurements when such methods are applied to a prospective patient cohort, which experiences shifts in patient case-mix and provider behavior over time. In addition, all of the methods used perform serial evaluations of the data, which can increase the false positive alerting rate. However, these methods are intended for screening large numbers of outcomes for a wide variety of medications and medical devices within an automated application. Such surveillance emphasizes early detection and accepts lower sensitivity for additional specificity in this setting. Because of this, in-depth manual review of identified signals must then be performed in order to determine whether the signal is a true positive. Additional work will be required to satisfactorily adjust the sensitivity and specificity of the alerts to a manageable rate for manual review of the results from this application.
These methods are intended for use in prospective observational cohort surveillance within a health care environment, whether it is one hospital or a network of hospitals and outpatient clinics. Once a surveillance methodology is validated and established, selection of the baseline or expected data becomes critical for risk adjustment purposes and defines the nature of any resulting alerts. For example, a medical product just released to market could use phase 3 trial data as a baseline, which would evaluate whether the observed population experienced safety outcomes in excess of that reference group. However, such trials are well-known to recruit healthy patients, and sample sizes are generally low. Alternatively, outcome data from a closely related product with the same indication could be collected in the local environment for this purpose. This has the benefit of a larger sample size and could allow more granular data collection (since data elements in phase 3 trial data are expensive to collect) but might also suffer from missing data or collection, recall, or other biases. Further work must be done in this area to establish data selection hierarchies and protocols in order to inform such a process.
In conclusion, the SPC and LR-SPC methods performed well when evaluating randomized controlled trial data for significant safety event rate elevations. For monitoring where large amounts of data are available to provide the expected event rate (and threshold), SPC and LR-SPC appear to outperform BUS monitoring. Further work is required to establish risk adjustment performance in the LR-SPC method, and to establish BUS performance for event rate monitoring in conditions with sparse prior data or when highly variable trends in safety are present.