To investigate major methods of design and statistical analysis in controlled clinical acupuncture trials published in the West during the past six years (2003–2009) and, based on this analysis, to provide recommendations that address methodological issues and challenges in clinical acupuncture research.
PubMed was searched for acupuncture RCTs published in Western journals in English between 2003 and 2009. The keyword used was acupuncture.
One hundred and eight qualified reports of acupuncture trials that included more than 30 symptoms/conditions were identified, analyzed, and grouped into efficacy (explanatory), effectiveness (pragmatically beneficial) and other (unspecified) studies. All were randomized controlled clinical trials (RCTs). In spite of significant improvement in the quality of acupuncture RCTs in the last 30 years, these reports show that some methodological issues and shortcomings in design and analysis remain. Moreover, the quality of the efficacy studies was not superior to that of the other types of studies. Research design and reporting problems include unclear patient criteria and inadequate practitioner eligibility, inadequate randomization and blinding, deficiencies in the selection of controls, and improper outcome measurements. Problems in statistical analysis included insufficient sample sizes and power calculations, inadequate handling of missing data and multiple comparisons, and inefficient methods for dealing with repeated-measure and cluster data, baseline value adjustment, and confounding issues.
Despite recent advancements in acupuncture research, acupuncture RCTs can be improved, and more rigorous research methods should be carefully considered.
acupuncture; randomized controlled clinical trials; methodology design; statistical analysis
Receiver operating characteristic (ROC) curves can be used to assess the accuracy of tests measured on ordinal or continuous scales. The most commonly used measure for the overall diagnostic accuracy of diagnostic tests is the area under the ROC curve (AUC). A gold standard test on the true disease status is required to estimate the AUC. However, a gold standard test may sometimes be too expensive or infeasible. Therefore, in many medical research studies, the true disease status of the subjects may remain unknown. Under the normality assumption on test results from each disease group of subjects, using the expectation-maximization (EM) algorithm in conjunction with a bootstrap method, we propose a maximum likelihood based procedure for construction of confidence intervals for the difference in paired areas under ROC curves in the absence of a gold standard test. Simulation results show that the proposed interval estimation procedure yields satisfactory coverage probabilities and interval lengths. The proposed method is illustrated with two examples.
Area under the ROC curve; EM algorithm; bootstrap method; gold standard test; maximum likelihood estimation
To estimate the multivariate regression model from multiple individual studies, it would be challenging to obtain results if the input from individual studies only provide univariate or incomplete multivariate regression information. Samsa et al. (J. Biomed. Biotechnol. 2005; 2:113–123) proposed a simple method to combine coefficients from univariate linear regression models into a multivariate linear regression model, a method known as synthesis analysis. However, the validity of this method relies on the normality assumption of the data, and it does not provide variance estimates. In this paper we propose a new synthesis method that improves on the existing synthesis method by eliminating the normality assumption, reducing bias, and allowing for the variance estimation of the estimated parameters.
synthesis analysis; meta-analysis; linear models
In ROC analysis, covariate adjustment is advocated when the covariates impact the magnitude or accuracy of the test under study. Meanwhile, for many large scale screening tests, the true condition status may be subject to missingness because it is expensive and/or invasive to ascertain the disease status. The complete-case analysis may end up with a biased inference, also known as “verification bias”. To address the issue of covariate adjustment with verification bias in ROC analysis, we propose several estimators for the area under the covariate-specific and covariate-adjusted ROC curves (AUCx and AAUC). The AUCx is directly modelled in the form of binary regression, and the estimating equations are based on the U statistics. The AAUC is estimated from the weighted average of AUCx over the covariate distribution of the diseased subjects. We employ reweighting and imputation techniques to overcome the verification bias problem. Our proposed estimators are initially derived assuming that the true disease status is missing at random (MAR), and then with some modification, the estimators can be extended to the not-missing-at-random (NMAR) situation. The asymptotic distributions are derived for the proposed estimators. The finite sample performance is evaluated by a series of simulation studies. Our method is applied to a data set in Alzheimer's disease research.
Alzheimer's disease; area under ROC curve; covariate adjustment; U statistics; verification bias; weighted estimating equations
The association between malaria and meteorological factors is complex due to the lagged and non-linear pattern. Without fully considering these characteristics, existing studies usually concluded inconsistent findings. Investigating the lagged correlation pattern between malaria and climatic variables may improve the understanding of the association and generate possible better prediction models. This is especially beneficial to the south-west China, which is a high-incidence area in China.
Thirty counties in south-west China were selected, and corresponding weekly malaria cases and four weekly meteorological variables were collected from 2004 to 2009. The Multilevel Distributed Lag Non-linear Model (MDLNM) was used to study the temporal lagged correlation between weekly malaria and weekly meteorological factors. The counties were divided into two groups, hot and cold weathers, in order to compare the difference under different climatic conditions and improve reliability and generalizability within similar climatic conditions.
Rainfall was associated with malaria cases in both hot and cold weather counties with a lagged correlation, and the lag range was relatively longer than those of other meteorological factors. Besides, the lag range was longer in hot weather counties compared to cold weather counties. Relative humidity was correlated with malaria cases at early and late lags in hot weather counties.
Minimum temperature had a longer lag range and larger correlation coefficients for hot weather counties compared to cold weather counties. Maximum temperature was only associated with malaria cases at early lags.
Using weekly malaria cases and meteorological information, this work studied the temporal lagged association pattern between malaria cases and meteorological information in south-west China. The results suggest that different meteorological factors show distinct patterns and magnitudes for the lagged correlation, and the patterns will depend on the climatic condition. Existing inconsistent findings for climatic factors’ lags could be due to either the invalid assumption of a single fixed lag or the distinct temperature conditions from different study sites. The lag pattern for meteorological factors should be considered in the development of malaria early warning system.
For censored survival outcomes, it can be of great interest to evaluate the predictive power of individual markers or their functions. Compared with alternative evaluation approaches, the time-dependent ROC (receiver operating characteristics) based approaches rely on much weaker assumptions, can be more robust, and hence are preferred. In this article, we examine evaluation of markers’ predictive power using the time-dependent ROC curve and a concordance measure which can be viewed as a weighted area under the time-dependent AUC (area under the ROC curve) profile. This study significantly advances from existing time-dependent ROC studies by developing nonparametric estimators of the summary indexes and, more importantly, rigorously establishing their asymptotic properties. It reinforces the statistical foundation of the time-dependent ROC based evaluation approaches for censored survival outcomes. Numerical studies, including simulations and application to an HIV clinical trial, demonstrate the satisfactory finite-sample performance of the proposed approaches.
time-dependent ROC; concordance measure; inverse-probability-of-censoring weighting; marker evaluation; survival outcomes
While the experimental Alzheimer's drugs recently developed by pharmaceutical companies failed to stop the progression of Alzheimer's disease, clinicians strive to seek clues on how the patients would be when they visit back next year, based upon the patients' current clinical and neuropathologic diagnosis results. This is related to how to precisely identify the transitional patterns of Alzheimer's disease. Due to the complexities of the diagnosis of Alzheimer's disease, the condition of the disease is usually characterized by multiple clinical and neuropathologic measurements, including Clinical Dementia Rating (CDRGLOB), Mini-Mental State Examination (MMSE), a score derived from the clinician judgement on neuropsychological tests (COGSTAT), and Functional Activities Questionnaire (FAQ). In this research article, we investigate a class of novel joint random-effects transition models that are used to simultaneously analyze the transitional patterns of multiple primary measurements of Alzheimer's disease and, at the same time, account for the association between the measurements. The proposed methodology can avoid the bias introduced by ignoring the correlation between primary measurements and can predict subject-specific transitional patterns.
Rational and Objectives
Receiver operating characteristic analysis (ROC) is often used to find the optimal combination of biomarkers. When the subject level covariates affect the magnitude and/or accuracy of the biomarkers, the combination rule should take into account of the covariate adjustment. The authors propose two new biomarker combination methods that make use of the covariate information.
Materials and Methods
The first method is to maximize the area under covariate-adjusted ROC curve (AAUC). To overcome the limitations of the AAUC measure, the authors further proposed the area under covariate standardized ROC curve (SAUC), which is an extension of the covariate-specific ROC curve. With a series of simulation studies, the proposed optimal AAUC and SAUC methods are compared with the optimal AUC method that ignores the covariates. The biomarker combination methods are illustrated by an example from Alzheimer's disease research.
The simulation results indicate that the optimal AAUC combination performs well in the current study population. The optimal SAUC method is flexible to choose any reference populations, and allows the results to be generalized to different populations.
The proposed optimal AAUC and SAUC approaches successfully address the covariate adjustment problem in estimating the optimal marker combination. The optimal SAUC method is preferred for practical use, because the biomarker combination rule can be easily evaluated for different population of interest.
Biomarker combination; covariate adjustment; AUC; covariate standardization
In observational studies, interest often lies in estimation of the population-level relationship between the explanatory variables and dependent variables, and the estimation is often done using longitudinal data. Longitudinal data often feature sampling error and bias due to non-random drop-out. However, inclusion of population-level information can increase estimation efficiency. In this paper we consider a generalized partially linear model for incomplete longitudinal data in the presence of the population-level information. A pseudo-empirical likelihood-based method is introduced to incorporate population-level information, and non-random drop-out bias is corrected by using a weighted generalized estimating equations method. A three-step estimation procedure is proposed, which makes the computation easier. Several methods that are often used in practice are compared in simulation studies, which demonstrate that our proposed method can correct the non-random drop-out bias and increase the estimation efficiency, especially for small sample size or when the missing proportion is high. We apply this method to an Alzheimer's disease study.
Auxiliary; drop-out; longitudinal data; partially linear model; population-level information; pseudo-empirical likelihood
In this article we propose a separation curve method to identify the range of false positive rates for which two ROC curves differ or one ROC curve is superior to the other. Our method is based on a general multivariate ROC curve model, including interaction terms between discrete covariates and false positive rates. It is applicable with most existing ROC curve models. Furthermore, we introduce a semiparametric least squares ROC estimator and apply the estimator to the separation curve method. We derive a sandwich estimator for the covariance matrix of the semiparametric estimator. We illustrate the application of our separation curve method through two real life examples.
Confidence band; Empirical distribution function; Least squares
Many analyses for incomplete longitudinal data are directed to examining the impact of covariates on the marginal mean responses. We consider the setting in which longitudinal responses are collected from individuals nested within clusters. We discuss methods for assessing covariate effects on the mean and association parameters when covariates are incompletely observed. Weighted first and second order estimating equations are constructed to obtain consistent estimates of mean and association parameters when covariates are missing at random. Empirical studies demonstrate that estimators from the proposed method have negligible finite sample biases in moderate samples. An application to the National Alzheimer’s Coordinating Center (NACC) Uniform Data Set (UDS) demonstrates the utility of the proposed method.
Association; Generalized estimating equation; Longitudinal data; Missing covariates
As a useful tool for geographical cluster detection of events, the spatial scan statistic is widely applied in many fields and plays an increasingly important role. The classic version of the spatial scan statistic for the binary outcome is developed by Kulldorff, based on the Bernoulli or the Poisson probability model. In this paper, we apply the Hypergeometric probability model to construct the likelihood function under the null hypothesis. Compared with existing methods, the likelihood function under the null hypothesis is an alternative and indirect method to identify the potential cluster, and the test statistic is the extreme value of the likelihood function. Similar with Kulldorff’s methods, we adopt Monte Carlo test for the test of significance. Both methods are applied for detecting spatial clusters of Japanese encephalitis in Sichuan province, China, in 2009, and the detected clusters are identical. Through a simulation to independent benchmark data, it is indicated that the test statistic based on the Hypergeometric model outweighs Kulldorff’s statistics for clusters of high population density or large size; otherwise Kulldorff’s statistics are superior.
Longitudinal studies often feature incomplete response and covariate data. Likelihood-based methods such as the expectation–maximization algorithm give consistent estimators for model parameters when data are missing at random (MAR) provided that the response model and the missing covariate model are correctly specified; however, we do not need to specify the missing data mechanism. An alternative method is the weighted estimating equation, which gives consistent estimators if the missing data and response models are correctly specified; however, we do not need to specify the distribution of the covariates that have missing values. In this article, we develop a doubly robust estimation method for longitudinal data with missing response and missing covariate when data are MAR. This method is appealing in that it can provide consistent estimators if either the missing data model or the missing covariate model is correctly specified. Simulation studies demonstrate that this method performs well in a variety of situations.
Doubly robust; Estimating equation; Missing at random; Missing covariate; Missing response
In this article, we first study parameter identifiability in randomized clinical trials with noncompliance and missing outcomes. We show that under certain conditions the parameters of interest are identifiable even under different types of completely nonignorable missing data: that is, the missing mechanism depends on the outcome. We then derive their maximum likelihood and moment estimators and evaluate their finite-sample properties in simulation studies in terms of bias, efficiency, and robustness. Our sensitivity analysis shows that the assumed nonignorable missing-data model has an important impact on the estimated complier average causal effect (CACE) parameter. Our new method provides some new and useful alternative nonignorable missing-data models over the existing latent ignorable model, which guarantees parameter identifiability, for estimating the CACE in a randomized clinical trial with noncompliance and missing data.
Causal inference; Identifiability; Maximum likelihood estimates; Missing data; Noncompliance; Nonignorable
Identifying risk factors for transition rates among normal cognition, mildly cognitive impairment, dementia and death in an Alzheimer’s disease study is very important. It is known that transition rates among these states are strongly time dependent. While Markov process models are often used to describe these disease progressions, the literature mainly focuses on time homogeneous processes, and limited tools are available for dealing with non-homogeneity. Further, patients may choose when they want to visit the clinics, which creates informative observations. In this paper, we develop methods to deal with non-homogeneous Markov processes through time scale transformation when observation times are pre-planned with some observations missing. Maximum likelihood estimation via the EM algorithm is derived for parameter estimation. Simulation studies demonstrate that the proposed method works well under a variety of situations. An application to the Alzheimer’s disease study identifies that there is a significant increase in transition rates as a function of time. Furthermore, our models reveal that the non-ignorable missing mechanism is perhaps reasonable.
Markov; Missing data; Non-homogeneous; Transformation
Incomplete multi-level data arise commonly in many clinical trials and observational studies. Because of multi-level variations in this type of data, appropriate data analysis should take these variations into account. A random effects model can allow for the multi-level variations by assuming random effects at each level, but the computation is intensive because high-dimensional integrations are often involved in fitting models. Marginal methods such as the inverse probability weighted generalized estimating equations can involve simple estimation computation, but it is hard to specify the working correlation matrix for multi-level data. In this paper, we introduce a latent variable method to deal with incomplete multi-level data when the missing mechanism is missing at random, which fills the gap between the random effects model and marginal models. Latent variable models are built for both the response and missing data processes to incorporate the variations that arise at each level. Simulation studies demonstrate that this method performs well in various situations. We apply the proposed method to an Alzheimer’s disease study.
estimating equation; latent variable; missing at random; missing response; multi-level
In estimation of the ROC curve, when the true disease status is subject to nonignorable missingness, the observed likelihood involves the missing mechanism given by a selection model. In this paper, we proposed a likelihood-based approach to estimate the ROC curve and the area under ROC curve when the verification bias is nonignorable. We specified a parametric disease model in order to make the nonignorable selection model identifiable. With the estimated verification and disease probabilities, we constructed four types of empirical estimates of the ROC curve and its area based on imputation and reweighting methods. In practice, a reasonably large sample size is required to estimate the nonignorable selection model in our settings. Simulation studies showed that all the four estimators of ROC area performed well, and imputation estimators were generally more efficient than the other estimators proposed. We applied the proposed method to a data set from research in the Alzheimer’s disease.
Alzheimer’s disease; nonignorable missing data; ROC curve; verification bias
Covariate-specific ROC curves are often used to evaluate the classification accuracy of a medical diagnostic test or a biomarker, when the accuracy of the test is associated with certain covariates. In many large-scale screening tests, the gold standard is subject to missingness due to high cost or harmfulness to the patient. In this paper, we propose a semiparametric estimation of the covariate-specific ROC curves with a partial missing gold standard. A location-scale model is constructed for the test result to model the covariates’ effect, but the residual distributions are left unspecified. Thus the baseline and link functions of the ROC curve both have flexible shapes. With the gold standard missing at random (MAR) assumption, we consider weighted estimating equations for the location-scale parameters, and weighted kernel estimating equations for the residual distributions. Three ROC curve estimators are proposed and compared, namely, imputation-based, inverse probability weighted and doubly robust estimators. We derive the asymptotic normality of the estimated ROC curve, as well as the analytical form the standard error estimator. The proposed method is motivated and applied to the data in an Alzheimer's disease research.
Alzheimer's disease; covariate-specific ROC curve; ignorable missingness; verification bias; weighted estimating equations
Our research is motivated by 2 methodological problems in assessing diagnostic accuracy of traditional Chinese medicine (TCM) doctors in detecting a particular symptom whose true status has an ordinal scale and is unknown—imperfect gold standard bias and ordinal scale symptom status. In this paper, we proposed a nonparametric maximum likelihood method for estimating and comparing the accuracy of different doctors in detecting a particular symptom without a gold standard when the true symptom status had an ordered multiple class. In addition, we extended the concept of the area under the receiver operating characteristic curve to a hyper-dimensional overall accuracy for diagnostic accuracy and alternative graphs for displaying a visual result. The simulation studies showed that the proposed method had good performance in terms of bias and mean squared error. Finally, we applied our method to our motivating example on assessing the diagnostic abilities of 5 TCM doctors in detecting symptoms related to Chills disease.
Bootstrap; Diagnostic accuracy; EM algorithm; MSE; Ordinal tests; Traditional Chinese medicine (TCM); Volume under the ROC surface (VUS)
In this paper, we derive sequential conditional probability ratio tests to compare diagnostic tests without distributional assumptions on test results. The test statistics in our method are nonparametric weighted areas under the receiver-operating characteristic curves. By using the new method, the decision of stopping the diagnostic trial early is unlikely to be reversed should the trials continue to the planned end. The conservatism reflected in this approach to have more conservative stopping boundaries during the course of the trial is especially appealing for diagnostic trials since the end point is not death. In addition, the maximum sample size of our method is not greater than a fixed sample test with similar power functions. Simulation studies are performed to evaluate the properties of the proposed sequential procedure. We illustrate the method using data from a thoracic aorta imaging study.
diagnostic accuracy; ROC; AUC; weighted AUC; SCPRT
In this paper, we proposed a semi-parametric single-index two-part regression model to weaken assumptions in parametric regression methods that were frequently used in the analysis of skewed data with additional zero values. The estimation procedure for the parameters of interest in the model was easily implemented. The proposed estimators were shown to be consistent and asymptotically normal. Through a simulation study, we showed that the proposed estimators have reasonable finite-sample performance. We illustrated the application of the proposed method in one real study on the analysis of health care costs.
Logistic regression; Kernel regression; Skewed data; Single-index model
Existing research shows differences in medication use for Alzheimer's disease (AD) based on demographics such as race, ethnicity, and geographical location. To determine individual and community characteristics associated with differences in acetylcholinesterase inhibitor (AChEI) and memantine use in AD, 3,049 AD subjects were drawn from 30 centers and evaluated using the Uniform data set (UDS). Cases were evaluated at the individual level within the context of 31 communities (one center encompassed two separate geographical regions). Multivariate analysis was used to determine the significance of individual variables on medication use. Compared to non-Hispanic Whites, Blacks were less likely to use AChEI and memantine with odds ratios (OR) of 0.59 (95% CI 0.46-0.76) and 0.43 (95% CI 0.32-0.57), respectively. Compared to non-Hispanic Whites, non-Black Hispanics were less likely to use memantine (OR = 0.69 (95% CI 0.49-0.98)). No association was found between the proportion of Blacks or non-Black Hispanics versus non-Hispanic Whites at an Alzheimer Disease Center and individual use of AChEI or memantine. Other significant variables include gender, age, marital status, dementia severity, source of referral, AChEI use, and education. Education and age somewhat mitigated disparity. Significant racial and ethnic differences in AChEI and memantine use exist at the individual level regardless of the racial and ethnic composition of the individual's community. Research and initiatives at the societal level may be an important consideration toward addressing these differences.
acetylcholinesterase inhibitor; Alzheimer's disease; disparity; ethnicity; memantine; race
Although correlations of cervical cytology to human papillomavirus (HPV) load and histopathology are recognized, it is largely undetermined whether viral load-related risks of cervical intraepithelial neoplasia III (CIN3) differ by cytology.
Study subjects were 821 women enrolled in the ASCUS-LSIL Triage Study who were positive for HPV16 at entry. Women were followed semi-annually over 2 years. Baseline HPV16 load was measured by real-time PCR; expressed as log10 [HPV16 copies per-nanogram of cellular DNA].
CIN3 was confirmed in 34.8% of 821 women during 2-year follow-up. The adjusted odds ratio (OR) associating 2-year cumulative risk of CIN3 with per log10–unit increase in HPV16 load was 1.46 (95%CI, 1.29-1.64). The ORs varied from 1.66 (95%CI, 1.16-2.37) for women with normal cytology at enrollment to 0.86 (95%CI, 0.61-1.20) for those with HSIL. Among women with normal cytology at enrollment, the area under the receiver operating characteristic curve for detecting CIN3 by viral load was 0.70 (95%CI, 0.61-0.78).
HPV16 DNA load was associated with CIN3 risk but the associations varied with cytology detected at the time when the viral load was measured. Clinical utility of testing for HPV16 load for CIN3 detection was minimal even in women with normal cytology.
Human Papillomavirus; Viral load; Cervical Intraepithelial Neoplasia
In this paper, we propose a new semi-parametric maximum likelihood (ML) estimate of an ROC curve that satisfies the property of invariance of the ROC curve and is easy to compute. We show that our new estimator is
n-consistent and has an asymptotically normal distribution. Our extensive simulation studies show the proposed method is efficient, robust, and simple to compute. Finally, we illustrate the application of the proposed estimator in a real data set.
ROC curves; Sensitivity and specificity; Semi-parametric maximum likelihood estimators
Translation of evidence-based guidelines into clinical practice has been inconsistent. We performed a randomized, controlled trial of guideline-based care suggestions delivered to physicians when writing orders on computer workstations.
Inner-city academic general internal medicine practice.
Randomized, controlled trial of 246 physicians (25 percent faculty general internists, 75 percent internal medicine residents) and 20 outpatient pharmacists. We enrolled 706 of their primary care patients with asthma or chronic obstructive pulmonary disease. Care suggestions concerning drugs and monitoring were delivered to a random half of the physicians and pharmacists when writing orders or filling prescriptions using computer workstations. A 2 × 2 factorial randomization of practice sessions and pharmacists resulted in four groups of patients: physician intervention, pharmacist intervention, both interventions, and controls.
Data Extraction/Collection Methods
Adherence to the guidelines and clinical activity was assessed using patients' electronic medical records. Health-related quality of life, medication adherence, and satisfaction with care were assessed using telephone questionnaires.
During their year in the study, patients made an average of five scheduled primary care visits. There were no differences between groups in adherence to the care suggestions, generic or condition-specific quality of life, satisfaction with physicians or pharmacists, medication compliance, emergency department visits, or hospitalizations. Physicians receiving the intervention had significantly higher total health care costs. Physician attitudes toward guidelines were mixed.
Care suggestions shown to physicians and pharmacists on computer workstations had no effect on the delivery or outcomes of care for patients with reactive airways disease.
medical decision making; guidelines; quality improvement