Translation of evidence-based guidelines into clinical practice has been inconsistent. We performed a randomized, controlled trial of guideline-based care suggestions delivered to physicians when writing orders on computer workstations.
Inner-city academic general internal medicine practice.
Randomized, controlled trial of 246 physicians (25 percent faculty general internists, 75 percent internal medicine residents) and 20 outpatient pharmacists. We enrolled 706 of their primary care patients with asthma or chronic obstructive pulmonary disease. Care suggestions concerning drugs and monitoring were delivered to a random half of the physicians and pharmacists when writing orders or filling prescriptions using computer workstations. A 2 × 2 factorial randomization of practice sessions and pharmacists resulted in four groups of patients: physician intervention, pharmacist intervention, both interventions, and controls.
Data Extraction/Collection Methods
Adherence to the guidelines and clinical activity was assessed using patients' electronic medical records. Health-related quality of life, medication adherence, and satisfaction with care were assessed using telephone questionnaires.
During their year in the study, patients made an average of five scheduled primary care visits. There were no differences between groups in adherence to the care suggestions, generic or condition-specific quality of life, satisfaction with physicians or pharmacists, medication compliance, emergency department visits, or hospitalizations. Physicians receiving the intervention had significantly higher total health care costs. Physician attitudes toward guidelines were mixed.
Care suggestions shown to physicians and pharmacists on computer workstations had no effect on the delivery or outcomes of care for patients with reactive airways disease.
medical decision making; guidelines; quality improvement
To investigate major methods of design and statistical analysis in controlled clinical acupuncture trials published in the West during the past six years (2003–2009) and, based on this analysis, to provide recommendations that address methodological issues and challenges in clinical acupuncture research.
PubMed was searched for acupuncture RCTs published in Western journals in English between 2003 and 2009. The keyword used was acupuncture.
One hundred and eight qualified reports of acupuncture trials that included more than 30 symptoms/conditions were identified, analyzed, and grouped into efficacy (explanatory), effectiveness (pragmatically beneficial) and other (unspecified) studies. All were randomized controlled clinical trials (RCTs). In spite of significant improvement in the quality of acupuncture RCTs in the last 30 years, these reports show that some methodological issues and shortcomings in design and analysis remain. Moreover, the quality of the efficacy studies was not superior to that of the other types of studies. Research design and reporting problems include unclear patient criteria and inadequate practitioner eligibility, inadequate randomization and blinding, deficiencies in the selection of controls, and improper outcome measurements. Problems in statistical analysis included insufficient sample sizes and power calculations, inadequate handling of missing data and multiple comparisons, and inefficient methods for dealing with repeated-measure and cluster data, baseline value adjustment, and confounding issues.
Despite recent advancements in acupuncture research, acupuncture RCTs can be improved, and more rigorous research methods should be carefully considered.
acupuncture; randomized controlled clinical trials; methodology design; statistical analysis
Diagnostic trials often require the use of a homogeneity test among several markers. Such a test may be necessary to determine the power both during the design phase and in the initial analysis stage. However, no formal method is available for the power and sample size calculation when the number of markers is greater than two and marker measurements are clustered in subjects. This article presents two procedures for testing the accuracy among clustered diagnostic markers. The first procedure is a test of homogeneity among continuous markers based on a global null hypothesis of the same accuracy. The result under the alternative provides the explicit distribution for the power and sample size calculation. The second procedure is a simultaneous pairwise comparison test based on weighted areas under the receiver operating characteristic curves. This test is particularly useful if a global difference among markers is found by the homogeneity test. We apply our procedures to the BioCycle Study designed to assess and compare the accuracy of hormone and oxidative stress markers in distinguishing women with ovulatory menstrual cycles from those without.
ROC curve; biomarker; homogeneity test; sample size
In the evaluation of a biomarker for risk prediction, one can assess the performance of the biomarker in the population of interest by displaying the predictiveness curve. In conjunction with an assessment of the classification accuracy of a biomarker, the predictiveness curve is an important tool for assessing the usefulness of a risk prediction model. Inference for a single biomarker or for multiple biomarkers can be performed using summary measures of predictiveness curve. We propose two partial summary measures, the partial total gain and the partial proportion of explained variation, that summarize the predictiveness curve over a restricted range of risk. The methods we describe can be used to compare two biomarkers when there are existing thresholds for risk stratification. We describe inferencial tools for one and two-samples that are shown to have adequate power in a simulation study. The methods are illustrated by assessing the accuracy of a risk score for predicting the onset of Alzheimer's Disease.
Biomarker; Classification; Prediction; Summary statistic
To evaluate the validity and reliability of the structured Composite International Diagnostic Interview (CIDI) in diagnosing current major depressive disorder (MDD) among East African adults.
A sample of 926 patients attending a major referral hospital participated in this diagnostic assessment study. We used a two stage-study design where participants were first interviewed using an Amharic version of the CIDI and a stratified random sample underwent a follow-up semi-structured clinical interview conducted by a psychiatrist, blinded to the screening results, using the Schedules for Clinical Assessment in Neuropsychiatry (SCAN) instrument. We tested construct validity by examining the association of the CIDI and World Health Organization Quality of Life (WHO-QOL) questionnaire. We calculated the psychometric properties of the CIDI using the SCAN diagnostic interview as a gold standard.
We found that the Amharic version of the CIDI diagnostic interview has good internal reliability (Cronbach’s alpha= 0.97) among Ethiopian adults. Compared to the SCAN reference standard, the CIDI had fair specificity (72.2%) but low sensitivity (51.0%). Our study provided evidence for unidimensionality of core depression screening questions on the CIDI interview with good factor loadings on a major core depressive factor.
The Amharic language version of the CIDI had fair specificity and low sensitivity in detecting MDD compared with psychiatrist administered SCAN diagnosis. Our findings are generally consistent with prior studies. Use of fully structured interviews such as the CIDI for MDD diagnosis in clinical settings might lead to under detection of DSM-IV MDD.
CIDI; Validation; Africa; Ethiopia; Depression; MDD
Malaria transmission is strongly determined by the environmental temperature and the environment is rarely constant. Therefore, mosquitoes and parasites are not only exposed to the mean temperature, but also to daily temperature variation. Recently, both theoretical and laboratory work has shown, in addition to mean temperatures, daily fluctuations in temperature can affect essential mosquito and parasite traits that determine malaria transmission intensity. However, so far there is no epidemiological evidence at the population level to this problem.
Thirty counties in southwest China were selected, and corresponding weekly malaria cases and weekly meteorological variables were collected from 2004 to 2009. Particularly, maximum, mean and minimum temperatures were collected. The daily temperature fluctuation was measured by the diurnal temperature range (DTR), the difference between the maximum and minimum temperature. The distributed lag non-linear model (MDLNM) was used to study the correlation between weekly malaria incidences and weekly mean temperatures, and the correlation pattern was allowed to vary over different levels of daily temperature fluctuations.
The overall non-linear patterns for mean temperatures are distinct across different levels of DTR. When under cooler temperature conditions, the larger mean temperature effect on malaria incidences is found in the groups of higher DTR, suggesting that large daily temperature fluctuations act to speed up the malaria incidence in cooler environmental conditions. In contrast, high daily fluctuations under warmer conditions will lead to slow down the mean temperature effect. Furthermore, in the group of highest DTR, 24-25°C or 21-23°C are detected as the optimal temperature for the malaria transmission.
The environment is rarely constant, and the result highlights the need to consider temperature fluctuations as well as mean temperatures, when trying to understand or predict malaria transmission. This work may be the first epidemiological study confirming that the effect of the mean temperature depends on temperature fluctuations, resulting in relevant evidence at the population level.
Receiver operating characteristic (ROC) curves can be used to assess the accuracy of tests measured on ordinal or continuous scales. The most commonly used measure for the overall diagnostic accuracy of diagnostic tests is the area under the ROC curve (AUC). A gold standard test on the true disease status is required to estimate the AUC. However, a gold standard test may sometimes be too expensive or infeasible. Therefore, in many medical research studies, the true disease status of the subjects may remain unknown. Under the normality assumption on test results from each disease group of subjects, using the expectation-maximization (EM) algorithm in conjunction with a bootstrap method, we propose a maximum likelihood based procedure for construction of confidence intervals for the difference in paired areas under ROC curves in the absence of a gold standard test. Simulation results show that the proposed interval estimation procedure yields satisfactory coverage probabilities and interval lengths. The proposed method is illustrated with two examples.
Area under the ROC curve; EM algorithm; bootstrap method; gold standard test; maximum likelihood estimation
To estimate the multivariate regression model from multiple individual studies, it would be challenging to obtain results if the input from individual studies only provide univariate or incomplete multivariate regression information. Samsa et al. (J. Biomed. Biotechnol. 2005; 2:113–123) proposed a simple method to combine coefficients from univariate linear regression models into a multivariate linear regression model, a method known as synthesis analysis. However, the validity of this method relies on the normality assumption of the data, and it does not provide variance estimates. In this paper we propose a new synthesis method that improves on the existing synthesis method by eliminating the normality assumption, reducing bias, and allowing for the variance estimation of the estimated parameters.
synthesis analysis; meta-analysis; linear models
Life history data arising in clusters with prespecified assessment time points for patients often feature incomplete data since patients may choose to visit the clinic based on their needs. Markov process models provide a useful tool describing disease progression for life history data. The literature mainly focuses on time homogeneous process. In this paper we develop methods to deal with non-homogeneous Markov process with incomplete clustered life history data. A correlated random effects model is developed to deal with the nonignorable missingness, and a time transformation is employed to address the non-homogeneity in the transition model. Maximum likelihood estimate based on the Monte-Carlo EM algorithm is advocated for parameter estimation. Simulation studies demonstrate that the proposed method works well in many situations. We also apply this method to an Alzheimer's disease study.
Cluster; missing not at random; Markov non-homogeneous; random effects; transition intensity
It is valuable to study the spatiotemporal pattern of Japanese encephalitis (JE) and its association with the contextual risk factors in southwest China, which is the most endemic area in China. Using data from 2004 to 2009, we applied GISmapping and spatial autocorrelation analysis to analyze reported incidence data of JE in 438 counties in southwest China, finding that JE cases were not randomly distributed, and a Bayesian hierarchical spatiotemporal model identified the east part of southwest China as a high risk area. Meanwhile, the Bayesian hierarchical spatial model in 2006 demonstrated a statistically significant association between JE and the agricultural and climatic variables, including the proportion of rural population, the pig-to-human ratio, the monthly precipitation and the monthly mean minimum and maximum temperatures. Particular emphasis was placed on the time-lagged effect for climatic factors. The regression method and the Spearman correlation analysis both identified a two-month lag for the precipitation, while the regression method found a one-month lag for temperature. The results show that the high risk area in the east part of southwest China may be connected to the agricultural and climatic factors. The routine surveillance and the allocation of health resources should be given more attention in this area. Moreover, the meteorological variables might be considered as possible predictors of JE in southwest China.
Japanese encephalitis; contextual risk factors; meteorological factors; southwest China; Bayesian hierarchical model
In ROC analysis, covariate adjustment is advocated when the covariates impact the magnitude or accuracy of the test under study. Meanwhile, for many large scale screening tests, the true condition status may be subject to missingness because it is expensive and/or invasive to ascertain the disease status. The complete-case analysis may end up with a biased inference, also known as “verification bias”. To address the issue of covariate adjustment with verification bias in ROC analysis, we propose several estimators for the area under the covariate-specific and covariate-adjusted ROC curves (AUCx and AAUC). The AUCx is directly modelled in the form of binary regression, and the estimating equations are based on the U statistics. The AAUC is estimated from the weighted average of AUCx over the covariate distribution of the diseased subjects. We employ reweighting and imputation techniques to overcome the verification bias problem. Our proposed estimators are initially derived assuming that the true disease status is missing at random (MAR), and then with some modification, the estimators can be extended to the not-missing-at-random (NMAR) situation. The asymptotic distributions are derived for the proposed estimators. The finite sample performance is evaluated by a series of simulation studies. Our method is applied to a data set in Alzheimer's disease research.
Alzheimer's disease; area under ROC curve; covariate adjustment; U statistics; verification bias; weighted estimating equations
The association between malaria and meteorological factors is complex due to the lagged and non-linear pattern. Without fully considering these characteristics, existing studies usually concluded inconsistent findings. Investigating the lagged correlation pattern between malaria and climatic variables may improve the understanding of the association and generate possible better prediction models. This is especially beneficial to the south-west China, which is a high-incidence area in China.
Thirty counties in south-west China were selected, and corresponding weekly malaria cases and four weekly meteorological variables were collected from 2004 to 2009. The Multilevel Distributed Lag Non-linear Model (MDLNM) was used to study the temporal lagged correlation between weekly malaria and weekly meteorological factors. The counties were divided into two groups, hot and cold weathers, in order to compare the difference under different climatic conditions and improve reliability and generalizability within similar climatic conditions.
Rainfall was associated with malaria cases in both hot and cold weather counties with a lagged correlation, and the lag range was relatively longer than those of other meteorological factors. Besides, the lag range was longer in hot weather counties compared to cold weather counties. Relative humidity was correlated with malaria cases at early and late lags in hot weather counties.
Minimum temperature had a longer lag range and larger correlation coefficients for hot weather counties compared to cold weather counties. Maximum temperature was only associated with malaria cases at early lags.
Using weekly malaria cases and meteorological information, this work studied the temporal lagged association pattern between malaria cases and meteorological information in south-west China. The results suggest that different meteorological factors show distinct patterns and magnitudes for the lagged correlation, and the patterns will depend on the climatic condition. Existing inconsistent findings for climatic factors’ lags could be due to either the invalid assumption of a single fixed lag or the distinct temperature conditions from different study sites. The lag pattern for meteorological factors should be considered in the development of malaria early warning system.
For censored survival outcomes, it can be of great interest to evaluate the predictive power of individual markers or their functions. Compared with alternative evaluation approaches, the time-dependent ROC (receiver operating characteristics) based approaches rely on much weaker assumptions, can be more robust, and hence are preferred. In this article, we examine evaluation of markers’ predictive power using the time-dependent ROC curve and a concordance measure which can be viewed as a weighted area under the time-dependent AUC (area under the ROC curve) profile. This study significantly advances from existing time-dependent ROC studies by developing nonparametric estimators of the summary indexes and, more importantly, rigorously establishing their asymptotic properties. It reinforces the statistical foundation of the time-dependent ROC based evaluation approaches for censored survival outcomes. Numerical studies, including simulations and application to an HIV clinical trial, demonstrate the satisfactory finite-sample performance of the proposed approaches.
time-dependent ROC; concordance measure; inverse-probability-of-censoring weighting; marker evaluation; survival outcomes
While the experimental Alzheimer's drugs recently developed by pharmaceutical companies failed to stop the progression of Alzheimer's disease, clinicians strive to seek clues on how the patients would be when they visit back next year, based upon the patients' current clinical and neuropathologic diagnosis results. This is related to how to precisely identify the transitional patterns of Alzheimer's disease. Due to the complexities of the diagnosis of Alzheimer's disease, the condition of the disease is usually characterized by multiple clinical and neuropathologic measurements, including Clinical Dementia Rating (CDRGLOB), Mini-Mental State Examination (MMSE), a score derived from the clinician judgement on neuropsychological tests (COGSTAT), and Functional Activities Questionnaire (FAQ). In this research article, we investigate a class of novel joint random-effects transition models that are used to simultaneously analyze the transitional patterns of multiple primary measurements of Alzheimer's disease and, at the same time, account for the association between the measurements. The proposed methodology can avoid the bias introduced by ignoring the correlation between primary measurements and can predict subject-specific transitional patterns.
Rational and Objectives
Receiver operating characteristic analysis (ROC) is often used to find the optimal combination of biomarkers. When the subject level covariates affect the magnitude and/or accuracy of the biomarkers, the combination rule should take into account of the covariate adjustment. The authors propose two new biomarker combination methods that make use of the covariate information.
Materials and Methods
The first method is to maximize the area under covariate-adjusted ROC curve (AAUC). To overcome the limitations of the AAUC measure, the authors further proposed the area under covariate standardized ROC curve (SAUC), which is an extension of the covariate-specific ROC curve. With a series of simulation studies, the proposed optimal AAUC and SAUC methods are compared with the optimal AUC method that ignores the covariates. The biomarker combination methods are illustrated by an example from Alzheimer's disease research.
The simulation results indicate that the optimal AAUC combination performs well in the current study population. The optimal SAUC method is flexible to choose any reference populations, and allows the results to be generalized to different populations.
The proposed optimal AAUC and SAUC approaches successfully address the covariate adjustment problem in estimating the optimal marker combination. The optimal SAUC method is preferred for practical use, because the biomarker combination rule can be easily evaluated for different population of interest.
Biomarker combination; covariate adjustment; AUC; covariate standardization
In observational studies, interest often lies in estimation of the population-level relationship between the explanatory variables and dependent variables, and the estimation is often done using longitudinal data. Longitudinal data often feature sampling error and bias due to non-random drop-out. However, inclusion of population-level information can increase estimation efficiency. In this paper we consider a generalized partially linear model for incomplete longitudinal data in the presence of the population-level information. A pseudo-empirical likelihood-based method is introduced to incorporate population-level information, and non-random drop-out bias is corrected by using a weighted generalized estimating equations method. A three-step estimation procedure is proposed, which makes the computation easier. Several methods that are often used in practice are compared in simulation studies, which demonstrate that our proposed method can correct the non-random drop-out bias and increase the estimation efficiency, especially for small sample size or when the missing proportion is high. We apply this method to an Alzheimer's disease study.
Auxiliary; drop-out; longitudinal data; partially linear model; population-level information; pseudo-empirical likelihood
In this article we propose a separation curve method to identify the range of false positive rates for which two ROC curves differ or one ROC curve is superior to the other. Our method is based on a general multivariate ROC curve model, including interaction terms between discrete covariates and false positive rates. It is applicable with most existing ROC curve models. Furthermore, we introduce a semiparametric least squares ROC estimator and apply the estimator to the separation curve method. We derive a sandwich estimator for the covariance matrix of the semiparametric estimator. We illustrate the application of our separation curve method through two real life examples.
Confidence band; Empirical distribution function; Least squares
Many analyses for incomplete longitudinal data are directed to examining the impact of covariates on the marginal mean responses. We consider the setting in which longitudinal responses are collected from individuals nested within clusters. We discuss methods for assessing covariate effects on the mean and association parameters when covariates are incompletely observed. Weighted first and second order estimating equations are constructed to obtain consistent estimates of mean and association parameters when covariates are missing at random. Empirical studies demonstrate that estimators from the proposed method have negligible finite sample biases in moderate samples. An application to the National Alzheimer’s Coordinating Center (NACC) Uniform Data Set (UDS) demonstrates the utility of the proposed method.
Association; Generalized estimating equation; Longitudinal data; Missing covariates
As a useful tool for geographical cluster detection of events, the spatial scan statistic is widely applied in many fields and plays an increasingly important role. The classic version of the spatial scan statistic for the binary outcome is developed by Kulldorff, based on the Bernoulli or the Poisson probability model. In this paper, we apply the Hypergeometric probability model to construct the likelihood function under the null hypothesis. Compared with existing methods, the likelihood function under the null hypothesis is an alternative and indirect method to identify the potential cluster, and the test statistic is the extreme value of the likelihood function. Similar with Kulldorff’s methods, we adopt Monte Carlo test for the test of significance. Both methods are applied for detecting spatial clusters of Japanese encephalitis in Sichuan province, China, in 2009, and the detected clusters are identical. Through a simulation to independent benchmark data, it is indicated that the test statistic based on the Hypergeometric model outweighs Kulldorff’s statistics for clusters of high population density or large size; otherwise Kulldorff’s statistics are superior.
Longitudinal studies often feature incomplete response and covariate data. Likelihood-based methods such as the expectation–maximization algorithm give consistent estimators for model parameters when data are missing at random (MAR) provided that the response model and the missing covariate model are correctly specified; however, we do not need to specify the missing data mechanism. An alternative method is the weighted estimating equation, which gives consistent estimators if the missing data and response models are correctly specified; however, we do not need to specify the distribution of the covariates that have missing values. In this article, we develop a doubly robust estimation method for longitudinal data with missing response and missing covariate when data are MAR. This method is appealing in that it can provide consistent estimators if either the missing data model or the missing covariate model is correctly specified. Simulation studies demonstrate that this method performs well in a variety of situations.
Doubly robust; Estimating equation; Missing at random; Missing covariate; Missing response
In this article, we first study parameter identifiability in randomized clinical trials with noncompliance and missing outcomes. We show that under certain conditions the parameters of interest are identifiable even under different types of completely nonignorable missing data: that is, the missing mechanism depends on the outcome. We then derive their maximum likelihood and moment estimators and evaluate their finite-sample properties in simulation studies in terms of bias, efficiency, and robustness. Our sensitivity analysis shows that the assumed nonignorable missing-data model has an important impact on the estimated complier average causal effect (CACE) parameter. Our new method provides some new and useful alternative nonignorable missing-data models over the existing latent ignorable model, which guarantees parameter identifiability, for estimating the CACE in a randomized clinical trial with noncompliance and missing data.
Causal inference; Identifiability; Maximum likelihood estimates; Missing data; Noncompliance; Nonignorable
Identifying risk factors for transition rates among normal cognition, mildly cognitive impairment, dementia and death in an Alzheimer’s disease study is very important. It is known that transition rates among these states are strongly time dependent. While Markov process models are often used to describe these disease progressions, the literature mainly focuses on time homogeneous processes, and limited tools are available for dealing with non-homogeneity. Further, patients may choose when they want to visit the clinics, which creates informative observations. In this paper, we develop methods to deal with non-homogeneous Markov processes through time scale transformation when observation times are pre-planned with some observations missing. Maximum likelihood estimation via the EM algorithm is derived for parameter estimation. Simulation studies demonstrate that the proposed method works well under a variety of situations. An application to the Alzheimer’s disease study identifies that there is a significant increase in transition rates as a function of time. Furthermore, our models reveal that the non-ignorable missing mechanism is perhaps reasonable.
Markov; Missing data; Non-homogeneous; Transformation
Incomplete multi-level data arise commonly in many clinical trials and observational studies. Because of multi-level variations in this type of data, appropriate data analysis should take these variations into account. A random effects model can allow for the multi-level variations by assuming random effects at each level, but the computation is intensive because high-dimensional integrations are often involved in fitting models. Marginal methods such as the inverse probability weighted generalized estimating equations can involve simple estimation computation, but it is hard to specify the working correlation matrix for multi-level data. In this paper, we introduce a latent variable method to deal with incomplete multi-level data when the missing mechanism is missing at random, which fills the gap between the random effects model and marginal models. Latent variable models are built for both the response and missing data processes to incorporate the variations that arise at each level. Simulation studies demonstrate that this method performs well in various situations. We apply the proposed method to an Alzheimer’s disease study.
estimating equation; latent variable; missing at random; missing response; multi-level
In estimation of the ROC curve, when the true disease status is subject to nonignorable missingness, the observed likelihood involves the missing mechanism given by a selection model. In this paper, we proposed a likelihood-based approach to estimate the ROC curve and the area under ROC curve when the verification bias is nonignorable. We specified a parametric disease model in order to make the nonignorable selection model identifiable. With the estimated verification and disease probabilities, we constructed four types of empirical estimates of the ROC curve and its area based on imputation and reweighting methods. In practice, a reasonably large sample size is required to estimate the nonignorable selection model in our settings. Simulation studies showed that all the four estimators of ROC area performed well, and imputation estimators were generally more efficient than the other estimators proposed. We applied the proposed method to a data set from research in the Alzheimer’s disease.
Alzheimer’s disease; nonignorable missing data; ROC curve; verification bias
Covariate-specific ROC curves are often used to evaluate the classification accuracy of a medical diagnostic test or a biomarker, when the accuracy of the test is associated with certain covariates. In many large-scale screening tests, the gold standard is subject to missingness due to high cost or harmfulness to the patient. In this paper, we propose a semiparametric estimation of the covariate-specific ROC curves with a partial missing gold standard. A location-scale model is constructed for the test result to model the covariates’ effect, but the residual distributions are left unspecified. Thus the baseline and link functions of the ROC curve both have flexible shapes. With the gold standard missing at random (MAR) assumption, we consider weighted estimating equations for the location-scale parameters, and weighted kernel estimating equations for the residual distributions. Three ROC curve estimators are proposed and compared, namely, imputation-based, inverse probability weighted and doubly robust estimators. We derive the asymptotic normality of the estimated ROC curve, as well as the analytical form the standard error estimator. The proposed method is motivated and applied to the data in an Alzheimer's disease research.
Alzheimer's disease; covariate-specific ROC curve; ignorable missingness; verification bias; weighted estimating equations