|Home | About | Journals | Submit | Contact Us | Français|
To determine whether MRI measurements observed in the Alzheimer's Disease Neuroimaging Initiative (ADNI; convenience-sample) differ from those observed in the Mayo Clinic Study of Aging (MCSA; population-based sample).
Comparison of two samples.
59 recruiting sites for the ADNI in US/Canada, and the MCSA, a population-based cohort in Olmsted County, MN.
Cognitively normal (CN) subjects and amnestic mild cognitive impairment (aMCI) subjects were selected from the ADNI convenience cohort and MCSA population-based cohort. Two samples were selected; the first was a simple random sample of subjects from both cohorts in the same age range, and the second applied matching for age, sex, education, apolipoprotein E genotype, and Mini-Mental State Examination.
Baseline hippocampal volumes and annual percent decline in hippocampal volume.
In the population-based sample, MCSA subjects were older, less educated, performed worse on MMSE, and less often had family history of AD than ADNI subjects. Baseline hippocampal volumes were larger in ADNI compared to MCSA CN subjects in the random sample, although no differences were observed after matching. Rates of decline in hippocampal volume were greater in ADNI compared to MCSA for both CN and aMCI, even after matching.
Rates of decline in hippocampal volume suggest that ADNI subjects have more aggressive brain pathology than MCSA subjects, and hence may not be representative of the general population. These findings have implications for treatment trials that employ ADNI-like recruitment mechanisms and for studies validating new diagnostic criteria for AD in its various stages.
Imaging plays an important role in the study of Alzheimer's disease (AD). Imaging biomarkers can track disease progression1, detect changes early in the mild cognitive impairment (MCI) phase2, 3 and help predict which subjects may later develop AD4, 5. Imaging measures have become common outcome measures in clinical treatment trials because they may reduce sample size6, 7. Increasing interest in using imaging in clinical trials led to the development of the Alzheimer's Disease Neuroimaging Initiative (ADNI) which aimed to improve methods for clinical trials and validate imaging and other biomarkers8, 9. ADNI is an observational study of MCI and AD that used identical recruitment mechanisms as typical trials, including advertising and recruitment from memory clinics. Therefore, ADNI is based on a highly selected convenience-sample. Because ADNI data are freely available, a large number of studies are published each year using these data. However, it is unclear to what extent subjects recruited through these mechanisms are representative of the general population, and hence whether results are generalizable.
We aimed to determine whether imaging measures would differ in ADNI participants compared to the population-based cohort of the Mayo Clinic Study of Aging (MCSA). We assessed hippocampal volume and rates of decline in hippocampal volume because they are established and widely studied biomarkers of AD6, 10, 11. Because results could be influenced by differences in inclusion characteristics and demographics, we compared the cohorts both before and after matching for specific demographic and cognitive features.
Subjects with a clinical diagnosis of amnestic mild cognitive impairment (aMCI) and cognitively normal (CN) subjects who had been recruited into either the MCSA (and had agreed to MRI studies) or ADNI were analyzed.
The MCSA is a longitudinal epidemiologic study of normal ageing and MCI in Olmsted County, Minnesota. The recruitment mechanisms have been reported in detail previously12. Briefly, all Olmsted County residents aged 70–89 years on October 1, 2004 were identified using the medical records-linkage system of the Rochester Epidemiology Project13, 14. The population was also resampled in 2008 and 2009 in order to replenish the cohort. Subjects were randomly selected from this enumeration. Subjects received a letter of invitation giving them the opportunity to refuse participation by returning a letter of refusal. Subjects who did not return the letter then received a follow-up telephone call inviting them to participate. MRI was performed in all subjects who agreed to participate and did not have any contraindications to MRI. Subjects with imaging in the MCSA have very similar demographic characteristics to those that did not undergo imaging (Table 1). Subjects were characterized as CN by consensus12, 15, and when their age-adjusted neuropsychological test scores were consistent with normative data developed in this community16. Diagnostic criteria for MCI were as follows17: 1) cognitive concern by subject, informant (from Clinical Dementia Rating scale (CDR)18), nurse or physician; 2) impairment in 1 or more of the 4 cognitive domains (from cognitive battery); 3) essentially normal functional activities (using CDR and Functional Activities Questionnaire); and 4) absence of dementia (DSM-IV)19. Subjects were categorized as amnestic MCI (aMCI) if memory was impaired. The diagnosis of aMCI was made on clinical grounds without the use of rigid cutoffs on psychometric scores.
ADNI is a longitudinal multi-site observational study of CN, aMCI and AD (www.ADNI-info.org)8. Subjects were recruited using local Alzheimer's Disease Research Centers, memory clinics, newspaper ads, radio, and other public media campaigns. Diagnostic criteria for ADNI were largely the same as for the MCSA. Criteria for CN subjects were: Mini Mental State Exam (MMSE)20 scores between 24 and 30 inclusive; no memory complaints; objective memory performance in the normal range; and a CDR score of 0 and memory box score of 0. Diagnostic criteria for aMCI were: 1) memory complaint verified by an informant; 2) objective memory impairment measured by education adjusted score on the Wechsler Memory Scale-Revised, Logical Memory II; 3) MMSE scores between 24 and 30 inclusive; 4) CDR score of 0.5 and memory box score of at least 0.5, and 5) preservation of general cognition and functional activities of daily living. Subjects enrolled for ADNI were between 55 and 90 years old. The ADNI AD subjects were not included in our analysis because the MCSA does not follow AD subjects.
Informed consent was obtained from all subjects. The MCSA was approved by the Mayo Clinic IRB, and ADNI was approved by the IRB at each individual site.
We selected two samples of subjects: the first was a random sample of all available MCSA and ADNI subjects, and the second sample applied matching for demographic and cognitive variables. Both cross-sectional and longitudinal samples were selected. The first available MRI was used for the cross-sectional analysis and as baseline for the longitudinal analysis. Two serial MRI were used for the longitudinal analysis for each subject. Scan interval was approximately 12-months for ADNI and 15-months for the MCSA (the routine follow-up interval in the MCSA).
For the cross-sectional analysis, the total number of available CN subjects was 229 in ADNI and 1,283 in the MCSA. The total number of aMCI subjects available was 397 in ADNI and 179 in the MCSA. To obtain comparable sample sizes between ADNI and MCSA, we took a simple random sample of the MCSA CN subjects resulting in 229 subjects. Similarly, we took a simple random sample of the ADNI aMCI subjects resulting in 179 subjects. Because of the random sub-sampling strategy, the samples used for our analyses were representative (within sampling error) of the parent cohorts from which they were drawn (Supplemental Table e–1). For the longitudinal analysis, there were 206 ADNI CN subjects with serial scans and 686 MCSA CN subjects. There were 347 ADNI aMCI subjects with serial data and 92 MCSA aMCI subjects. Once again, to obtain comparable group sizes, we took a random sample of the MCSA CN subjects and ADNI aMCI subjects, resulting in 206 MCSA CN subjects and 92 ADNI aMCI subjects.
In sample 2, the ADNI and MCSA subjects were frequency matched by age, sex, education, apolipoprotein (APOE) genotype, and MMSE score. All variables were dichotomized into strata: age (70–79 and 80–90 years), sex (men and women), education (6–13 and 14–20 years), and MMSE (24–28 and 29–30 for CN; 22–25 and 26–30 for aMCI). ADNI and MCSA subjects were matched with a one-to-one frequency by taking a random sample within each of the 32 strata of the larger study group to match the number of subjects in the smaller study group. CN and aMCI subjects were matched separately. Subjects that could not be matched were excluded. For the cross-sectional analysis, 212 CN subjects and 97 aMCI subjects were selected for both ADNI and MCSA. For the longitudinal analysis, 191 CN subjects and 65 aMCI subjects were selected for both ADNI and MCSA.
Subject demographics for the two cross-sectional and longitudinal samples are shown in Tables 2 and and3.3. The samples used for analysis differ slightly from those reported above because some subjects were excluded due to poor quality imaging.
MRI acquisition protocols were very similar for MCSA and ADNI subjects, although MCSA subjects were scanned at 3T while ADNI subjects were scanned at 1.5T. ADNI collects 1.5T MRI scans in all subjects and 3T scans in only 25% of the sample; therefore ADNI 1.5T MRI scans were used for this study. To ensure that field strength did not bias our results, we compared hippocampal volumes at 1.5T and 3T in ADNI subjects that were scanned at both field strengths. Similar to a previous study21, hippocampal measurements were comparable across field strengths (Figure 1).
MCSA subjects were imaged with a 3D magnetization prepared rapid acquisition gradient echo (MPRAGE) sequence developed at Mayo for ADNI9. Parameters were: sagittal plane, TR/TE/TI, 2300/3/900ms; flip angle 8°, 26cm field of view (FOV); 256×256 in-plane matrix with a phase FOV of 0.94 and slice thickness of 1.2mm. ADNI is a multi-site study and there are minor variations in the MRI protocol based on the specific hardware/software configuration on each scanner. The nominal parameters of the ADNI MPRAGE were: sagittal plane, TR/TE/TI, 2400/3/1000ms; flip angle 8°, 24cm FOV; 192×192 in-plane matrix and slice thickness of 1.2mm9.
All images were corrected for gradient non-linearity22 and intensity inhomogeneity23. Hippocampal volumes were measured using Freesurfer software version 4.5.024. The cross-sectional analysis pipeline was used to calculate hippocampal volumes for the cross-sectional sample, and the longitudinal analysis pipeline was used to assess rates of hippocampal change for the longitudinal sample. Hippocampal measurements calculated using Freesurfer have been previously validated against manual measurements25. Total intracranial volumes (TIV) were measured using an algorithm developed in-house26.
Analyses were performed in R version 2.11.0 (R Development Core Team: A Language and Environment for Statistical Computing. 2010 http://www.r-project.org) and tests of statistical significance were conducted at the two-sided alpha level of 0.05. For the cross-sectional analysis, we used hippocampal volume adjusted for TIV. We fit a linear regression model of hippocampal volume (y) versus TIV (x) in all ADNI and MCSA CN subjects with available data (n=1,480) and then used the intercept (b0) and slope (b1) estimates from the model to calculate hippocampal volume adjusted for TIV (HVa) as a residual [HVa = HP − (b0 + b1 × TIV)]. For the longitudinal analysis, the annual percent decline in hippocampal volume was calculated as follows using unadjusted hippocampal volumes: (follow-up volume – baseline volume)/(baseline volume × years between scans) × 100.
Wilcoxon rank-sum/Mann-Whitney U tests were used to test differences in continuous measures between ADNI and MCSA groups, and chi-squared tests with continuity correction or Fisher's exact test were used to test differences in categorical variables. We summarized group differences in imaging measures using the probabilistic index (PI) (corresponding to the area under the receiver operator characteristic curve)27. The PI is a non-parametric estimate of group-wise differences or discrimination that measures the probability that the value from a randomly selected subject in one group is higher than the value from a randomly selected subject in the other group. A PI of 0.50 (or 50%) indicates no difference across groups.
Differences in demographic features across MCSA and ADNI were similar for cross-sectional and longitudinal cohorts (Tables 2 and and3).3). In sample 1, MCSA subjects (aMCI and CN) were older, less educated, and performed worse on the MMSE than the ADNI subjects. The MCSA aMCI subjects had a lower proportion of APOE ε4 carriers than ADNI. No differences were observed in sex, education, or APOE genotype between the MCSA and ADNI subjects in sample 2. Despite frequency matching, age (cross-sectional sample only) and MMSE in the CN subjects still differed across the cohorts, although the median and interquartile ranges were similar. ADNI had a higher proportion of family history of AD, and of minorities across all samples for the CN subjects with a similar trend for aMCI in the cross-sectional sample.
In sample 1, hippocampal volume adjusted for TIV was significantly smaller in the MCSA CN subjects compared to the ADNI CN subjects, with no differences between the groups for the aMCI subjects (Figure 2A). After matching for age, sex, education, APOE genotype, and MMSE in sample 2, no differences in hippocampal volume adjusted for TIV were observed between MCSA and ADNI in either the CN or aMCI subjects (Figure 2B).
In sample 1, the annual percent decline in hippocampal volume was greater in ADNI compared to the MCSA for both aMCI and CN subjects (Figure 3A). After matching for age, sex, education, APOE genotype, and MMSE in sample 2, these differences across ADNI and MCSA were still observed (Figure 3B).
This study highlights demographic differences in subjects recruited into the convenience-sample ADNI cohort compared with subjects recruited into the population-based MCSA cohort, and demonstrates that imaging biomarkers from these two different recruitment mechanisms differ.
The most striking difference was that rates of decline in hippocampal volume were greater in ADNI compared to the MCSA, for both CN and aMCI subjects. This difference was observed even after matching for key demographic and cognitive variables. Increased rates of decline in hippocampal volume in CN subjects predict a faster rate of progression to dementia28, suggesting that the ADNI CN population includes a larger proportion of subjects on the path to AD dementia. While it was somewhat unexpected that the proportion of APOE e4 carriers was not higher in the ADNI CN subjects, our findings are consistent with the unusually high proportion (50%) of ADNI controls who showed amyloid pathology as measured by Pittsburgh Compound B (PiB)29. By contrast, the proportion of PiB positive controls in the MCSA was only 30%30. The pathological diagnosis of AD was also more common in controls from a clinic versus a community setting in a previous study31. The ADNI CN subjects were more highly educated than the MCSA CN subjects; therefore, cognitive reserve mechanisms may have protected them from clinical decline even though they are on a steeper downward trajectory of brain atrophy. Similarly, the higher rates of atrophy suggest that the aMCI group in ADNI consists of a higher proportion of subjects with a more aggressive disease than the MCSA. Indeed, the aMCI subjects in ADNI had a higher proportion of APOE e4 carriers than those in the MCSA in sample 1. Once again, the aMCI subjects in ADNI were more highly educated than those in the MCSA suggesting that cognitive reserve mechanisms may have protected them from decline on the MMSE and progression to a clinical diagnosis of AD.
We hypothesize that this bias in ADNI is due to the recruitment mechanism. We can speculate that CN subjects who are worried about their cognition would be more likely to attend memory clinics and be more motivated to answer advertisements for the study. Both CN and aMCI subjects with higher education are also more likely to seek medical help at a memory clinic and become involved in observational studies. These highly educated subjects could have a more aggressive underlying disease but are able to compensate cognitively. Amnestic MCI subjects recruited through a population-based study are less likely to have sought medical care at a memory clinic and may have a broader spectrum of cognitive function. In addition, an important motivator for participation in ADNI, and other convenience studies, could be the presence of a family history of dementia. Indeed, ADNI did show a higher proportion of family history compared to MCSA. Although one may assume that similar biases would be observed in the MCSA subjects that agreed to imaging, we have demonstrated that this is not the case; likely because less effort was required to agree for imaging than seek out participation in ADNI. The clinical inclusion criteria for both CN and aMCI differed slightly across the two cohorts. A diagnosis of CN in the MCSA was made by multidisciplinary consensus, which may be more conservative than the method employed in ADNI. Similarly, the diagnosis of aMCI in the MCSA is based on clinical grounds, whereas ADNI relied more on a specific cut-point on a memory test. The ADNI approach is likely to result in the recruitment of more impaired subjects. The reason that this is not reflected in the MMSE scores could be because higher education is providing cognitive reserve, and the MMSE may be insensitive to subtle cognitive impairment. The ADNI also recruited younger subjects than the MCSA, which could also have resulted in the recruitment of subjects with more aggressive disease. Rates of atrophy have been found to be greater in younger aMCI subjects32, possibly because they have a more pure, and hence aggressive, AD pathology compared to older subjects. Older subjects are more likely to have a mixture of pathologies33, including cerebrovascular disease34, 35. However, the trend for higher APOE e4 carrier frequency, younger age, and higher education in convenience samples compared to population-based samples has been observed in other cohorts36–39, suggesting that this bias may be due to the general recruitment mechanism rather than the specific inclusion criteria employed in ADNI. Our findings suggest that CN and aMCI subjects in ADNI are not representative of the general population, and, importantly, suggest that subjects included in future pre-clinical prevention trials using the same recruitment mechanisms will also not be representative of the population. Finally, our results indicate that even rigorous demographic matching efforts are insufficient to correct for the selection bias.
The only difference observed in baseline hippocampal volumes between ADNI and MCSA was in the CN subjects in sample 1, with larger hippocampal volumes observed in ADNI. This difference is likely being driven by the younger age of the ADNI cohort, since hippocampal volume has been shown to decrease with age40. After matching for demographic features no differences in hippocampal volume were observed across cohorts. Cross-sectional hippocampal volumes also did not differ across ADNI and the MCSA within the aMCI subjects in sample 1, despite the observed differences in age, education, APOE genotype, and MMSE score. This could suggest that rates of decline in hippocampal volume are more sensitive markers of incident AD than cross-sectional hippocampal volume, perhaps because of the large degree of inter-subject variability in hippocampal volume. TIV also differed between MCSA and ADNI. We suspect that MCSA subjects have larger TIVs because of the northern European heritage of many Minnesotan residents, and the link between these nationalities and tall height41.
The strengths of this study are the large numbers of subjects and the use of two samples with and without restrictive correction for major demographic or cognitive confounders. A limitation however, is that while matching was performed on the major demographic factors, it may not eliminate other potential differences, such as in other comorbidities, medication, family history, race and ethnicity, that may influence the imaging findings. ADNI had a higher proportion of minorities than MCSA. The MCSA and ADNI cohorts underwent imaging at different field strengths; however, we demonstrated excellent agreement between hippocampal volumes measured across field strengths (Figure 1). Scan intervals also differed between ADNI and MCSA, although we adjusted for these differences. While atrophy rates have been shown to accelerate over time in AD32, the trajectory of change is likely to be approximately linear over these relatively short intervals. Lastly, while the MCSA is a population-based study there may also be some inherent participation biases12, as is the case with any survey. The MCSA is however representative of Olmsted County in Minnesota, and of US Caucasians in general. The incidence of MCI and the demographic predictors of incident MCI in the MCSA are also similar to those reported in other population-based studies42–44, including studies that have assessed other racial groups45.
Overall, our findings show that subjects recruited into ADNI are not representative of the general population, and instead more closely resemble clinical populations. The imaging findings all point towards ADNI including more CN subjects that are on the path to AD dementia and more aMCI subjects that have a pure and aggressive disease phenotype. Therefore, convenience clinical series may be limited by selection biases. These findings have important implications for the design of future treatment trials. If studies that assess power calculations and sample size estimates are performed in biased convenience samples, the high rates of atrophy will lead to smaller than appropriate sample size estimates and therefore trials could be underpowered to detect treatment effects in the population. In addition, treatment trials that utilize a convenience sample will include a higher proportion of subjects with a pure and aggressive disease, and hence are more likely to detect a treatment effect. However, the magnitude of the treatment effect is likely to be less than expected when the treatment is applied to an unbiased population in which subjects are less likely to have pure AD. Care should also be taken when interpreting imaging studies from convenience samples, like ADNI. Biomarkers identified from these highly selected convenience-samples may not perfectly translate to the general population, and will need to be validated in a population-based sample. This will be particularly important for studies seeking to validate new diagnostic criteria for AD in its various stages, in which imaging biomarkers play an important role.
Investigators on this study were funded by the NIH (U01-AG024904-01, R01-AG11378, P50-AG16574, U01-AG06786, R21-AG38736, R01-DC010367, R01-AG037491, K01-AG028573, U24-AG026395, RO1-AG15866, R01-AG034676, R01-AG023195, R01-HL70825), the DHHS/OS [90BC0009], the Dana Foundation, the Alexander Family Alzheimer's Disease Research Professorship of the Mayo Foundation, and the Robert H. and Clarice Smith and Abigail Van Buren Alzheimer s Disease Research Program of the Mayo Foundation.
Data collection and sharing for this project was funded by the Alzheimer's Disease Neuroimaging Initiative (ADNI) (National Institutes of Health Grant U01-AG024904). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: Abbott, AstraZeneca AB, Bayer Schering Pharma AG, Bristol-Myers Squibb, Eisai Global Clinical Development, Elan Corporation, Genentech, GE Healthcare, GlaxoSmithKline, Innogenetics, Johnson and Johnson, Eli Lilly and Co., Medpace, Inc., Merck and Co., Inc., Novartis AG, Pfizer Inc, F. Hoffman-La Roche, Schering-Plough, Synarc, Inc., as well as non-profit partners the Alzheimer's Association and Alzheimer's Drug Discovery Foundation, with participation from the U.S. Food and Drug Administration. Private sector contributions to ADNI are facilitated by the Foundation for the National Institutes of Health (www.fnih.org). The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer's Disease Cooperative Study at the University of California, San Diego. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of California, Los Angeles. This research was also supported by NIH grants P30 AG010129, K01 AG030514, and the Dana Foundation.