PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Arch Clin Neuropsychol. Author manuscript; available in PMC 2007 April 2.
Published in final edited form as:
PMCID: PMC1847360
NIHMSID: NIHMS18851

Practice effects in the prediction of long-term cognitive outcome in three patient samples: A novel prognostic index

Abstract

Practice effects, defined as improvements in cognitive test performance due to repeated exposure to the test materials, have traditionally been viewed as sources of error. However, they might provide useful information for predicting cognitive outcome. The current study used three separate patient samples (older adults with mild cognitive impairments, individuals who were HIV +, individuals with Huntington’s disease) to examine the relationship between practice effects and cognitive functioning at a later point. Across all three samples, practice effects accounted for as much as 31 to 83% of the variance in the follow-up cognitive scores, after controlling for baseline cognitive functioning. If these findings can be replicated in other patients with neurodegenerative disorders, clinicians and researchers may be able to develop predictive models to identify the individuals who are most likely to demonstrate continued cognitive decline across time. The ability to utilize practice effects data would add a simple, convenient, and non-invasive marker for monitoring an individual patient’s cognitive status. Additionally, this prognostic index could be used to offer interventions to patients who are in the earliest stages of progressive neurodegenerative disorders.

Keywords: practice effects, cognitive outcome, Mild Cognitive Impairment, HIV, Huntington’s disease

Practice effects, defined as improvements in cognitive test performance due to repeated evaluation with the same test materials, have traditionally been viewed as sources of error variance rather than diagnostically useful information. Recently, however, it has been suggested that this psychometric phenomenon might prove useful in predicting cognitive outcome. For example, Darby et al. (2002) repeatedly administered a computerized battery of cognitive tasks to a group of patients with Mild Cognitive Impairment (MCI) and matched controls over the course of a single day. The results of this study indicated that healthy control participants improved across testing sessions, whereas individuals with MCI did not, even though the two groups were comparable at baseline testing. This short-term difference might have significant prognostic value. In another study, Newman et al. (2001) followed coronary-artery bypass surgery patients with serial neuropsychological testing and were able to predict their future cognitive status using the absence of practice effects between a baseline and a one-week follow-up evaluation. In that study, practice effects predicted cognitive status at 5-year follow-up.

Therefore, the purpose of the current study is to examine the relationship between the practice effects and later cognitive outcomes in patients diagnosed with one of three progressive, neurodegenerative disorders. If short-term changes in cognition (e.g., practice effects) are predictive of longer-term changes in cognition, then important decisions (e.g., initiating interventions, stopping driving, moving to a more dependent living situation) can be made sooner by the patient and his/her family.

Sample 1

Methods

Participants

Eight older adults (mean age = 72.4 [5.8] years, mean education = 15.9 [2.0] years, 2 males) who were enrolled in a treatment trial of a cholinesterase inhibitor provided data for the current study. Additional details about these individuals and the treatment trial are presented in Schultz et al. (2003). Briefly, all participants complained of memory problems and were determined to have mild cognitive impairments by a geriatric psychiatrist during an initial screening visit (Time 1).

Procedure

Approximately two weeks after the screening visit, all participants completed a baseline cognitive assessment (Time 2) and were randomized to placebo or medication groups. Most cognitive measures given in the screening and baseline visits were unique to that visit, but two tests were repeated: Brief Visuospatial Memory Test – Revised (BVMT-R, Benedict, 1997, Form 1), a measure of visual learning and memory, and the Mini Mental Status Examination (MMSE, Folstein et al., 1975), a screening measure of global cognitive functioning. The BVMT-R Total Recall score is the cumulative points earned across three learning trials; the MMSE Total score is the cumulative points earned across the measure. Although the BVMT-R has alternate forms, the same form was used throughout this study. Practice effects were calculated as the difference between scores at the screening and baseline visits for the BVMT-R (i.e., Time 2 BVMT-R Total Recall – Time 1 BVMT-R Total Recall) and the MMSE (i.e., Time 2 MMSE Total – Time 1 MMSE Total).

Participants were re-assessed at three and six months (Time 3 and 4, respectively) to look for changes in cognition. The follow-up battery included measures of global cognition (MMSE), memory (BVMT-R, Hopkins Verbal Learning Test – Revised [HVLT-R, Brandt et al., 1997]), psychomotor speed (Trail Making Test [TMT, Reitan, 1958] Parts A and B, Digit Symbol subtest of the Wechsler Adult Intelligence Scale – Revised [WAIS-R, Wechsler, 1981]), and language (30-item version of the Boston Naming Test [BNT, Goodglass et al., 1983], Controlled Oral Word Association Test [COWAT, Benton et al., 1983]).

Relationships between practice effects and follow-up cognitive scores were assessed with partial correlations (pr), controlling for baseline BVMT-R Total Recall. Partial correlations were used because we wanted to know the effect of practice on follow-up cognitive scores, above and beyond baseline cognition. The alpha level for statistical analyses was set at 0.05.

Results

Descriptive statistics for Times 1 – 4 scores, as well as the practice effects, are presented in Table 1.

Table 1
Descriptive statistics for Mild Cognitive Impairment sample.

Time 1 and Time 2 scores for the BVMT-R were moderately related (r[8] = .71, p < .05). The relationship between Time 1 and Time 3 scores on this measure, however, were not statistically significant (r[8] = .23, p = .58). Similarly, Time 1 and Time 4 were not correlated (r[8] = .39, p = .33).

When the BVMT-R Total Recall score from the Time 1 visit was controlled for, practice effects were significantly related to Delayed Recall of the BVMT-R at Time 3 (pr[5] = .86, p < .01, see Figure 1) and a strong trend was present for the BVMT-R Total Score at this time point (pr[5] = .68, p = .09). Practice effects were also related to Time 3 scores on another memory measure (HVLT-R Delayed Recall: pr[4] = .87, p < .05), with a modest trend on a measure of psychomotor speed (TMT-B: pr[5] = −.64, p = .12). At the 6-month follow-up (i.e., Time 4), HVLT-R Delayed Recall scores were significantly related to BVMT-R practice effects after removing the effects of Time 1 memory functioning (pr[5] = .89, p<.01). Trends were also present at the 6-month follow-up for other memory (BVMT-R Total Recall: pr[5] = .74, p = .06; BVMT-R Delayed Recall: pr[5] = .73, p = .06; HVLT-R Total Recall: pr[5] = .56, p = .19) and language (BNT: pr[5] = .55, p = .20; COWAT: pr[5] = .58, p = .17) measures.

Figure 1
Partial correlation of practice effects on BVMT-R Delayed Recall at Time 3, controlling for BVMT-R Total Recall at Time 1.

There were fewer relationships between practice effects on the MMSE and follow-up cognition performances. At the 3-month follow-up (i.e., Time 3), HVLT-R Delayed Recall scores were related to MMSE practice effects after removing the effects of Time 1 global functioning (pr[4] = .87, p<.05). Trends were present on other variables (COWAT at 3-months: pr[4] = .72, p = .11; HVLT-R Delayed Recall at 6-months: pr[5] = .67, p = .10; COWAT at 6-months: pr[5] = .57, p = .18), but fewer than seen with the BVMT-R practice effects.

Discussion

Despite small sample sizes and the medication confound, relationships were found between immediate practice effects and cognitive functioning at 3- and 6-month follow-up visits in this sample of older adults with mild cognitive problems. Although baseline memory testing (i.e., Time 1 BVMT-R Total Score) provided little information about follow-up memory performances (i.e., Times 3 and 4 scores), up to 79% of the variance of these follow-up measures was accounted for by practice effects after controlling for baseline levels of cognition. The relationship between practice effects and cognition was not solely confined to follow-up memory measures, but also provided information about future functioning on measures of psychomotor speed and language. Practice effects on memory measures (e.g., BVMT-R) appeared to provide more information than practice effects calculated from global measures of cognition (e.g., MMSE), although restriction of range likely limited the value of this latter measure. If these trends continue and can be replicated in other, well-defined samples, then earlier and more accurate identification of these individuals at-risk for cognitive decline can be obtained.

Sample 2

Methods

Participants

Thirty-three adults (mean age = 37.3 [8.8] years, mean education = 15.2 [2.3], 30 males) who were participating in a study examining the neuropsychological effects of HIV provided data for the current study. Additional details about these individuals are presented elsewhere (Duff et al., 2001; McCaffrey et al., 1995). Briefly, two groups of HIV positive participants were recruited: clinically symptomatic or clinically asymptomatic.

Procedure

All participants were administered the Brief NIMH Neuropsychological Battery for HIV Infection and AIDS (Butters et al., 1990), which includes the Paced Auditory Serial Addition Test (PASAT, Gronwall, 1977), California Verbal Learning Test (CVLT, Delis et al., 1987), Visual Search Test (VST, Rennick, 1979), and Choice Reaction Time (CRT). Participants were administered the battery twice within two weeks (Times 1 and 2), and then at 6- and 12-months (Times 3 and 4). Only participants who had completed these four assessments were included in the current study. Since there were no significant group differences on any of demographic or cognitive test data at the first testing point, the two groups were combined for the following analyses. Practice effects were calculated for each cognitive test as the difference between the first two administrations of that test (e.g., Time 2 PASAT – Time 1 PASAT; Time 2 CVLT – Time 1 CVLT). Relationships between practice effects and follow-up cognitive scores were assessed with partial correlations, controlling for initial cognitive test performance. The alpha level for statistical analyses was set at 0.05.

Results

Descriptive statistics for Times 1 – 4 scores, as well as the practice effects, are presented in Table 2.

Table 2
Descriptive statistics for HIV+ sample.

Time 1 and Time 2 scores for the PASAT were significantly related (r[33] = .61, p < .01). The relationship between Time 1 and Time 3 scores on this measure were more modestly related (r[33] = .47, p <.01). Similarly, Time 1 and Time 4 were modestly correlated (r[33] = .49, p <.01).

PASAT practice effects were significantly related to Time 3 PASAT scores (pr[30] = .90, p<.05) and Time 4 PASAT scores (pr[30] = .91, p<.05, see Figure 2) after controlling for Time 1 PASAT scores. When Time 1 CVLT scores were partialled out, CVLT practice effects were significantly correlated with Time 3 CVLT scores (pr[30] = .61, p<.05) and Time 4 CVLT scores (pr[30] = .54, p<.05). Controlling for Time 1 CRT scores, practice effects were significantly related to Time 3 CRT scores (pr[30] = .48, p<.05), with a trend towards correlating with Time 4 CRT scores (pr[30] = .28, p = .12). VST practice effects also demonstrated trends of relationships with respective follow-up scores (Time 3 VST: pr[30] = .25, p = .17; Time 4 VST: pr[30] = .33, p = .06) when partialling out Time 1 VST scores.

Figure 2
Partial correlation of practice effects on PASAT at Time 4, controlling for PASAT at Time 1.

Discussion

Similar to the findings with the first sample, initial practice effects (baseline and two week retest interval) were related to cognitive functioning over the following year in this group of HIV+ individuals. The strongest relationship was found between the practice effects data and the one-year scores on the PASAT, a speeded measure of working memory, accounting for up to 83% of the shared variance. This finding is in line with the underlying pathology known to accompany this disease (i.e., subcortical dysfunction) and the cognitive deficits associated with that pathology (i.e., decreased processing speed). If these findings are supported in other independent investigations, then differential practice effects could be expected based on the underlying disease, which could have differential relationships with long-term cognitive outcome.

Sample 3

Methods

Participants

One hundred seventy adults (mean age = 48.2 [11.8] years, mean education = 13.4 [3.0], 86 males) who were participating in a longitudinal clinical study examining the motor, cognitive, and psychiatric effects of Huntington’s disease (HD) provided data for the current study. Additional details about these individuals are presented elsewhere (Paulsen et al., 2001). Briefly, all participants had been previously diagnosed with HD, most were rated as mildly impaired (i.e., Total Functioning Capacity scores of 7 or greater [Shoulson & Fahn, 1979]), and all were followed clinically on an “as needed” basis.

Procedure

During each clinical visit, participants completed the Unified Huntington’s Disease Rating Scale (Huntington Study Group, 1996), which includes three cognitive tests: 1) Symbol Digit, 2) Stroop Color Word Test, and 3) verbal fluency. Practice effects were calculated for each cognitive test as the difference between the first two administrations of that test (e.g., Time 2 Symbol Digit – Time 1 Symbol Digit; Time 2 verbal fluency – Time 1 verbal fluency). The amount of time between Time 1 and Time 2 averaged 220 days (SD = 122.0), and ranged from 26 – 731. Relationships between practice effects and follow-up cognitive scores were assessed with partial correlations, controlling for initial cognitive test performance. The amount of time between Time 2 and the follow-up point, Time 3, averaged 890.4 days (SD = 273.1), and ranged from 153 – 1690. Given the variability in retest intervals for these patients, it was included as an additional covariate in the partial correlations, but it did not significantly change the findings and is not reported below. The alpha level for statistical analyses was set at 0.05.

Results

Descriptive statistics for Times 1 – 3 scores, as well as the practice effects, are presented in Table 3.

Table 3
Descriptive statistics for Huntington’s disease sample.

Time 1 and Time 2 scores for the Symbol Digit were strongly correlated (r[161] = .88, p < .01), as was the relationship between Time 1 and Time 3 scores on this measure (r[164] = .77, p <.01).

Symbol Digit practice effects were significantly related to Time 3 Symbol Digit scores (pr[158] = .56, p<.01, see Figure 3) after controlling for Time 1 Symbol Digit scores. Symbol Digit practice effects were also related to other cognitive scores, but to a lesser degree (verbal fluency: pr[158] = .29, p<.01; Stroop Interference: pr[154] = .37, p<.01). Similarly, after partialling out the effects of Time 1 verbal fluency, the practice effects of verbal fluency were more highly related to Time 3 verbal fluency scores (pr[165] = .51, p<.01) than they were to the other cognitive scores (Symbol Digit: pr[165] = .36, p<.01; Stroop Interference: pr[161] = .25, p<.01). Finally, Stroop Interference practice effects were significantly related to Time 3 Stroop Interference scores (pr[157] = .36, p<.01) after controlling for Time 1 Stroop Interference scores. Stroop practice effects were also related to other cognitive scores (verbal fluency: pr[160] = .25, p<.05; Symbol Digit: pr[160] = .19, p<.05).

Figure 3
Partial correlation of practice effects on Digit Symbol at Time 3, controlling for Digit Symbol at Time 1.

Discussion

Consistent with the other samples, changes in test performance on repeated assessments were related to long-term cognitive performance in this large sample of patients with manifest HD. Digit Symbol showed the strongest relationship between its practice effects and 29 month follow-up (sharing 31% of the variance), possibly because of its more prominent motor component than the other UHDRS cognitive tests and the underlying neuropathology of HD. In this sample, a more liberal definition of practice effects was used, considering changes across an average of 7 months. Similarly, the follow-up points were considerably longer than used with the other two samples. Comparing the results across the three samples, the findings in the HD sample were attenuated; possibly because of these longer retest intervals.

General Discussion

Across all three samples, practice effects were related to cognitive performances at some later point, sharing as much as 31 to 83% of the variance, even after controlling for baseline cognitive functioning. This relationship was robust, occurring in three different patient samples, using different cognitive measures, and across varying practice effects and follow-up intervals. If this finding can be replicated in other samples, practice effects have the potential to provide prognostically useful information to clinicians and researchers, months or years earlier than is currently ascertained.

This series of analyses also demonstrated that the clinical utility of practice effects might provide some specific across different cognitive domains and patient samples. For example, in primarily memory disorders, such as the participants in Sample 1, practice effects on the memory measure provided the most useful information about follow-up cognition than did practice effects on a measure of global cognition. Conversely, practice effects on the memory measure was not the best predictor of cognitive outcome in the participants with HIV; practice effects on a processing speed measure was. As noted earlier, the differential relationships could be due to underlying disease and cognitive profiles of the diseases. HIV and AIDS can lead to a “subcortical dementia,” which is characterized by slowed processing speed (Grant et al., 1994). Therefore, practice effects on the PASAT might be the most informative measure of subtle processing speed deficits. Similarly, the Symbol Digit practice effects by the HD participants in Sample 3 might have been the most valuable measure of psychomotor decline in this disorder of motor, cognition, and psychiatric functioning. The findings for the HD sample are consistent with those of Paulsen et al. (2001), who reported that longitudinal change scores were useful in identifying individuals who were likely to develop manifest HD over a 2 year period. It should be noted that this apparent specificity of practice effects across different cognitive measures and patient samples is preliminary and needs to be replicated. In addition to other limitations in the current studies (see below), comprehensive neuropsychological evaluations were not administered to all patient samples. For example, in the MCI sample, a limited battery (BVMT-R and MMSE) was used to quantify practice effects, whereas other measures that were not given repeatedly (e.g., TMT, COWAT) might have led to stronger predictions of future cognition. Nonetheless, future studies could target practice effects in different cognitive domains based on expected neuropathology and neuropsychological profiles. In these studies, more comprehensive assessments might truly test this hypothesis of specificity of practice effects.

The current findings do not necessarily suggest that practice effects are the preferred method of predicting future cognition in these patient samples. Several studies (Duff et al., 2004; Duff et al., 2005; Hermann et al., 1996; McSweeny et al., 1993; Sawrie et al., 1996) indicate that initial cognitive performances are the best predictor of future cognitive performances. For example, in Sample 2, Time 1 PASAT accounts for 24% of the variance of Time 4 PASAT scores. Practice effects of PASAT accounts for only 1% of the variance of Time 4 PASAT. When, however, the shared variance of Time 1 PASAT is removed from practice effects and Time 4, these latter two scores share 83% of the remaining variance. This seems to indicate that initial cognitive performance provides a lot of information about follow-up performances, but practice effects significantly contributes unique information. The unique contribution of practice effects might be particularly salient in patient populations where some decline in functioning is expected.

Within the current study, practice effects were calculated as the difference between Times 1 and 2, but other methods are available. For example, Time 2 scores could have been used instead of practice effects, since Time 2 scores are roughly comprised of baseline functioning and practice effects. Indeed, if Time 2 scores had been used in the current analyses, the results would have been identical. Additionally, the Reliable Change Index (Jacobson & Traux, 1991) or regression-based change formulas (McSweeny et al., 1993) are other options since these formulas can account for additional variables known to affect change (e.g., “normal” practice effects, age, education, retest intervals; McCaffrey et al., 2000). We choose our method of simple subtraction because it is intuitively appealing and clinically useful. We are not, however, advocating for one change method over another, but suggest that any change method will provide information about cognitive outcome, above and beyond what is known from baseline functioning. Future studies might compare these different methods to see which provides the best estimates of future cognition.

Several limitations of the current studies should be noted. These three studies all have weaknesses in their patient characterization and/or methods. The participants in Sample 1, for example, presented with memory complaints and mild cognitive impairments on testing, but did not fulfill the diagnostic criteria for MCI-amnestic type (Petersen et al., 1999). Additionally, half the group was randomized to either a cholinesterase inhibitor or placebo after visit 2. Participants in Sample 2 were all HIV positive, but some were symptomatic for AIDS and others were not. Additionally, other clinical information about their disease (e.g., CD4 count, viral load) was not known. Finally, the participants in Sample 3 were all diagnosed with HD, but individuals were at different points in their illness. Therefore, the generalizability of this information to other samples is unclear, and further investigation of practice effects to “cleaner” samples of MCI, HIV, and HD is needed. Nonetheless, the robustness of this finding across these three imperfect samples is encouraging.

In conclusion, practice effects, frequently considered to be error that needs to be minimized, might hold valuable information for clinicians and researchers about cognitive outcomes in a variety of patient samples. Practice effects, as a simple, convenient, and non-invasive marker for monitoring an individual patient’s cognitive status, would have the added benefit of perhaps being able to offer interventions to patients who are in the earliest stages of progressive neurodegenerative disorders.

Acknowledgments

This research is supported by the National Institutes of Health (NIMH MH61801, NIMH MH48639, NS440068 and MH01579), The High Q Foundation, the Huntington’s Disease Society of America, the Huntington’s Society of Canada, and the Hereditary Disease Foundation to the Huntington Study Group.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  • Benedict RHB. Brief Visuospatial Memory Test-Revised. Odessa, FL: Psychological Assessment Resources, Inc; 1997.
  • Benton AL, Hamsher K. Multilingual Aphasia Examination. Iowa City, IA: AJA Associates Inc; 1983.
  • Brandt J, Benedict RHB. Hopkins Verbal Learning Test-Revised. Odessa, FL: Psychological Assessment Resources, Inc; 1997.
  • Butters N, Grant I, Haxby J, Judd LL, Martin A, McClelland J, Pequegant W, Schacter D, Stover E. Assessment of AIDS-related cognitive changes: Recommendations of the NIMH workshop on neuropsychological assessment approaches. Journal of Clinical and Experimental Neuropsychology. 1990;12:963–978. [PubMed]
  • Darby D, Maruff P, Collie A, McStephen M. Mild cognitive impairment can be detected by multiple assessments in a single day. Neurology. 2002;59:1042–1046. [PubMed]
  • Delis DC, Kramer JH, Kaplan E, Ober BA. California Verbal Learning Test – Research Edition Manual. San Antonio, TX: The Psychological Corporation; 1987.
  • Dikmen SS, Heaton RK, Grant I, Temkin NR. Test-retest reliability and practice effects of Expanded Halstead-Reitan Neuropsychological Test Battery. Journal of the International Neuropsychological Society. 1999;5:346–356. [PubMed]
  • Duff K, Schoenberg MR, Patton D, Mold J, Scott JG, Adams RL. Predicting change with the RBANS in a community dwelling elderly sample. Journal of the International Neuropsychological Society. 2004;10:828–834. [PubMed]
  • Duff K, Schoenberg MR, Patton D, Paulsen JS, Bayless J, Mold J, Scott JG, Adams RL. Regression-based formulas for predicting change in RBANS subtests with older adults. Archives of Clinical Neuropsychology. 2005;20:281–290. [PubMed]
  • Duff K, Westervelt HJ, McCaffrey RJ, Haase RF. Practice effects, test-retest stability, and dual baseline assessments with the California Verbal Learning Test in an HIV sample. Archives of Clinical Neuropsychology. 2001;16:461–476. [PubMed]
  • Folstein MF, Folstein SE, McHugh PR. “Mini Mental State”: A practical guide of grading the cognitive state of patients for the clinician. Journal of Psychiatric Research. 1975;12:189–198. [PubMed]
  • Goodglass H, Kaplan E. Boston Diagnostic Aphasia Examination. Philadelphia: Lea & Febiger; 1983.
  • Grant I, Martin A. Neuropsychology of HIV Infection. New York: Oxford University Press; 1994.
  • Gronwall DMA. Paced auditory serial-addition task: A measure of recovery from concussion. Perceptual and Motor Skills. 1977;44:367–373. [PubMed]
  • Hermann BP, Seidenberg M, Schoenfeld J, Peterson J, Leveroni C, Wyler AR. Empirical techniques for determining the reliability, magnitude, and pattern of neuropsychological change after epilepsy surgery. Epilepsia. 1996;37:942–950. [PubMed]
  • Huntington Study Group. Unified Huntington's Disease Rating Scale: Reliability and consistency. Movement Disorders. 1996;11:136–142. [PubMed]
  • McCaffrey RJ, Cousins JP, Westervelt HJ, Martynowicz M, Remick SC, Szebenyi S, Wagle WA, Bottomley PA, Hardy CJ, Haase RF. Practice effects with the NIMH AIDS abbreviated neuropsychological battery. Archives of Clinical Neuropsychology. 1995;10:241–250. [PubMed]
  • McCaffrey RJ, Duff K, Westervelt HJ. Practitioner’s Guide to Evaluating Change with Neuropsychological Assessment Instruments. New York: Kluwer Academic/Plenum; 2000.
  • McSweeny AJ, Naugle RI, Chelune GJ, Luders H. “T scores for change”: An illustration of a regression approach to depicting change in clinical neuropsychology. The Clinical Neuropsychologist. 1993;7:300–312.
  • Newman MF, Kirchner JL, Phillips-Bute B, Gaver V, Grocott H, Hones RH, Mark DB, Reves JG, Blumenthal JA. Longitudinal assessment of neurocognitive function after coronary-artery bypass surgery. The New England Journal of Medicine. 2001;344:395–402. [PubMed]
  • Paulsen JS, Zhao H, Stout JC, Brinkman RR, Guttman M, Ross CA, Como P, Manning C, Hayden MR, Shoulson I. the Huntingon Study Group. Clinical markers of early disease in persons near onset of Huntington’s disease. Neurology. 2001;57:658–662. [PubMed]
  • Petersen RC, Smith GE, Waring SC, Ivnik RJ, Tangalos EG, Kokmen E. Mild cognitive impairment: Clinical characterization and outcome. Archives of Neurology. 1999;56:303–308. [PubMed]
  • Reitan RM. Validity of the Trail Making Test as an indicator of organic brain damage. Perceptual and Motor Skills. 1958;8:271–276.
  • Rennick PM. Color Naming and Visual Search Tests for Repeatable Cognitive-Perceptual Motor Battery. Grosse Point Park, MI: Axon Publishing; 1979.
  • Sawrie SM, Chelune GJ, Naugle RI, Luders HO. Empirical methods for assessing meaningful neuropsychological change following epilepsy surgery. Journal of the International Neuropsychological Society. 1996;2:556–564. [PubMed]
  • Schultz SK, Boles Ponto L, Moser DJ, Magnotta V, Watkins GL, Hichwa R. Cerebrovascular Reserve Capacity in Mild Cognitive Impairment Assessed by [15O]water Positron Emission Tomography. Presentation made at the annual meeting of the American College of Neuropsychopharmacology; San Juan, Puerto Rico. 2003.
  • Shoulson I, Fahn S. Huntington disease: clinical care and evaluation. Neurology. 1979;29:1–3. [PubMed]