|Home | About | Journals | Submit | Contact Us | Français|
Practice effects, defined as improvements in cognitive test performance due to repeated exposure to the test materials, have traditionally been viewed as sources of error. However, they might provide useful information for predicting cognitive outcome. The current study used three separate patient samples (older adults with mild cognitive impairments, individuals who were HIV +, individuals with Huntington’s disease) to examine the relationship between practice effects and cognitive functioning at a later point. Across all three samples, practice effects accounted for as much as 31 to 83% of the variance in the follow-up cognitive scores, after controlling for baseline cognitive functioning. If these findings can be replicated in other patients with neurodegenerative disorders, clinicians and researchers may be able to develop predictive models to identify the individuals who are most likely to demonstrate continued cognitive decline across time. The ability to utilize practice effects data would add a simple, convenient, and non-invasive marker for monitoring an individual patient’s cognitive status. Additionally, this prognostic index could be used to offer interventions to patients who are in the earliest stages of progressive neurodegenerative disorders.
Practice effects, defined as improvements in cognitive test performance due to repeated evaluation with the same test materials, have traditionally been viewed as sources of error variance rather than diagnostically useful information. Recently, however, it has been suggested that this psychometric phenomenon might prove useful in predicting cognitive outcome. For example, Darby et al. (2002) repeatedly administered a computerized battery of cognitive tasks to a group of patients with Mild Cognitive Impairment (MCI) and matched controls over the course of a single day. The results of this study indicated that healthy control participants improved across testing sessions, whereas individuals with MCI did not, even though the two groups were comparable at baseline testing. This short-term difference might have significant prognostic value. In another study, Newman et al. (2001) followed coronary-artery bypass surgery patients with serial neuropsychological testing and were able to predict their future cognitive status using the absence of practice effects between a baseline and a one-week follow-up evaluation. In that study, practice effects predicted cognitive status at 5-year follow-up.
Therefore, the purpose of the current study is to examine the relationship between the practice effects and later cognitive outcomes in patients diagnosed with one of three progressive, neurodegenerative disorders. If short-term changes in cognition (e.g., practice effects) are predictive of longer-term changes in cognition, then important decisions (e.g., initiating interventions, stopping driving, moving to a more dependent living situation) can be made sooner by the patient and his/her family.
Eight older adults (mean age = 72.4 [5.8] years, mean education = 15.9 [2.0] years, 2 males) who were enrolled in a treatment trial of a cholinesterase inhibitor provided data for the current study. Additional details about these individuals and the treatment trial are presented in Schultz et al. (2003). Briefly, all participants complained of memory problems and were determined to have mild cognitive impairments by a geriatric psychiatrist during an initial screening visit (Time 1).
Approximately two weeks after the screening visit, all participants completed a baseline cognitive assessment (Time 2) and were randomized to placebo or medication groups. Most cognitive measures given in the screening and baseline visits were unique to that visit, but two tests were repeated: Brief Visuospatial Memory Test – Revised (BVMT-R, Benedict, 1997, Form 1), a measure of visual learning and memory, and the Mini Mental Status Examination (MMSE, Folstein et al., 1975), a screening measure of global cognitive functioning. The BVMT-R Total Recall score is the cumulative points earned across three learning trials; the MMSE Total score is the cumulative points earned across the measure. Although the BVMT-R has alternate forms, the same form was used throughout this study. Practice effects were calculated as the difference between scores at the screening and baseline visits for the BVMT-R (i.e., Time 2 BVMT-R Total Recall – Time 1 BVMT-R Total Recall) and the MMSE (i.e., Time 2 MMSE Total – Time 1 MMSE Total).
Participants were re-assessed at three and six months (Time 3 and 4, respectively) to look for changes in cognition. The follow-up battery included measures of global cognition (MMSE), memory (BVMT-R, Hopkins Verbal Learning Test – Revised [HVLT-R, Brandt et al., 1997]), psychomotor speed (Trail Making Test [TMT, Reitan, 1958] Parts A and B, Digit Symbol subtest of the Wechsler Adult Intelligence Scale – Revised [WAIS-R, Wechsler, 1981]), and language (30-item version of the Boston Naming Test [BNT, Goodglass et al., 1983], Controlled Oral Word Association Test [COWAT, Benton et al., 1983]).
Relationships between practice effects and follow-up cognitive scores were assessed with partial correlations (pr), controlling for baseline BVMT-R Total Recall. Partial correlations were used because we wanted to know the effect of practice on follow-up cognitive scores, above and beyond baseline cognition. The alpha level for statistical analyses was set at 0.05.
Descriptive statistics for Times 1 – 4 scores, as well as the practice effects, are presented in Table 1.
Time 1 and Time 2 scores for the BVMT-R were moderately related (r = .71, p < .05). The relationship between Time 1 and Time 3 scores on this measure, however, were not statistically significant (r = .23, p = .58). Similarly, Time 1 and Time 4 were not correlated (r = .39, p = .33).
When the BVMT-R Total Recall score from the Time 1 visit was controlled for, practice effects were significantly related to Delayed Recall of the BVMT-R at Time 3 (pr = .86, p < .01, see Figure 1) and a strong trend was present for the BVMT-R Total Score at this time point (pr = .68, p = .09). Practice effects were also related to Time 3 scores on another memory measure (HVLT-R Delayed Recall: pr = .87, p < .05), with a modest trend on a measure of psychomotor speed (TMT-B: pr = −.64, p = .12). At the 6-month follow-up (i.e., Time 4), HVLT-R Delayed Recall scores were significantly related to BVMT-R practice effects after removing the effects of Time 1 memory functioning (pr = .89, p<.01). Trends were also present at the 6-month follow-up for other memory (BVMT-R Total Recall: pr = .74, p = .06; BVMT-R Delayed Recall: pr = .73, p = .06; HVLT-R Total Recall: pr = .56, p = .19) and language (BNT: pr = .55, p = .20; COWAT: pr = .58, p = .17) measures.
There were fewer relationships between practice effects on the MMSE and follow-up cognition performances. At the 3-month follow-up (i.e., Time 3), HVLT-R Delayed Recall scores were related to MMSE practice effects after removing the effects of Time 1 global functioning (pr = .87, p<.05). Trends were present on other variables (COWAT at 3-months: pr = .72, p = .11; HVLT-R Delayed Recall at 6-months: pr = .67, p = .10; COWAT at 6-months: pr = .57, p = .18), but fewer than seen with the BVMT-R practice effects.
Despite small sample sizes and the medication confound, relationships were found between immediate practice effects and cognitive functioning at 3- and 6-month follow-up visits in this sample of older adults with mild cognitive problems. Although baseline memory testing (i.e., Time 1 BVMT-R Total Score) provided little information about follow-up memory performances (i.e., Times 3 and 4 scores), up to 79% of the variance of these follow-up measures was accounted for by practice effects after controlling for baseline levels of cognition. The relationship between practice effects and cognition was not solely confined to follow-up memory measures, but also provided information about future functioning on measures of psychomotor speed and language. Practice effects on memory measures (e.g., BVMT-R) appeared to provide more information than practice effects calculated from global measures of cognition (e.g., MMSE), although restriction of range likely limited the value of this latter measure. If these trends continue and can be replicated in other, well-defined samples, then earlier and more accurate identification of these individuals at-risk for cognitive decline can be obtained.
Thirty-three adults (mean age = 37.3 [8.8] years, mean education = 15.2 [2.3], 30 males) who were participating in a study examining the neuropsychological effects of HIV provided data for the current study. Additional details about these individuals are presented elsewhere (Duff et al., 2001; McCaffrey et al., 1995). Briefly, two groups of HIV positive participants were recruited: clinically symptomatic or clinically asymptomatic.
All participants were administered the Brief NIMH Neuropsychological Battery for HIV Infection and AIDS (Butters et al., 1990), which includes the Paced Auditory Serial Addition Test (PASAT, Gronwall, 1977), California Verbal Learning Test (CVLT, Delis et al., 1987), Visual Search Test (VST, Rennick, 1979), and Choice Reaction Time (CRT). Participants were administered the battery twice within two weeks (Times 1 and 2), and then at 6- and 12-months (Times 3 and 4). Only participants who had completed these four assessments were included in the current study. Since there were no significant group differences on any of demographic or cognitive test data at the first testing point, the two groups were combined for the following analyses. Practice effects were calculated for each cognitive test as the difference between the first two administrations of that test (e.g., Time 2 PASAT – Time 1 PASAT; Time 2 CVLT – Time 1 CVLT). Relationships between practice effects and follow-up cognitive scores were assessed with partial correlations, controlling for initial cognitive test performance. The alpha level for statistical analyses was set at 0.05.
Descriptive statistics for Times 1 – 4 scores, as well as the practice effects, are presented in Table 2.
Time 1 and Time 2 scores for the PASAT were significantly related (r = .61, p < .01). The relationship between Time 1 and Time 3 scores on this measure were more modestly related (r = .47, p <.01). Similarly, Time 1 and Time 4 were modestly correlated (r = .49, p <.01).
PASAT practice effects were significantly related to Time 3 PASAT scores (pr = .90, p<.05) and Time 4 PASAT scores (pr = .91, p<.05, see Figure 2) after controlling for Time 1 PASAT scores. When Time 1 CVLT scores were partialled out, CVLT practice effects were significantly correlated with Time 3 CVLT scores (pr = .61, p<.05) and Time 4 CVLT scores (pr = .54, p<.05). Controlling for Time 1 CRT scores, practice effects were significantly related to Time 3 CRT scores (pr = .48, p<.05), with a trend towards correlating with Time 4 CRT scores (pr = .28, p = .12). VST practice effects also demonstrated trends of relationships with respective follow-up scores (Time 3 VST: pr = .25, p = .17; Time 4 VST: pr = .33, p = .06) when partialling out Time 1 VST scores.
Similar to the findings with the first sample, initial practice effects (baseline and two week retest interval) were related to cognitive functioning over the following year in this group of HIV+ individuals. The strongest relationship was found between the practice effects data and the one-year scores on the PASAT, a speeded measure of working memory, accounting for up to 83% of the shared variance. This finding is in line with the underlying pathology known to accompany this disease (i.e., subcortical dysfunction) and the cognitive deficits associated with that pathology (i.e., decreased processing speed). If these findings are supported in other independent investigations, then differential practice effects could be expected based on the underlying disease, which could have differential relationships with long-term cognitive outcome.
One hundred seventy adults (mean age = 48.2 [11.8] years, mean education = 13.4 [3.0], 86 males) who were participating in a longitudinal clinical study examining the motor, cognitive, and psychiatric effects of Huntington’s disease (HD) provided data for the current study. Additional details about these individuals are presented elsewhere (Paulsen et al., 2001). Briefly, all participants had been previously diagnosed with HD, most were rated as mildly impaired (i.e., Total Functioning Capacity scores of 7 or greater [Shoulson & Fahn, 1979]), and all were followed clinically on an “as needed” basis.
During each clinical visit, participants completed the Unified Huntington’s Disease Rating Scale (Huntington Study Group, 1996), which includes three cognitive tests: 1) Symbol Digit, 2) Stroop Color Word Test, and 3) verbal fluency. Practice effects were calculated for each cognitive test as the difference between the first two administrations of that test (e.g., Time 2 Symbol Digit – Time 1 Symbol Digit; Time 2 verbal fluency – Time 1 verbal fluency). The amount of time between Time 1 and Time 2 averaged 220 days (SD = 122.0), and ranged from 26 – 731. Relationships between practice effects and follow-up cognitive scores were assessed with partial correlations, controlling for initial cognitive test performance. The amount of time between Time 2 and the follow-up point, Time 3, averaged 890.4 days (SD = 273.1), and ranged from 153 – 1690. Given the variability in retest intervals for these patients, it was included as an additional covariate in the partial correlations, but it did not significantly change the findings and is not reported below. The alpha level for statistical analyses was set at 0.05.
Descriptive statistics for Times 1 – 3 scores, as well as the practice effects, are presented in Table 3.
Time 1 and Time 2 scores for the Symbol Digit were strongly correlated (r = .88, p < .01), as was the relationship between Time 1 and Time 3 scores on this measure (r = .77, p <.01).
Symbol Digit practice effects were significantly related to Time 3 Symbol Digit scores (pr = .56, p<.01, see Figure 3) after controlling for Time 1 Symbol Digit scores. Symbol Digit practice effects were also related to other cognitive scores, but to a lesser degree (verbal fluency: pr = .29, p<.01; Stroop Interference: pr = .37, p<.01). Similarly, after partialling out the effects of Time 1 verbal fluency, the practice effects of verbal fluency were more highly related to Time 3 verbal fluency scores (pr = .51, p<.01) than they were to the other cognitive scores (Symbol Digit: pr = .36, p<.01; Stroop Interference: pr = .25, p<.01). Finally, Stroop Interference practice effects were significantly related to Time 3 Stroop Interference scores (pr = .36, p<.01) after controlling for Time 1 Stroop Interference scores. Stroop practice effects were also related to other cognitive scores (verbal fluency: pr = .25, p<.05; Symbol Digit: pr = .19, p<.05).
Consistent with the other samples, changes in test performance on repeated assessments were related to long-term cognitive performance in this large sample of patients with manifest HD. Digit Symbol showed the strongest relationship between its practice effects and 29 month follow-up (sharing 31% of the variance), possibly because of its more prominent motor component than the other UHDRS cognitive tests and the underlying neuropathology of HD. In this sample, a more liberal definition of practice effects was used, considering changes across an average of 7 months. Similarly, the follow-up points were considerably longer than used with the other two samples. Comparing the results across the three samples, the findings in the HD sample were attenuated; possibly because of these longer retest intervals.
Across all three samples, practice effects were related to cognitive performances at some later point, sharing as much as 31 to 83% of the variance, even after controlling for baseline cognitive functioning. This relationship was robust, occurring in three different patient samples, using different cognitive measures, and across varying practice effects and follow-up intervals. If this finding can be replicated in other samples, practice effects have the potential to provide prognostically useful information to clinicians and researchers, months or years earlier than is currently ascertained.
This series of analyses also demonstrated that the clinical utility of practice effects might provide some specific across different cognitive domains and patient samples. For example, in primarily memory disorders, such as the participants in Sample 1, practice effects on the memory measure provided the most useful information about follow-up cognition than did practice effects on a measure of global cognition. Conversely, practice effects on the memory measure was not the best predictor of cognitive outcome in the participants with HIV; practice effects on a processing speed measure was. As noted earlier, the differential relationships could be due to underlying disease and cognitive profiles of the diseases. HIV and AIDS can lead to a “subcortical dementia,” which is characterized by slowed processing speed (Grant et al., 1994). Therefore, practice effects on the PASAT might be the most informative measure of subtle processing speed deficits. Similarly, the Symbol Digit practice effects by the HD participants in Sample 3 might have been the most valuable measure of psychomotor decline in this disorder of motor, cognition, and psychiatric functioning. The findings for the HD sample are consistent with those of Paulsen et al. (2001), who reported that longitudinal change scores were useful in identifying individuals who were likely to develop manifest HD over a 2 year period. It should be noted that this apparent specificity of practice effects across different cognitive measures and patient samples is preliminary and needs to be replicated. In addition to other limitations in the current studies (see below), comprehensive neuropsychological evaluations were not administered to all patient samples. For example, in the MCI sample, a limited battery (BVMT-R and MMSE) was used to quantify practice effects, whereas other measures that were not given repeatedly (e.g., TMT, COWAT) might have led to stronger predictions of future cognition. Nonetheless, future studies could target practice effects in different cognitive domains based on expected neuropathology and neuropsychological profiles. In these studies, more comprehensive assessments might truly test this hypothesis of specificity of practice effects.
The current findings do not necessarily suggest that practice effects are the preferred method of predicting future cognition in these patient samples. Several studies (Duff et al., 2004; Duff et al., 2005; Hermann et al., 1996; McSweeny et al., 1993; Sawrie et al., 1996) indicate that initial cognitive performances are the best predictor of future cognitive performances. For example, in Sample 2, Time 1 PASAT accounts for 24% of the variance of Time 4 PASAT scores. Practice effects of PASAT accounts for only 1% of the variance of Time 4 PASAT. When, however, the shared variance of Time 1 PASAT is removed from practice effects and Time 4, these latter two scores share 83% of the remaining variance. This seems to indicate that initial cognitive performance provides a lot of information about follow-up performances, but practice effects significantly contributes unique information. The unique contribution of practice effects might be particularly salient in patient populations where some decline in functioning is expected.
Within the current study, practice effects were calculated as the difference between Times 1 and 2, but other methods are available. For example, Time 2 scores could have been used instead of practice effects, since Time 2 scores are roughly comprised of baseline functioning and practice effects. Indeed, if Time 2 scores had been used in the current analyses, the results would have been identical. Additionally, the Reliable Change Index (Jacobson & Traux, 1991) or regression-based change formulas (McSweeny et al., 1993) are other options since these formulas can account for additional variables known to affect change (e.g., “normal” practice effects, age, education, retest intervals; McCaffrey et al., 2000). We choose our method of simple subtraction because it is intuitively appealing and clinically useful. We are not, however, advocating for one change method over another, but suggest that any change method will provide information about cognitive outcome, above and beyond what is known from baseline functioning. Future studies might compare these different methods to see which provides the best estimates of future cognition.
Several limitations of the current studies should be noted. These three studies all have weaknesses in their patient characterization and/or methods. The participants in Sample 1, for example, presented with memory complaints and mild cognitive impairments on testing, but did not fulfill the diagnostic criteria for MCI-amnestic type (Petersen et al., 1999). Additionally, half the group was randomized to either a cholinesterase inhibitor or placebo after visit 2. Participants in Sample 2 were all HIV positive, but some were symptomatic for AIDS and others were not. Additionally, other clinical information about their disease (e.g., CD4 count, viral load) was not known. Finally, the participants in Sample 3 were all diagnosed with HD, but individuals were at different points in their illness. Therefore, the generalizability of this information to other samples is unclear, and further investigation of practice effects to “cleaner” samples of MCI, HIV, and HD is needed. Nonetheless, the robustness of this finding across these three imperfect samples is encouraging.
In conclusion, practice effects, frequently considered to be error that needs to be minimized, might hold valuable information for clinicians and researchers about cognitive outcomes in a variety of patient samples. Practice effects, as a simple, convenient, and non-invasive marker for monitoring an individual patient’s cognitive status, would have the added benefit of perhaps being able to offer interventions to patients who are in the earliest stages of progressive neurodegenerative disorders.
This research is supported by the National Institutes of Health (NIMH MH61801, NIMH MH48639, NS440068 and MH01579), The High Q Foundation, the Huntington’s Disease Society of America, the Huntington’s Society of Canada, and the Hereditary Disease Foundation to the Huntington Study Group.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.