|Home | About | Journals | Submit | Contact Us | Français|
To assess the psychometric properties of the Valued Activity Inventory for Adults with Cancer (VAI-AC), a self-report instrument measuring activity limitations.
Fifty older adults undergoing chemotherapy.
Participants completed the VAI-AC and measures of physical and mental function, symptom intensity, and mood three days before and on the day of chemotherapy. Test-retest reliability was assessed by determining the average number of items for which the importance of an activity was rated consistently and by calculating the intraclass correlation coefficient (ICC) for the first and second VAI-AC scores. Convergent validity was assessed by correlating the VAI-AC scores with the other measures.
Participants consistently rated the importance of 90% of the items. Seventy-two hour test-retest reliability was ICC = 0.67. Participants with fewer activity limitations indicated better physical function (r = 0.58, p< 0.001), better mental function (r = 0.55, p< 0.001), lower symptom intensity (r = −0.57, p< 0.001), and fewer depressive symptoms (r = −0.68, p< 0.001).
The VAI-AC demonstrated evidence of test-retest reliability and convergent validity in this convenience sample of older adults undergoing chemotherapy for cancer.
Despite remarkable advances in survival associated with modern chemotherapy for cancer, treatment side effects remain a major concern for patients and providers. Common side effects of chemotherapy include fatigue, anemia, nausea, pain, and other problems (National Cancer Institute, 2008). These side effects, along with the disease itself, can make it difficult for people to eat, sleep, exercise, and do their daily activities (Cusick, Lawler, & Swain, 1987; de Jong, Candel, Schouten, Abu-Saad, & Courtens, 2006; Given, Given, Sikorskii, & Hadar, 2007). Although many measures assess clinical and biological treatment outcomes, validated approaches to evaluating the impact of treatment on valued activities are few. This report describes a new measure, the Valued Activity Inventory for Adults with Cancer (VAI-AC), and presents analyses of its reliability and validity.
Our research team has been developing telehealth interventions to help people undergoing chemotherapy find ways to maximize their ability to stay active during cancer treatment (Hegel, et al., 2010). We use problem-solving interventions, augmented by education about activity and environmental adaptation, to help people find ways to overcome challenges to their health and daily routines. In our work, we need to know what activities are important to our participants and how difficult they find those activities to be while undergoing chemotherapy. To meet this assessment need, we reviewed many instruments that measure activity performance of adults. Some instruments focused on mobility (Sayers, et al., 2004) or basic activities of daily living (Goodwin, Coleman, & Shaw, 2006) that are not as likely to be impaired in our population of outpatient cancer patients. Other tools were specifically developed only for women (Tulman & Fawcett, 2007) or for diagnoses other than cancer (Leidy, 1999). The available occupational therapy assessments (e.g., Canadian Occupational Performance Measure or the Activity Card Sort) were interview-based and too time-intensive for our purposes (Baum & Edwards, 2008; Law, et al., 1998). The Medical Outcomes Study SF-36 (Ware et al., 2007) is widely used to measure activity limitations that result from medical illness, yet it does not provide the opportunity to indicate which activities are personally valued or routinely performed by a participant.
Because the above instruments did not meet our current need, we decided to modify an instrument (described below) that was successfully used as a clinical tool in another problem-solving intervention trial (Rovner, Casten, Hegel, Leiby, & Tasman, 2007). We named our modification the Valued Activity Inventory for Adults with Cancer (VAI-AC). The VAI-AC asks participants to rate two dimensions for each of 25 activity items (see appendix). First, participants rate the degree to which it is important that they are able to do the activity. Second, participants rate the degree to which it is difficult for them to do the activity while undergoing chemotherapy. The VAI-AC is a client-centered measure of activity limitations. If an activity is not important to the participant, then it does not get factored into the score. A similar procedure has been used to measure disability from low vision (Massof, et al., 2005) and from arthritis (Katz, Morris, & Yelin, 2006).
The purpose of this analysis is to explore the psychometric properties of the VAI-AC. We asked three questions: (1) How consistent are participants in their designation of an activity as important to them versus not important to them? (2) How stable are participants’ reports of activity limitations over a 72-hour period? (3) Are activity limitations as measured by the VAI-AC correlated with established measures of physical and mental functioning and symptom intensity?
We recruited a convenience sample of participants primarily from the oncology clinics of a Veterans’ Administration Medical Center. In order to include women, recruitment also occurred at the oncology clinics at a National Cancer Institute-designated Comprehensive Cancer Center. Patients were invited to enroll if they were 65 years of age or older, diagnosed with solid or hematologic cancer for which chemotherapy is prescribed, and had completed at least one cycle of chemotherapy. Patients were excluded if they demonstrated impaired cognitive status as indicated by a score of 3 or less on a six-item cognitive screening tool (Callahan, Unverzagt, Hui, Perkins, & Hendrie, 2002). We limited the sample to adults over the age of 65 given that an exploratory aim of the study was to identify the challenges faced by older adults undergoing chemotherapy (data not reported).
The study design and procedures were approved by the Dartmouth College Committee for the Protection of Human Subjects (Institutional Review Board). Research assistants worked with oncologists and nurse practitioners to identify patients who might be eligible for the study. They approached patients during their chemotherapy visits to present the study, screen for cognitive impairment, answer questions, and obtain informed consent.
Upon enrollment, research assistants gave participants a packet containing a copy of their signed consent form, two copies of a self-administered survey labeled with the dates each survey should be completed, and two postage-paid return envelopes. The survey consisted of the following instruments.
The VAI-AC is a modification of the 14-item Visual Functioning Index (Steinberg, et al., 1994). The VF-14 was developed to measure functional impairment reported by persons with cataracts. Participants are first asked to indicate whether or not they engage in 14 activities (e.g., reading newspapers, reading street signs, writing checks, cooking, etc.) in daily life. Participants are then asked to indicate the degree of difficulty involved in doing the activity (“a little,” “a moderate amount,” “a great deal” of difficulty or “unable to do”). The authors report high internal consistency of the VF-14 (Cronbach’s α = 0.85) and adequate construct validity as indicated by moderate correlations with measures of visual acuity.
The VF-14 was used as a clinical tool in a study of problem-solving therapy (PST) for older adults with macular degeneration (Rovner, et al., 2007). Those investigators added importance ratings to the VF-14 and called it the Valued Activities Inventory. The tool allowed study participants to quickly identify activities that they were having difficulty performing. The importance ratings helped participants and therapists to quickly prioritize this problem list. Thus, from the brief, self-administered tool, therapists could begin the structured problem-solving sessions focusing on those activities most important to and most challenging for participants.
Because the Valued Activities Inventory was successfully used in the earlier PST study (Rovner, et al., 2007), we chose to modify and use the tool as part of the baseline assessment in a randomized controlled trial (RCT) of PST with women undergoing chemotherapy (Hegel, et al., 2010). To evaluate the content validity of the new tool we reviewed the literature to assess the types of items included in other measures of activity and participation. We created a list of activity items and presented the list to practitioners and researchers from the fields of nursing, medicine (oncology and psychiatry), occupational therapy, social and clinical psychology, and biostatistics. This exploration of other tools and discussion with expert reviewers allowed us to create a final tool (the VAI-AC) that consisted of 25 items appropriate to an outpatient population of persons with cancer.
The VAI-AC was used in our RCT of PST for women with breast cancer where face validity was demonstrated by participants’ ability to complete and use the tool to prioritize the targeted activities in the intervention. To explore the VAI-AC’s applicability with males, we pilot tested the tool with 10 men at a VA Medical Center. The men completed the VAI-AC on the day of a chemotherapy infusion and three days later. The men did not identify any additional activities that should be included in the VAI-AC. However, we noticed that the VAI-AC scores declined an average of 13 points upon the second administration. This is logical as the effects of chemotherapy are cumulative and side effects can worsen with each cycle. This pilot experience guided our selection of the data collection window described later.
Participants rate the importance of each item on a four-point scale: Not important, Slightly important, Moderately important, Very important. Any activity designated as “not important” is not factored into the VAI-AC score. Participants rate the difficulty of each item on a five-point scale (scored four to zero points corresponding to Not difficult, Slightly difficult, Moderately difficult, Very difficult, Impossible to do without someone’s help). The difficulty scores for items that are rated at least “slightly important” are averaged to provide a score between zero and four. The average difficulty score is multiplied by 25 to yield a total VAI-AC score ranging from zero (unable to do any important activities without help) to 100 (able to do all important activities with no difficulty).
The SF-36 is a multidimensional measure of health consisting of two standardized component summary scores (physical and mental). The scale scores are converted to a score from 0 to 100 with 0 representing worst health and 100 representing best health. The SF-36 is arguably the most widely used measure of function in health studies and has demonstrated adequate reliability and validity over a series of studies (summarized at www.sf-36org/tools/sf36.shtml).
The ESAS was designed to measure the effectiveness of symptom management interventions for persons with cancer. It assesses the severity of nine physical and mental symptoms on visual analogue scales (10-cm line). The ESAS includes the following symptoms often experienced by persons with cancer: pain, activity, nausea, depression, anxiety, drowsiness, appetite, sense of well-being, and shortness of breath. The sum of responses to these nine symptoms, in millimeters, is the ESAS total score. The ESAS was developed for a palliative care population and later validated in a general cancer population (Chang, Hwang, & Feuerman, 2000). In this latter validation study, internal consistency reliability was found to be 0.79 for the ESAS. Two day test-retest reliability was likewise adequate showing significant Spearman correlations between ratings of each item. Construct validity was supported by significant correlations between ESAS items and similar items on the Memorial Symptom Assessment Scale (Portenoy, et al., 1994). Because we used an optical scanning method for data entry, the scales were redesigned with discrete check boxes (0 – 10). An ESAS author indicated that the visual analogue scale can be replaced with a report of a single number for each symptom (Bruera, 1996).
The PHQ-9 is the nine item depression module from the Patient Health Questionnaire (Spitzer, Kroenke, & Williams, 1999). The items mirror the nine criteria used to diagnose depressive disorders according to the DSM-IV (American Psychiatric Association, 1994). Respondents use a four-point scale to indicate the frequency of depressive symptoms over the past two weeks (not at all, several days, more than half the days, nearly every day). The developers used the tool with 6,000 primary care or obstetrics-gynecology patients. Higher PHQ-9 scores were associated with poorer functioning on six SF-36 subscales. Criterion validity was established in that PHQ-9 scores ≥ 10 had sensitivity of 88% and specificity of 88% for major depression as determined by mental health professional interview. The developers report adequate test-retest and inter-method reliability (self-administered versus telephone administered).
In order to assess test-retest reliability of the VAI-AC, participants were instructed to complete the survey packet three days before their next chemotherapy appointment and on the day of their chemotherapy appointment. We hypothesized that this window would be a somewhat stable period during which there would be no reason to expect a change in functional status. The experience of how difficult it is to function while under the effects of the previous chemotherapy session would be fresh in their minds, but the side effects of the treatment should have largely diminished (as opposed to completing the first survey on the day of chemotherapy and completing the second survey three days later when the side effects of chemotherapy are strong). Hypothetically, this would provide a window of time in which any change in treatment score was due to either random error or instrument unreliability. As an additional check, a question was added to the second survey which asked: “Has there been any change in your life in the past 3 days that changes how difficult these activities have been for you?” (with a place to indicate either “yes” or “no”). Research assistants telephoned participants the day before the first survey was due to remind them to complete the survey and saw them in the infusion suite to inquire about and encourage completion of the second survey. Four people who completed the surveys agreed to participate in a debriefing interview to discuss their opinions of the VAI-AC. The four people were selected as a matter of convenience, because the first author had direct contact with them.
There are two elements that factor into the VAI-AC score: (1) the importance ratings and (2) the difficulty ratings. We explored the reliability of each factor. We first calculated descriptive statistics on the number of activities designated as important. Then we identified the proportion of items for which each person was consistent in his or her dichotomous importance designation. This created a proportion score for each participant. We then averaged the proportion scores and identified the 95% confidence interval associated with that average.
We scored the VAI-AC and calculated descriptive statistics. We used intraclass correlation coefficients (ICC) to explore the reliability between the first and second assessments. Because not every person completed the surveys exactly three days apart as instructed, we first used only the surveys completed three days apart. We then repeated the analysis using all surveys completed within six days of each other, and finally repeated the analysis with the entire sample of participants who completed two surveys.2
We explored the convergent validity of the tool by correlating the VAI-AC scores with the SF-36. We expected to see high, positive correlations showing that fewer activity limitations (higher VAI score) would be related to better functioning (higher SF-36 Physical Component Scores and Mental Component Scores). We also correlated the VAI-AC scores with the ESAS. We expected moderate negative correlations, hypothesizing that greater symptom burden (higher ESAS score) would be associated with reports of more difficulty engaging in activities (lower VAI-AC scores). As two hallmarks of depression are lack of interest or pleasure in activities and lack of energy (American Psychiatric Association, 1994), we also expected to see moderate, negative correlations between the VAI-AC scores and the PHQ-9 (higher scores indicating more depressive symptoms).
Eight of 63 approached patients declined to enroll, four citing lack of interest and four refusing due to the required effort of completing the surveys. The remaining 55 patients enrolled in the study. Five of them did not complete either survey (one was admitted to the hospital, one decided not to pursue chemotherapy, and three did not feel well enough to participate). Of the remaining 50 participants, 40 completed both surveys while 10 completed only the first survey. Thus, data from 40 participants were available for the reliability analyses and data from 50 participants were available for the cross-sectional validity analyses.
The demographics of the 50 survey completers are presented in Table 1. The participants were predominantly male, white, retired, with an income of less than $40,000 and a high school education. The average age of the sample was 75 (sd= 6).
The participants reported an average of 20.1 (sd = 3.7) important activities on the first survey and 20.6 (sd = 3.4) important activities on the second survey. The average proportion of items for which each participant’s importance designation remained consistent was 90% (95% CI = 87.3%, 92.6%).
The average VAI-AC score was 82.6 (sd = 16.8) on the first survey and 84.5 (sd = 13.3) on the second survey. The ICC between assessments was similar whether using the conservative subgroup of participants who completed the surveys within 72 hours of each other (n = 27, ICC = 0.67), the subsample of participants who completed the survey within six days of each other (n = 33, ICC= 0.71), or the full sample of participants who completed both surveys (n = 40, ICC = 0.65).
VAI-AC scores at Time 1 demonstrated moderate associations with the physical component summary score of the SF-36 (n = 44, r = 0.58, p< 0.001), the mental component summary score of the SF-36 (n = 44, r = 0.55, p< 0.001), the ESAS (n = 47, r = −0.57, p< 0.001), and the PHQ-9 (n = 47, r = −0.68, p< 0.001). As expected, participants who report more activity limitations report worse physical and mental functioning, greater cancer-related symptom intensity, and more depressive symptoms.
The four participants who discussed their experiences completing the VAI-AC indicated that “it depends” may be the most accurate answer to the question of how difficult it is to do activities while undergoing chemotherapy. The side effects, such as fatigue and nausea, vary over the days following chemotherapy (e.g., typically peaking at two to three days post-infusion and gradually tapering off). Therefore, participants’ perception and experience of activities fluctuates over the days, and even throughout the course of a day, depending on the adequacy of sleep and nutrition. The VAI-AC asks participants to provide an overall rating of difficulty, which may be hard for some people to identify.
When participants were explaining their answers, they sometimes stated that activities were not particularly difficult because they had already made accommodations to increase their ease or comfort. For example, one woman stated that housekeeping was not difficult but only because she is doing it differently than before (using a manual carpet sweeper instead of a heavy vacuum cleaner and using an automatic shower cleaner instead of manually scrubbing). The VAI-AC does not capture those distinctions and accommodations.
The VAI-AC demonstrated moderate test-retest reliability (Portney & Watkins, 1993), with the intraclass correlation coefficient being similar regardless of the time delay between the first and second administrations. While the debriefing participants indicated that their perception of the ease of performing an activity varies greatly in the week following chemotherapy, the reliability analyses suggest that participants are able to use the VAI-AC to provide stable responses on that varying perception. It is possible that the VAI-AC’s test-retest reliability would increase by providing more specific instructions e.g., rate how difficult the activity is when you are feeling your worst, or on a specific number of days after infusion. While this may be easier for participants to assess, it would also limit the utility of the tool, an important consideration as our objective has been to provide a general estimate of activity limitation while under treatment.
The analyses provided evidence of the construct validity of the VAI-AC in that there were logical associations with other measures. The VAI-AC asks people to rate how difficult it is to perform personally relevant activities while undergoing chemotherapy. The SF-36 asks people to rate the extent to which their physical and mental health limits them in performing various daily activities. These are similar, but not identical, constructs. The moderate (as opposed to high) correlations (Colton, 1974) could be indicative of the subtle differences in the constructs or could reflect a psychometric weakness of either instrument.
We also expected to see moderate associations between symptom intensity and activity limitations as measured by the VAI-AC. While the constructs of symptom experience and activity limitation are not similar, both cancer-related symptoms and depressive symptoms are associated with self-reported disability (Bruce, 2000; Dodd, Miaskowski, & Paul, 2001). Activity limitations as measured by the VAI-AC were related to both measures of symptom intensity, with the strongest association seen between activity limitations and depressive symptoms. The cross-sectional nature of the data does not allow us to definitively determine the direction of causation, though it is plausible that feeling pain or fatigue leads to the perception that activities are difficult. The relationship between activity limitations and depressive symptoms, however, could be bidirectional given that previous research has shown that depressive symptoms precede activity limitations (Bruce, Seeman, Merrill, & Blazer, 1994) and activity limitations precede depressive symptoms (Bruce & Hoff, 1994). Regardless of the explanation, the associations between the symptom measures and the VAI-AC offer preliminary evidence of the construct validity of the VAI-AC.
The VAI-AC allows rapid prioritization of difficult activities that need to be tackled in rehabilitation sessions. The VAI-AC would lend itself easily to computer-administration where a respondent would only be asked to rate the difficulty of activities that are important to him or her. However, the VAI-AC appears to have some potential limitations that clinicians and researchers should consider. Like the SF-36, the VAI-AC does not allow respondents to indicate if someone else is doing an activity for them or if they have modified the way in which they are doing an activity (e.g., grocery shopping online versus going to a store). If those factors are important to know for research or clinical purposes, another tool or additional items would need to be used to supplement the VAI-AC (see Katz and Morris, 2007 for an example).
There are limitations of our study that warrant consideration. This small, convenience sample was predominantly comprised of White, Non-Hispanic men. While hematological cancers and some gastro-intestinal cancers are slightly more prevalent in men (Altekruse, et al., 2010), the gender distribution in this sample is not reflective of population of persons who experience the cancers discussed here. The convenience sample limits our ability to understand how well the tool would perform for people of other ethnicities and to explore subgroup analyses based on gender or cancer type. Additionally, the participants in this sample were not experiencing high levels of activity limitations. Further studies are needed to see if these findings would be replicated in larger, diverse, and more functionally impaired populations.
It would also be helpful to subject the VAI-AC to formal cognitive interviewing (Tanur, 1992). Cognitive interviewing allows instrument developers to ascertain if respondents share a common understanding of each item and share a common process in providing an answer to each item. Our debriefing interviews in this study and clinical use of the VAI-AC in another study (Hegel, et al., 2010) helped us to learn what participants thought of each activity item, however, cognitive interviewing would help to further clarify the validity of the distinctions of the response choices (e.g., the difference between “moderately” and “very” difficult). Finally, our study did not include measures that would allow us to test the divergent validity of the VAI-AC (i.e., are the VAI-AC scores unrelated to other constructs that, theoretically, have nothing to do with activity limitations?). Although we made this choice to reduce participant burden, conclusions about the validity of the VAI-AC would be strengthened by exploring its associations with other distinct and conceptually unrelated constructs.
The VAI-AC is a brief, self-administered, and client-centered measure of activity limitations. The VAI-AC demonstrated provisional evidence of test-retest reliability and convergent validity in this sample of older adults undergoing chemotherapy for cancer. A clinician can use the VAI-AC importance ratings to initiate a discussion about prioritizing treatment activities. The scoring algorithm allows tracking of progress, regardless of whether the reduced difficulty is due to improved skills or effective adaptations. Future research on the psychometrics of this tool, including its sensitivity to change, is warranted to confirm these findings and further the development of clinical and outcome measures of activity limitations.
The authors thank Luann Graves and Daphne Ellis for their assistance in participant recruitment and data collection.
Grant support: This project was funded by the Tiffany Blake Fellowship awarded to the first author from the Hitchcock Foundation of Lebanon, NH and by a grant from the National Cancer Institute (1 R03 CA124200-01; PI: Hegel).
Previous presentation: An interim analysis of these data was presented in a poster at the American Occupational Therapy Association Annual Conference and Exposition in Houston, TX on April, 24, 2009.
Disclosure: The authors have no conflict of interest to disclose related to these instruments and analyses.
1Because each participant has a different set of items endorsed as “important” we are unable to calculate internal consistency of the VAI-AC score.
2Only two people reported a change in status that might affect their Time 2 answers. These participants were in the group of people completing the surveys within 72 hours of each other. Their data were not removed from the analyses.
Kathleen Doyle Lyons, Research Assistant Professor of Psychiatry, Dartmouth Medical School, Hanover, NH and Investigator, Norris Cotton Cancer Center, Lebanon, NH.
Mark T. Hegel, Associate Professor of Psychiatry, Dartmouth Medical School, Hanover, NH and Investigator, Norris Cotton Cancer Center, Lebanon, NH.
Jay G. Hull, Professor of Psychological and Brain Sciences, Dartmouth College, Hanover, NH.
Zhongze Li, Research Statistical Analyst, Biostatistics Shared Resource, Norris Cotton Cancer Center, Lebanon, NH.
Stefan Balan, Clinical Director, Norris Cotton Cancer Center, Manchester, NH.
Stephen Bartels, Professor of Psychiatry, Dartmouth Medical School, Hanover, NH.