|Home | About | Journals | Submit | Contact Us | Français|
Despite potential bias, researchers often rely on patient self-reported data of health care use. However, the validity and accuracy of self-reported data on cardiac rehabilitation (CR) use are unknown.
To assess the concordance between patient self-report and site-verified CR referral, enrollment and participation.
A consecutive sample of 661 coronary artery disease inpatients (mean [± SD] age 61.27±1.31 years; 157 women [23.8%]) treated at three acute care sites was recruited (75% response rate) as part of a larger study comparing automatic with usual referral methods. CR referral, enrollment (attendance at intake assessment) and participation (percentage of program attended) were discerned in a mailed survey nine months following discharge (n=506; 84.3% retention). A total of 24 CR sites were contacted for verification.
A total of 276 participants (54.5%) self-reported CR referral, and CR sites verified receipt of 262 referrals (51.8%) (Cohen’s kappa 0.899). A total of 232 participants (45.8%) self-reported CR enrollment, with site-verification for 208 participants (41.1%) (Cohen’s kappa 0.846). Self-reported data indicated that participants attended a mean of 81.78±25.84% of prescribed CR sessions, with CR sites reporting that participants completed 80.75±31.27% of the program (r=0.662; P<0.001). Equivalency testing revealed that the self-reported and site-verified rates of program participation were equivalent (z<1.96).
The almost perfect agreement between the self-reported and site-verified use of CR services suggests that self-administered items are highly valid in this population.
Malgré le biais potentiel, les chercheurs se fient souvent à l’utilisation des soins de santé déclarée par le patient. Cependant, on ne connaît pas la validité et la précision des données autodéclarées en réadaptation cardiaque (RC).
Évaluer la concordance entre les aiguillages de RC, les inscriptions et la participation déclarés par le patient et ceux vérifiés par établissement.
Un échantillon consécutif de 661 patients hospitalisés à cause d’une coronaropathie (âge moyen [±ÉT] de 61,27±1,31 ans; 157 femmes [23,8 %]) traités dans trois établissements de soins aigus ont été recrutés (taux de réponse de 75 %) dans le cadre d’une étude plus vaste comparant les méthodes d’aiguillage automatique aux méthodes habituelles. Les auteurs ont déterminé l’aiguillage en RC, l’inscription (la présence à l’évaluation d’admission) et la participation (le pourcentage de présence au programme) au moyen d’un sondage postal envoyé neuf mois après l’obtention du congé (n=506; rétention de 84,3 %). Ils ont pris contact avec un total de 24 établissements de RC afin de procéder à la vérification.
Au total, 276 participants (54,5 %) ont autodéclaré un aiguillage en RC, et les établissements de RC en ont confirmé 262 (51,8 %) (kappa de Cohen, 0,899). Au total, 232 participants (45,8 %) ont autodéclaré leur inscription en RC, et la vérification par établissement a confirmé 208 participants (41,1 %) (kappa de Cohen 0,846). D’après les données autodéclarées, les participants ont participé à une moyenne de 81,78±25,84 % de séances de RC prescrites, les établissements de RC indiquant que les participants avaient été présents à 80,75±31,27 % du programme (r=0,662; P<0,001). Les tests d’équivalence ont révélé que les taux de participation au programme autodéclarés et vérifiés par établissement étaient équivalents (z<1,96).
La concordance presque parfaite entre l’utilisation des services de RC autodéclarée et vérifiée par établissement indique que les outils autoadministrés sont hautement valides au sein de cette population.
Self-report questionnaires are a relatively simple and cost-efficient means of collecting data on health care use (1). Such data can provide insight into access and use disparities, and may also impact policy-making decisions. Despite potential social desirability and recall biases, researchers often rely on self-report surveys to obtain data on health care use. However, studies have reported significant inaccuracies with self-reported health care use (2–4). For example, in a study by Cronan and Walen (2), 70% of patients under-reported their total contacts with the health care system, and only 18% reported the same number of doctor or nurse visits as the agency (r=0.61).
Cardiac rehabilitation (CR) is an interdisciplinary and comprehensive program that incorporates secondary prevention measures to reduce cardiovascular disease progression, improve risk factors and prevent the recurrence of cardiac events (5). CR participation results in improved morbidity and mortality outcomes for heart disease patients (6,7). As is the case with other studies on health care use, studies on CR use mainly rely on self-report data. However, the validity and accuracy of data on self-reported CR use are unknown. The purpose of the present study was to assess the concordance between self-report and site-verified CR referral, enrollment and participation. Investigating the veracity of data on self-report CR use will establish the degree to which studies that rely on such data truly reflect use.
The present study is a cross-sectional component of a larger prospective controlled study on CR referral methods (8). Once ethics approval was obtained from all participating sites, consecutive coronary artery disease inpatients were recruited by a research assistant from three hospitals in southern Ontario, when medically stable. CR services are covered by health insurance in Ontario. Patients from one of these hospitals were automatically referred to the CR site closest to their home. Nine months later, participants completed a second mailed survey to discern CR referral, enrollment and participation. CR sites to which participants self-reported referral were then contacted to verify the data.
The inclusion criteria for the larger study were diagnosis with a confirmed myocardial infarction, unstable angina, or admission for percutaneous coronary intervention (PCI) or acute coronary bypass. Exclusion criteria included being medically unstable, too confused to participate, previous participation in CR, being ineligible for CR based on Canadian Association of Cardiac Rehabilitation guidelines (5) due to musculoskeletal, vision, psychiatric or other comorbidities, or being unable to read or speak English. Eligible and willing patients signed a consent form and were provided with a self-report questionnaire. Cardiac clinical data were also extracted from their medical charts. Six hundred sixty-one patients consented in the hospital to participate in the study. Characteristics of participants compared with decliners and ineligibles are described in an earlier paper (8).
In the nine-month post-discharge survey, participants self-reported whether they were referred to CR (yes/no) and at which site, whether they attended a CR assessment (yes/no), whether they participated in CR (yes/no), and provided an estimate of the percentage of prescribed sessions they attended. Twenty-four CR sites were contacted to verify the receipt of referrals (yes/no), participant enrollment or attendance at intake appointment (yes/no), and percentage of the program attended.
SPSS 14.0 (SPSS Inc, USA) was used for the following analyses. Following a descriptive examination of participant characteristics, a descriptive examination of self-reported and site-verified CR referral, enrollment and participation rates was performed. For simplicity, the term ‘use’ will represent both referral and enrollment below. The sensitivity (correctly reported yes to CR use/all CR-verified yes to CR use), specificity (correctly reported no to CR use/all CR-verified no to CR use), positive predictive value (correctly reported yes to CR use/all self-reported yes to CR use), negative predictive value (correctly reported no to CR use/all self-reported no to CR use) and total agreement (correctly reported yes and no to CR use/total number of participants) were computed (9).
Cohen’s kappa (κ) was used to assess the degree of concordance between self-reported and site-reported CR use. The classification system suggested by Fleiss (10) was applied for the Cohen’s κ statistic, where κ less than 0.40 represents poor to fair agreement, 0.40 to 0.60 represents moderate agreement, 0.60 to 0.80 represents substantial agreement, and 0.80 to 1.00 represents almost perfect agreement. A stratified analysis of κ coefficients was performed by sex, age (younger than 65 years of age versus 65 years or older), education (up to high school versus more than high school), index condition (PCI versus other), type of CR referral (automatic versus usual) and degree of self-reported CR participation (85% or greater of program versus less than 85%).
Finally, Pearson’s correlation was used to test the degree of agreement between the percentage of CR sessions attended as assessed via self- and site-report. A stratified analysis of correlation coefficients was also performed by the same variables listed above. In addition, an equivalency test (11) was used to test the null hypothesis of difference in self- and site-reported rates of participation in prescribed CR sessions.
Of the 661 consenting participants, 61 were ineligible and 506 were retained at the nine-month postdischarge assessment (retention rate 84.3% [506 of 600 participants]). Reasons for ineligibility were as follows: unable to reach the patient or incorrect contact information (n=34; 5.1%), too ill to participate (n=10; 1.5%), deceased (n=8; 1.2%) and other reasons (n=9; 1.4%) including the onset of conditions that precluded eligibility for CR. Characteristics of participants and nonparticipants at nine months postdischarge are summarized in Table 1. Retained participants were more likely to have had PCI as their index event, to be married, to be older, to have a lower body mass index, to be a nonsmoker, to describe their ethnocultural background as white and to have greater family income than participants lost to follow-up. There was an average of approximately 10.31 days between graduation from a CR program and survey completion.
Overall, 276 participants (54.5%) reported referral, 232 (45.8%) reported enrollment and 216 (42.7%) reported CR participation. Seven participants (3.02%) reported enrollment but no referral, nine (4.17%) reported participation but no referral, and one (0.46%) reported participation but no enrollment.
Table 2 presents the concordance between participant self-report and program-verified data regarding CR referral and enrollment. Using Fleiss’ (10) classification system, there was almost perfect agreement between the two sources for both CR referral and enrollment. CR participants stated that they attended a mean of 81.78±25.84% of their CR sessions, which significantly correlated with verified program attendance across all sites (80.75±31.27%, r=0.622; P<0.001). Results from the equivalency test (11) indicated that the self-reported and site-verified data regarding percentage of CR program completion were equivalent (z<1.96).
Table 3 shows the almost perfect agreement between self-report and site-verified CR referral and enrollment data by sex, age, education, type of cardiac condition and type of referral category. Although there did appear to be a slight difference in concordance based on ethnicity with greater discordance for nonwhites, overall these results suggest substantial agreement between self-report and site-verified data according to Fleiss’ classification system. Results also indicated moderate to high correlation coefficients between self-reported and site-verified percentage of CR program participation by sex, age, ethnicity, type of cardiac condition and type of referral. However, there was low agreement for those with less than high school education, and for those who reported a high degree of CR participation.
Researchers often rely on patient self-report in an attempt to understand health care use patterns, but the present study is the first to analyze the validity of such data relating to CR. The purpose of the present study was to assess the level of agreement between self-reported and program-verified data regarding CR use. Results have important implications for the credence given to previous literature on CR use.
According to Fleiss’ (10) classification system, our results showed almost perfect agreement between the two data sources, with approximately 90% concordance between self-reported and site-verified CR referral, and 85% concordance for enrollment. Previous studies have also reported moderate to high correlations between self-reported and health care agency use (12,13). An approximate 10% discordance is quite encouraging because other studies have found substantial disagreement between self-reported and site-verified data on health care use (2,14). However, the lack of studies assessing the validity of self-report data, specifically regarding cardiac specialist use or cardiac procedure use, make it difficult to compare our results with other literature.
The high rates of agreement may be due to several factors. First, because these patients have a life-threatening condition, they may be more likely to remember being referred to a program they were told could aid in their recovery. Also, the experience of enrolling and participating in CR is not as customary as going to see one’s physician and, hence, is likely more memorable. Moreover, going to an outpatient program such as CR involves multiple visits and is thus likely to be more memorable than a single visit with a physician or other single medical encounter. Such salient types of use are more likely to be reported accurately (15). It is likely that the distinctiveness of CR from usual health care accounts for the excellent agreement rates. Future studies should examine whether the validity of self-report data are based on characteristics of the health care service used (eg, type, frequency, duration), or even the type of CR program (ie, home- or hospital-based services). In the current study, variation between CR sites was not accounted for. However, CR is publicly funded in Ontario for six months of intervention, in which patients attend a median of two on-site exercise sessions per week (ie, an average of 48 prescribed sessions).
Based on the discordance in the positive predicted values, over-reporting was evident because more participants reported CR referral and enrollment than what was program-verified. This is contrary to the finding of a literature review by Bhandari and Wagner (15), which found that under-reporting was the most frequent problem when collecting self-report health care use data. In the present study, discrepancies may have occurred in which participants mistakenly perceived themselves to be referred to CR when they were, in fact, only being educated about CR and its effectiveness. Moreover, the stresses of a cardiac event and the influence of medication may have affected their memory if they were referred during their hospitalization. Because physicians or other allied health care providers ultimately make the CR referral, patients may have perceived a referral was made when, in fact, the actual referral was not submitted. Furthermore, uncertainty regarding what a CR assessment exactly entails may have also affected responses.
There appeared to be minimal internal discordance between self-report of CR referral, enrollment and participation, suggesting that there is little social desirability bias in self-report of CR use. Similar to the reasons presented above, discordance may have occurred when patients were unaware of whether a referral was made, even though they participated in the program.
Previous studies have found that older age, male sex, lower education, Caucasian ethnicity and poor health status are associated with self-report inaccuracy and under-reporting (3,14,16–19). However, our results failed to find meaningful differences in self-report and site-verified concordance rates based on sex, age, education, index cardiac condition or type of referral, although there was slightly lower concordance among participants reporting a nonwhite ethnocultural background. Future research should explore whether this difference is robust, and what means can reduce this discrepancy (eg, revising question terminology, translating questions, offering CR information in a patient’s first language).
Previous studies (16,20) have found increased under-reporting with greater frequency of health care use. Similarly, we found quite a low correlation between self-reported and site-verified percentage of program completion for those who reported 85% or greater of CR participation. However, in the present study, it appears that because of the multiple visits of a CR program, CR participants who self-reported a relatively high (85% or greater) percentage of attendance actually over-reported their level of CR participation.
Caution is warranted when interpreting these results. Because the present study is the first to verify self-report data of CR use, replication is warranted. Generalizability is limited by selection and retention biases. For instance, retained participants were more likely to be male, white, highly educated and have a higher family income and more social support than participants lost to follow-up – characteristics of patients who typically present in CR. Therefore, the study should be replicated with a broader population of patients who are generally under-represented in CR. Finally, generalizability is also limited to the health care system in which the present study was conducted. It is possible that because of the publicly funded universal health care system, which includes CR at no cost to the patient, patient recall may have actually been lower than if the patient had to pay for the program themself or through insurance. The present study should be replicated under nonuniversal health care coverage to compare concordance between self-report and site-verified CR use.
Our results suggest that cost-efficient self-administered questionnaires assessing CR use are highly valid, and represent a valuable means of providing information for researchers and health policy decision-makers. Future studies should use similar self-reported questionnaires in various patient populations to assess CR referral, enrollment and participation to determine reliability and validity. Such research may lead to the development of a psychometrically validated self-report measure of CR use, which may minimize bias and increase comparability across studies.
The present study was funded by the Canadian Institutes of Health Research, the Canadian Health Services Research Foundation, and the Ontario Ministry of Health and Long-Term Care. Dr Grace is supported by the Ontario Ministry of Health and Long-Term Care, and Ms Kayaniyil is supported by the Heart and Stroke Foundation of Ontario.