|Home | About | Journals | Submit | Contact Us | Français|
Clinical practice guidelines (CPG) are meant to consider important values such as patient preferences.
To assess how well clinical practice guidelines (CPGs) integrate evidence on patient preferences compared with that on treatment effectiveness.
A cross-sectional review of a listing in 2006 of CPGs judged to be the best in their fields by an external joint government and medical association body.
Exclusion criterion was unavailability in electronic format. Sixty-five of 71 listed CPGs met selection criteria.
Two instruments originally constructed to evaluate the overall quality of CPGs were adapted to specifically assess the quality of integrating information on patient preference vs. treatment effectiveness. Counts of words and references in each CPG associated with patient preferences vs. treatment effectiveness were performed. Two reviewers independently assessed each CPG.
Based on our adapted instruments, CPGs scored significantly higher (p<0.001) on the quality of integrating treatment effectiveness compared with patient preferences evidence (mean instrument one scores on a scale of 0.25 to 1.00: 0.65 vs. 0.43; mean instrument two scores on a scale of 0 to 1: 0.58 vs. 0.18). The average percentage of the total word count dedicated to treatment effectiveness was 24.2% compared with 4.6% for patient preferences. The average percentage of references citing treatment effectiveness evidence was 36.6% compared with 6.0% for patient preferences.
High quality CPGs poorly integrate evidence on patient preferences. Barriers to incorporating preference evidence into CPGs should be addressed.
The online version of this article (doi:10.1007/s11606-009-0987-8) contains supplementary material, which is available to authorized users.
A 59-year-old patient with atrial fibrillation and no cardiac risk factors would like to discuss antithrombotic treatment options with you. A cardiologist has told her that she does not need warfarin and that aspirin is sufficient. As a professional pianist, however, she worries about the possibility of a stroke affecting her hands. You read the American College of Chest Physicians (ACCP) guideline, which states that “In patients with…atrial fibrillation <65 years old and with no other risk factors, we recommend aspirin, 325 mg/day”1. You expect this recommendation has been based on randomized controlled trials (RCTs) and prognostic studies. But has this recommendation also incorporated evidence on patient preferences in a way that helps you and your patient make a shared decision?
For over a decade, evidence-based medicine (EBM) has been defending itself against claims that it is prescriptive, “cookbook” medicine2. Its advocates have defined the field as the “integration of the best research evidence with clinical expertise and patient values”3. CPGs are meant to embody this philosophy by synthesizing clinical evidence with “all the values that might sway a clinical recommendation”4. At the level of the patient-physician interaction, patient preferences are arguably the most important values that must be considered.
What are patient preferences? Colloquially, most clinicians might interpret the term as “what my patient wants.” In much of the medical literature, however, the notion of preferences is not clearly articulated and can include related concepts such as goals5, expectations6 and satisfaction7. Perhaps the most fully developed concept of patient preferences can be found in the field of decision analysis8.
In the example of atrial fibrillation, decision analyses have indicated that the best choice of antithrombotic therapy may change depending on how much patients value their quality of life if impaired by stroke and how much they value the inconvenience of monitoring warfarin and a hemorrhagic event9. These kinds of preferences are characterized as preferences for health outcomes and are commonly quantified by measuring a patient’s utility, which is a holistic quality of life measure on a 0–1 scale10. For example, a large survey of people at high risk for stroke indicates that their mean utility for a major stroke is 0.3011.
Within the decision analytic framework, we can also place the concept that patients have preferences for health decisions involving uncertain outcomes. After considering the available choices, patients may express their preferences for a specific option, a concept closer to the colloquial understanding of the term “preference.” In the case of atrial fibrillation, research on this type of treatment preference may include surveys of patient attitudes towards treatment12, studies of actual treatment decisions13 or studies on the effect of decision support tools. For instance, RCTs of decision aids in atrial fibrillation suggest that these tools improve patient understanding and change the proportion of patients who choose aspirin over warfarin14,15. Figure Figure11 illustrates how these notions of patient preferences can be characterized in a decision analytic conceptual framework.
Patient preferences are assuming a greater prominence in clinical decision making as reflected in the increasing role taken by patients in health-care decisions, number of studies on patient preferences and decisions, and prominence of the field of shared decision making16–18. An advisory report to the World Health Organization appreciates that differing preferences can lead to different CPG recommendations and states that “Values should always be considered in making recommendations…”19. Owens recognizes that “Because differences in patients’ preferences may lead to differences in the preferred therapy, a clinical practice guideline that does not consider patients’ preferences may provide recommendations that are not optimal”20. Failure to reconcile recommendations with preferences is often cited as a barrier to CPG adherence21. Given this need to better include values, it would seem that evidence regarding patient preferences should be systematically integrated into CPGs. The purpose of this study was to assess how well current CPGs incorporate published evidence on patient preferences for decisions and health outcomes.
We chose CPGs recommended by the Guidelines Advisory Committee (GAC)22, a joint body of the Ontario Ministry of Health and Long-Term Care and the Ontario Medical Association. The GAC conducts annual surveys of stakeholders to identify clinical topics of general importance in Ontario, performs literature searches to identify existing CPGs on those topics and then asks community-based physicians trained in CPG appraisal to evaluate the guidelines using the internationally developed AGREE instrument23. The GAC then endorses a single guideline on the basis of quality and relevance to Ontario practitioners.
At the time of this study’s initiation, the GAC listed 71 recommended guidelines. Two of the guidelines were simple randomized controlled trials rather than fully developed guidelines, and one was subsequently converted into two separate CPGs. This left 70 CPGs. We excluded guidelines not available in electronic format to simplify measurement of our intended outcomes, as described below. Thus, a total of 65 CPGs were available for analysis (reference list available online at URL). We extracted the key characteristics of each CPG including disease category as per International Classification of Diseases version 10 codes24, country of origin, institution source (i.e., government or medical association), year of publication, total word count and total number of references.
No standard methods are available to evaluate the quality of integrating preference-related evidence. We therefore adapted two instruments originally designed to assess the overall quality of CPGs to specifically appraise how well evidence on patient preferences was sought and integrated. Our first instrument was based on the AGREE instrument23 and included four items applying to the search strategies and incorporation of preference evidence into the recommendations. The second instrument was based on a tool developed by Shaneyfelt et al.25 and included seven items covering issues of search strategies, formally grading, extracting and combining preference evidence, and weighing risks and benefits from a preference perspective. Details on how these instruments were developed and validated are available as a Supplementary Appendix online at URL. Using the adapted instruments, two reviewers (CAKYC and IC) independently appraised each guideline. Scoring discrepancies for items on the AGREE instrument that differed by two or more points and any scoring differences for the Shaneyfelt et al. items were resolved by discussion between the reviewers.
In addition to using the two adapted instruments to assess how well patient preferences were incorporated into CPGs, we also quantified how much of the CPG text was actually devoted to preference issues. This endeavor involved counts of how many words and references discussed preferences. A summary of our coding rules are listed online in the Supplementary Appendix. Two authors (CAKYC and IC) independently read the CPGs and highlighted relevant text and references on patient preferences and treatment effectiveness. The percentage of total words and references applying to each issue was calculated. We allowed a relative difference of 15% between the two reviewers (e.g., 20% vs. 23%) or an absolute difference of <2%, and took the mean for primary analysis. When the relative or absolute differences exceeded the preset limits, the reviewers met to compare and resolve differences by discussion.
To give our findings context, a similar analysis was done assessing the quality and quantity of integrating evidence on treatment effectiveness, a key component of nearly all guidelines and thus an easily available reference standard. Further details are available in the online Supplementary Appendix.
The Spearman rank test was used to assess correlations among continuous variables. The paired Wilcoxon sign-rank test was used to compare differences between patient preferences and treatment effectiveness. We chose the nonparametric Spearman rank and paired Wilcoxon sign-rank tests because not all the data were normally distributed. Repeating all analyses with the parametric Pearson correlations and paired t-tests did not change any of the results; these results are not presented. The analyses were performed using SPSS version 14.0.
Table 1 outlines the characteristics of the CPG dataset. The CPGs cover a wide variety of medical illnesses. Most CPGs (72.3%) provided a general review of a topic, with the remainder being divided amongst those focusing specifically on screening, diagnostic tests or specific treatment modalities. Approximately half of the CPGs were from organizations in the United States. Medical associations and government-based groups authored approximately an equal number (~40%) of CPGs.
Tables 2 and and33 describe the adapted AGREE and Shaneyfelt et al. scores for assessing quality of patient preference integration. The overall mean adapted AGREE score (on a scale of 0.25–1.0) for incorporating preferences is 0.43; the overall mean adapted Shaneyfelt et al. score (on a scale of 0–1.0) is 0.18. To help put these values in context, note that for incorporating evidence on treatment effectiveness, the mean adapted AGREE and Shaneyfelt et al. scores were 0.65 and 0.58, respectively (p<0.001).
The percentages of text and references dedicated to patient preferences were 4.2% [standard deviation (SD) 8.0] and 6.0% (SD 9.1), respectively. As the SDs indicate, however, there was great variability in these values. Again, to put these numbers in context, we found the percentages of text and references addressing treatment effectiveness were 24.2% (SD 18.0) and 30.6% (SD 20.0), respectively (p<0.001)
Six of the CPGs could be classified as addressing acute life-threatening issues and thus might be considered less preference-sensitive. However, these six CPGs did not differ significantly from the remaining CPGs on any of the outcome measures.
CPGs published more recently appeared to better incorporate preference evidence. A later publication year was weakly correlated with a higher adapted AGREE score (r=0.247, p=0.047) and Shaneyfelt et al. score (r=0.230, p=0.065).
This study empirically demonstrates that high quality CPGs from a variety of disciplines generally do not systematically seek or integrate evidence on patient preferences. Five percent of CPGs cited a method to identify preference evidence. Only 50% of CPGs cited any preference-related evidence. Less than five percent included text on attitudes toward benefits and harms, combined preference evidence formally or characterized the methods used to extract preference evidence.
Why is evidence on patient preferences not better incorporated into CPGs? We suggest that there are several barriers that need to be overcome.
First, there needs to be an appreciation that published research on patient preferences actually counts as evidence. For most clinicians, “evidence” related to treatment decision making is embodied by the clinical trial26 that can be searched for, integrated into CPGs and then applied in individual decision making. Patient preferences, on the other hand, are typically considered only on an individual level, in the process of applying the “scientific” evidence at the bedside27. In this conceptualization, there is not a significant role for, and therefore less value attached to, formal studies of other patients. Preference is seen as fundamentally a property of the individual, highly subjective and variable across persons. We believe, however, that scientific studies of patient preferences are more common than is generally appreciated and furthermore are a valid means of understanding general trends in patient values and of identifying decisions that are particularly “preference-sensitive.” Such studies can be highly informative, particularly in circumstances in which other evidence is weak or there are competing risks and benefits. A first step towards legitimizing this body of research is to extend the common usage of the term “evidence” to include preference studies.
Second, a clear taxonomy for studies of patient preferences does not exist. Traditional medical evidence can be categorized into studies on natural history, diagnosis, prognosis and treatment, and may take the form of a case report, cohort study, randomized trial or meta-analysis3. For preference-related research, there is a less well-developed understanding of what kind of literature exists, how to label it and how to extract it. Preference research is distributed across disciplines that use not only varying study designs, but varying terminology. In this paper, we used a decision analytic approach, but it is important to recognize that “preference” is conceptualized and measured differently across disciplines. For decision analysts and economists, preference research may involve studying utilities for health outcomes, processes, or treatment choices8. Health psychologists may focus on treatment satisfaction, desires and expectations28. Qualitative researchers may study narratives of patient and provider experiences29. Ethicists may use the language of informed consent to understand preference. Economists may use utilities, conjoint analysis and/or use citizen juries to determine how patients or citizens value the attributes of a health-care service30. An important step towards trying to find the relevant evidence is to develop a consistent language for preference-related research that respects the contributions of relevant disciplines as well as a systematic categorization (taxonomy) for the types of preference-related research.
The absence of a clear taxonomy compounds a third problem; there is no simple, generally accepted method to synthesize evidence on preferences. Formal methods have been developed to combine results of randomized trials and studies of diagnostic accuracy3. But how does one integrate a cross-sectional survey on patient satisfaction with a qualitative narrative study? Some types of preference research, such as those on quality of life and utilities, may be combined using summary statistics. More recently, techniques for meta-synthesis have been emerging to combine qualitative studies31. In general, however, methods for integrating the results of complementary preference studies are not well developed.
Fourth, and perhaps most importantly, there needs to be agreement that including preference evidence benefits clinical practice. Some may argue that including such studies adds a level of complexity that unduly hampers the development of timely CPGs and the translation of knowledge into practice. Certainly, there is much to be said for simplicity. In one study, a simplification of hypertension treatment into an abridged algorithm allowed more uncomplicated patients to reach blood pressure targets than using CPG-based practice32. It must be acknowledged, however, that to ignore preference studies would result in the systematic exclusion of one class of relevant evidence. Before a class of data is excluded on grounds of expediency, convenience or usefulness, one needs to think carefully whether doing so optimally serves patients, clinicians and health systems.
Our study has several limitations. We could not use pre-validated methods for auditing the CPGs for inclusion of content related to patient preferences, and no gold standard exists. We do believe, however, that there are clearly circumstances in which preferences must be included and have drawn attention to the difference between preference and non-preference evidence because we think the issue needs to be further explored and the gold standard defined. In the interim, we were obliged to employ novel methods and thus included four distinct outcome measures and performed extensive testing to evaluate the performance characteristics of our instruments (see Supplementary Appendix). The original nature of some of the instruments may have biased the results towards showing better effectiveness integration. For example, an original Shaneyfelt et al. item is worded as “formal methods of combining evidence or expert opinion are used and described.” As mentioned earlier, there is no broadly prescribed means of unifying preference evidence, so preference integration scores would naturally be lower on this item. As well, in using percentages of text and references as measures, we do not mean to imply that a pre-specified amount of a CPG must be devoted to preferences, and we appreciate that these results may reflect the fact that there is an overall paucity of preference-based literature. Indeed, it is possible that preference evidence is represented in CPGs at a higher percentage than exists in the broader literature. We also note that there is no consensus on the extent to which preferences should be included in CPGs. A second limitation is our consideration of a relatively small and heterogeneous dataset authored primarily by North American agencies. Choosing several CPGs on a few similar topics, but across different countries and funding agencies may have revealed some interesting differences in nation or author source. Third, there are clearly some medical situations that are much more preference-sensitive than others (e.g., treating septic shock versus treating early stage breast cancer); our study did not fully consider this variable in assessing CPG preference incorporation.
We believe that a natural corollary of patient-centred medicine is the incorporation of preference evidence into CPGs. To take the example of screening for prostate cancer with prostate-specific antigen (PSA), we suggest that preference evidence can help identify recommendations that are particularly sensitive to individual preferences (e.g., deciding to screen for PSA or not), identify and describe the specific types of health outcomes that a physician should focus on in their discussions with patients (e.g., the specific prostate cancer treatment side effects of sexual, urinary and bowel problems) and identify the best available means of making a shared decision with patients (e.g., using a PSA screening decision aid)33–35. In this manner, physicians can help patients make the choices most consonant with their own personal values. It is encouraging that we found more recent CPGs appeared to better incorporate preference evidence. While the above barriers to using preferences will be difficult to overcome, a positive first step would be the acknowledgment that this type of evidence does exist and is not being utilized to its full potential.
Case Resolution You note that after the ACCP recommendation to use aspirin in your patient with atrial fibrillation, there is a note, based on preferences studies, stating that “Individual lower-risk patients may rationally choose anticoagulation over aspirin therapy to gain greater protection against ischemic stroke if they value protection against stroke much more highly than reducing risk of hemorrhage and burden of managing anticoagulation”1. To help your patient better understand the potential risks and benefits, you refer her to an atrial fibrillation decision aid available on the Internet36. Your patient is not adverse to monitoring anticoagulation, and given the high value she places on stroke prevention, you both decide to opt for warfarin therapy.
Below is the link to the electronic supplementary material.
(DOC 84 KB)
The authors would like to thank Dr. Andreas Laupacis for discussing this project’s feasibility, Dr. Antoine Boivin for helpful discussions and for sharing his unpublished work, Drs. Rachel Chong, Geoffrey Nguyen and Angeline Chong for reviewing drafts, and the Toronto Health Economics Network for their comments on this manuscript. Funding/Support: Ms. Chen was supported by a Student Research Award from the Faculty of Pharmacy, University of Toronto. Dr. Naglie is supported by the Mary Trimmer Chair in Geriatric Medicine at the University of Toronto. Dr. Krahn is supposed by the F. Norman Hughes Chair in Pharmacoeconomics at the University of Toronto.
Conflict of interest None disclosed.
Electronic supplementary material
The online version of this article (doi:10.1007/s11606-009-0987-8) contains supplementary material, which is available to authorized users.