|Home | About | Journals | Submit | Contact Us | Français|
Risk-stratified treatment recommendations facilitate treatment decision-making that balances patient-specific risks and preferences. It is unclear if and how such recommendations are developed in clinical practice guidelines (CPGs). Our aim was to assess if and how CPGs develop risk-stratified treatment recommendations for the prevention or treatment of common chronic diseases.
We searched the United States National Guideline Clearinghouse for US, Canadian and National Institute for Health and Clinical Excellence (United Kingdom) CPGs for heart disease, stroke, cancer, chronic obstructive pulmonary disease and diabetes that make risk-stratified treatment recommendations. We included only those CPGs that made risk-stratified treatment recommendations based on risk assessment tools. Two reviewers independently identified CPGs and extracted information on recommended risk assessment tools; type of evidence about treatment benefits and harms; methods for linking risk estimates to treatment evidence and for developing treatment thresholds; and consideration of patient preferences.
We identified 20 CPGs that made risk-stratified treatment recommendations out of 133 CPGs that made any type of treatment recommendations for the chronic diseases considered in this study. Of the included 20 CPGs, 16 (80%) used evidence about treatment benefits from randomized controlled trials, meta-analyses or other guidelines, and the source of evidence was unclear in the remaining four (20%) CPGs. Nine CPGs (45%) used evidence on harms from randomized controlled trials or observational studies, while 11 CPGs (55%) did not clearly refer to harms. Nine CPGs (45%) explained how risk prediction and evidence about treatments effects were linked (for example, applying estimates of relative risk reductions to absolute risks), but only one CPG (5%) assessed benefit and harm quantitatively and three CPGs (15%) explicitly reported consideration of patient preferences.
Only a small proportion of CPGs for chronic diseases make risk-stratified treatment recommendations with a focus on heart disease and stroke prevention, diabetes and breast cancer. For most CPGs it is unclear how risk-stratified treatment recommendations were developed. As a consequence, it is uncertain if CPGs support patients and physicians in finding an acceptable benefit- harm balance that reflects both profile-specific outcome risks and preferences.
An important goal of evidence-based health care is to maximize benefits and minimize harms from medical treatments. To achieve an optimal balance, patients' individual profiles and preferences need to be considered . For example, inhaled corticosteroids are used to prevent exacerbations in patients with chronic obstructive pulmonary disease (COPD) [2-4], but these drugs are associated with an increased risk for pneumonia and fractures [5,6]. In patients at high risk for exacerbations, the potential benefits (preventing exacerbations) are likely to be larger than harms, while patients at low risk for exacerbations may experience more harms from inhaled corticosteroids than benefits.
Risk-stratified treatment recommendations are potentially useful to support personalized medicine. Personalized medicine aims at optimizing the benefit-harm balance by considering patient profiles (combination of characteristics) and preferences . For the prevention and treatment of chronic disease, most health care decisions are sensitive to patient profiles and preferences . Risk-stratified treatment recommendations suggest different treatment regimens for patients who are at different risks for outcomes . For example, in the Third Report of the National Cholesterol Education Program's Adult Treatment Panel treatment algorithm , the recommendation for primary prevention of coronary heart disease is based on the Framingham Risk Score. According to different risk categories predicted by the Framingham Risk Score, individuals with higher predicted absolute risk (10-year risk > 20%) are recommended for more intensive treatments (such as combined pharmacological and non-pharmacological treatments) than those with lower predicted risk (10-year risk < 10%). There is evidence that using risk-stratified treatments is superior to treatments that are not informed by a risk assessment tool [11-13].
Risk-stratified treatment recommendations only serve their purpose of supporting personalized medicine if valid methods were used to develop them. Because it is not known what proportion of clinical practice guidelines (CPGs) make risk-stratified treatment recommendations and what methods were used to develop them, our aim was to assess the methods CPGs applied in developing risk-stratified treatment recommendations for the prevention or treatment of selected common chronic diseases.
We started by forming a framework for developing risk-stratified treatment recommendations. Figure Figure11 outlines the major steps for developing risk-stratified treatment recommendations, each of which requires high quality evidence from observational studies (development and validation of risk assessment tools), randomized trials (evidence about treatment effects) and studies to elicit patient preferences (using various study designs, for example, discrete choice experiments). It is well known for all guidelines that evidence about treatment effects on benefit and harm outcomes must be available. In addition, a risk assessment tool should be available that allows the assigning of patients to different risk categories. A method is required to estimate how treatment evidence applies to patients at different risks and how the benefits compare to the harms in patients at different risks. As a result of such a benefit-harm assessment, treatment thresholds can be defined for patients with different risk profiles that maximize the chance for benefits while minimizing harms. In addition, patient preferences for outcomes would ideally be explicitly considered for the development of risk-stratified treatment recommendations or their application in practice.
We performed an environmental scan of CPGs, which included a limited literature search (described below) but not a comprehensive, systematic review of all CPGs. We focused on CPGs for major chronic diseases and from the United States (US), Canada, or the United Kingdom (UK) National Institute for Health and Clinical Excellence (NICE). The completed PRISMA checklist is available as Additional file 1.
We searched the US National Guideline Clearinghouse (NGC) database on February 5 2011 for CPGs with treatment recommendations for five major chronic diseases. The top five chronic diseases in the US are heart disease, cancer, stroke, COPD and diabetes, accounting for more than two-thirds of all deaths . In the NGC database, guidelines were categorized by disease topics that were linked to a specific term derived from the US National Library of Medicine's Medical Subject Headings classification.
For heart disease and stroke, we performed our search within the Cardiovascular Diseases section of the database (n = 442) and considered CPGs specific for primary prevention of heart disease and stroke, that is, the prevention of an event in persons free of established cardiovascular diseases. For cancer, we chose to examine three types of cancer with the highest mortality rates in the US (lung cancer, prostate cancer and breast cancer) . We searched for CPGs within the Lung Neoplasms (n = 53), Prostatic Neoplasms (n = 26) and Breast Neoplasms (n = 52) sections, respectively. For COPD, we considered CPGs specific for COPD within the Respiratory Tract Diseases section (n = 102). For diabetes mellitus, we considered CPGs for type II diabetes within the Diabetes Mellitus, Type 2 section (n = 44).
We included CPGs that recommended using risk assessment tools to inform treatment decisions. Risk assessment tools are tools to calculate the probability of developing an event or a disease based on a prediction model (binary outcome), or tools that make projections about the course of disease measured by patient-reported or other continuous outcomes (for example, decline of functional status over time). We excluded CPGs if they were not from the US, Canada or NICE (UK); focused on childhood diseases; made recommendations on screening, genetic counseling or diagnostic work-up alone; or did not use any risk assessment tools to inform risk-stratified treatment decisions. This latter excluded category involved guidelines that recommended treatments according to diagnostic criteria, as for example based on pathological staging, rather than according to prognostic information (for example, the risk stratification scheme proposed by D'Amico et al. in prostate cancer guidelines ).
Two reviewers (TY and DV) independently reviewed the Guideline Summary section of each CPG on the NGC website to assess its potential eligibility. We excluded the CPGs labeled ineligible by both reviewers. For the other CPGs, we retrieved and examined the full text and resolved any discrepancies in eligibility through discussion or arbitration by a third reviewer (MP).
We developed a standardized form to extract data from the included CPGs and the background documents detailing the methods used in developing CPGs when available. We extracted general items such as guideline title, bibliographic source, date released and guideline developer. We then extracted information related to five key components for developing risk-stratified treatment recommendations (Figure (Figure1).1). We extracted the following information on risk assessment tools: the name of the prediction model, the outcome and the timeframe (for example, 10 years) used in the model, and whether validation of the model (for example, assessment of discrimination and/or calibration) was discussed in the CPGs. We extracted information on the type of evidence used to determine the effects of treatments on benefit and harm outcomes (observational studies, single or several randomized controlled trials (RCTs), or meta-analyses). We recorded the methods to link risk prediction and evidence on treatment effects (for example, applying relative risk reductions to different absolute risks calculated from the risk assessment tool). We recorded the way the treatment benefits and harms were assessed and how treatment thresholds (based on risk assessment tools) were determined. We also extracted information on assumptions made for linking risk prediction and treatment evidence (for example, assumption of constant relative risk reductions across the risk spectrum) and on assumptions made for the assessment of benefits and harms (for example, assumption that benefit and harm outcomes can be put on a single scale and the overall net benefit expressed as a single number indicating benefit or harm). Finally, we noted whether patient preferences (for example, relative importance of different benefit and harm outcomes) were considered for developing risk-stratified treatment recommendations. Because some CPGs were very brief, without detailing the development process but referring to other documents, we considered those documents for data extraction to avoid underestimating the rigor of the development process of a CPG. Two reviewers (TY and DV) independently extracted all relevant information from each CPG and the discrepancies were resolved by discussion or third-party (MP) arbitration. We constructed a table for comparison of recommendations from each of the included CPGs.
Most CPGs that we excluded (Figure (Figure2)2) were on topics not related to our study question or because they were not from the US, Canada or NICE (UK). We excluded 60 CPGs based on NGC website review and 49 CPGs based on the full text because they did not recommend using a risk assessment tool. We excluded four additional CPGs because they recommended using a risk assessment tool but did not make any link to treatments (Figure (Figure2).2). Thus out of 133 CPGs that made treatment recommendations for the chronic diseases of interest (60 + 49 + 4 + 20 = 133 CPGs), 20 made risk-stratified treatment recommendations (15%) for heart disease, stroke, type II diabetes or breast cancer (Figure (Figure2)2) [10,17-36]. The characteristics of the 20 included CPGs are summarized in Table Table11.
A large proportion of CPGs (16 out of 20, 80%) were on type II diabetes or on primary prevention of heart disease and stroke [10,18-32], and the remaining four CPGs were on breast cancer management [33-36]. All 16 CPGs on diabetes or cardiovascular disease recommended one or several risk assessment tools to assess 10-year cardiovascular disease risk. The Framingham Risk Score was explicitly suggested in 12 CPGs [10,19-29]; the UK Prospective Diabetes Study Risk Engine was suggested in four CPGs [21,24,30,31] for patients with diabetes, and the Prospective Cardiovascular Münster and the Systematic Coronary Risk Evaluation risk tools were used in one CPG . Of the 16 CPGs on diabetes or cardiovascular disease, two (13%) did not clearly specify the prediction model used for calculating 10-year cardiovascular disease risk [18,32]. Among the four CPGs on breast cancer [33-36], two recommended using the National Cancer Institute breast cancer risk assessment tool to calculate 5-year risk of invasive breast cancer [33,35]; one recommended using the Nottingham Prognostic Index to calculate 10-year survival ; one mentioned different risk assessment tools but did not clearly define risk categories . Information on the validation of the risk models was reported in seven (35%) of the 20 included CPGs (Table (Table2)2) [10,19,20,22,29,30,35].
Of the 16 CPGs for type II diabetes and primary prevention of heart disease and stroke, nine (56%) suggested specific target lipid levels for each risk category when making recommendations about lifestyle management or pharmacotherapy (for example, aspirin, statins and antihypertensive drugs) [10,19-21,26-29,31]. The four CPGs on breast cancer [33-36] provided recommendations on surgery or pharmacotherapy (for example, tamoxifen, raloxifene and aromatase inhibitors) according to risk levels.
Sixteen (80%) of the 20 CPGs reported using evidence on treatment benefits from RCTs or meta-analyses or other guidelines [10,18-25,27,29-32,35,36]. One CPG (5%) did not report quantitative information on treatment benefits  and three (15%) did not clearly specify the type of studies considered [26,33,34]. Treatment harms were only reported in 13 of the 20 CPGs (65%) [10,18,22,23,25-27,29,30,32,33,35,36]. The source of evidence on harms was specified in nine out of these 13 CPGs (69%) and included observational studies, RCTs and meta-analyses [10,18,32,23,25,27,30,35,36]. Heterogeneity of treatment effects were assessed in eight (40%) of the included 20 CPGs (Table (Table2)2) [10,18,22,23,25,26,32,35].
In reviewing how CPGs made the link between risk prediction and treatment effects, we found fewer than half of the CPGs (eight out of 20, 40%) explicitly or implicitly stated that they applied evidence of relative risk reductions from RCTs and/or meta-analyses to different absolute risks [10,18,23-27,32]. For instance, the U.S. Preventive Services Task Force (USPSTF) guideline  applied a 32% risk reduction of myocardial infarction (in men) and a 17% risk reduction of strokes (in women) with regular aspirin use to absolute outcome risks and assumed that the effects were constant across risk levels and age categories. One (5%) of the 20 included CPGs , instead of applying treatment evidence to all risk levels, used the evidence from RCTs with the same risk profile (high breast cancer risk) population for which the recommendation was made. Eleven (55%) of the included CPGs did not report the way in which they linked risk prediction to treatment effects (Table (Table2)2) [19-22,28-31,34-36].
Only a small proportion (two out of 20, 10% [25,35]) of CPGs explicitly stated that they planned to perform benefit and harm assessment as the basis for making risk-stratified treatment recommendations. To define treatment thresholds, only the USPSTF guideline quantitatively weighed treatment benefits and harms by putting the expected benefit and harm outcomes on the same scale (events per 1,000 persons treated over 10 years). The USPSTF guideline recommended using aspirin when the treatment benefits (number of myocardial infarctions or strokes prevented per 1,000 persons treated over 10 years) outweigh the treatment harms (number of gastrointestinal bleedings or hemorrhagic strokes per 1,000 persons treated over 10 years). For example, the expected number of myocardial infarctions prevented by aspirin was estimated to be 16 per 1,000 men of age 60 to 69 years if men had a 10-year risk for myocardial infarction of 5%, while the expected number of excess gastrointestinal bleedings was 24 and hemorrhagic strokes was one. Because the number of excess events exceeded the number of prevented myocardial infarctions, the USPSTF recommended against the use of aspirin in men at 5% risk for myocardial infarction and an age of 60 to 69 years. Based on observational studies, the USPSTF assumed different risks for gastrointestinal bleeding with aspirin according to age. Finally, the USPSTF presented their benefit-harm assessment and the resulting treatment thresholds as a matrix table with categories for age and risk for myocardial infarction defining each cell.
Three (15%) of the 20 CPGs qualitatively weighed the treatment benefits and harms [23,29,32]. Nine (45%) of the 20 CPGs made the recommendation on thresholds based on expert consensus or referred to other guidelines [18,19,21,22,26-28,33,33]. Seven (35%) of the 20 CPGs did not report how they determined the treatment thresholds when making recommendations [10,20,24,30,31,34,36]. With regard to involving patient preferences when developing treatment recommendations, only three (15%) of the 20 CPGs explicitly reported that they considered patient preferences in the process (Table (Table2)2) [25,30,36]. For example, the USPSTF focused on major benefit (myocardial infarction) and harm events (gastrointestinal bleeding and hemorrhagic stroke) and assumed equal preferences (that is, importance) for those outcomes.
We found a rather small proportion of CPGs for heart disease, cancer, stroke, COPD and diabetes that made risk-stratified treatment recommendations using risk assessment tools. Most of these CPGs recommend risk assessment tools that had been shown to accurately predict outcome risk in the target population of the CPGs and most of the treatment evidence is based on RCTs and meta-analyses. For the majority of the CPGs, however, it was not explicitly explained how treatment effects on benefit and harm outcomes were estimated for patients at different risks. Perhaps most importantly, it was unclear for all but one CPG how treatment thresholds were determined to generate risk-stratified treatment recommendations.
We formed a framework for the development of risk-stratified treatment recommendations (Figure (Figure1)1) to systematically identify the strengths and weaknesses of current CPGs. Our findings suggest that risk assessment tools were carefully appraised and selected during the development of CPGs. For example, some CPG developers critically appraised validation studies of risk tools to judge their calibration (agreement between predicted and observed risk) and discrimination (probability that those with an event receive higher risk predictions that those without an event) [10,30]. Minimizing misclassification of outcome risks is important to avoid over- or under-treatment [37-39]. While some CPGs recommended specific risk assessment tools, one CPG suggested using the risk assessment tool that is most likely to be accurate in the specific population of interest . However, the set of CPGs selected in this study may give an overoptimistic picture of risk assessment tools proposed by guidelines. For many diseases and geographical locations other than the US, Canada and the UK, calibrated and discriminative risk assessment tools may not exist. A strength of existing CPGs is that the majority of them relied on RCTs and meta-analyses of RCTs for intervention effectiveness. The CPG developers recognized limitations within this body of evidence, including insufficient evidence on treatment heterogeneity (that is, subgroup effects) and scarcity of data on harm outcomes.
We discovered a number of major limitations in how CPGs develop risk-stratified treatment recommendations. It should be noted that some limitations propagated from single, prominent CPG (for example, National Cholesterol Education Program) to other CPGs that adopted the approach or even the recommendations. For example, it was often unclear how the benefit and harm outcomes were estimated for different risk profiles. Some CPGs applied estimates on relative risk reduction to absolute risks. This approach relies on the assumption of constant (relative) effects across the risk spectrum. This assumption of constant relative treatment effects may be justifiable in many instances but it is usually difficult to verify. No alternative approaches for linking the absolute risk with treatment evidence were used. Additional sensitivity analyses may sometimes be appropriate to explore the assumption of relative treatment effects. For example, one could obtain risk-specific treatment estimates from large trials using individual patient data . Or, one could employ simulation studies to estimate the probability of outcomes in the population of interest by combining observational data and treatment effects from randomized trials. It is currently unclear what the most appropriate approach is to link risk predictions with evidence from randomized trials. Nevertheless, we believe CPGs should be explicit about the method they use and acknowledge the associated advantages and limitations (for example, assumption of constant relative risk reduction).
In our view, the greatest limitation of current CPGs is that it is unclear how treatment thresholds were developed for most of them. Some CPGs stated that the thresholds were determined by experts. The USPSTF guideline on aspirin  was the only guideline that conducted a formal quantitative assessment by comparing the expected number of benefit and harm events for patients at different risk for myocardial infarction and major gastrointestinal bleeding. We believe that transparency will be enhanced by conducting quantitative benefit-harm assessments alongside more qualitative approaches, such as using expert consensus about treatment thresholds.
Treatment thresholds are important because medical decision-making is discrete (to treat the patient or not). It is challenging to determine thresholds because clear cuts on the (commonly) continuous benefit-harm scale may not exist. In addition, there may often be substantial uncertainty about harms and heterogeneity of treatment effects as a consequence of poor reporting or a lack of evidence from primary studies. However, this should, in our view, not prevent CPG developers from making risk-stratified recommendations because health care providers need evidence-based guidance nevertheless and because variability in delivering health care may be unacceptably high in the absence of guidance. Quanstrum and Hayward  recently suggested an approach that acknowledges uncertainty about treatment decision thresholds and proposed two thresholds instead of one: one above which physicians should recommend treatments (benefits outweighing harms irrespective of patient preferences and uncertainties about evidence base) and one below which physicians should recommend against treatments (harms outweighing benefits). The interval between the two thresholds represents an area where treatment could provide small benefits or harms depending on patient preferences but also where uncertainty about the evidence precludes CPG developers from making recommendations. Alternatively, CPG developers could frame strong recommendations for or against treatment for patients at outcomes risks above or below the two thresholds, respectively, and weak recommendations for patients at outcome risks between the two thresholds .
One may criticize the approach used by the USPSTF, assigning equal weight to benefit and harm outcomes to calculate events expected per 1,000 people treated over 10 years, because empirical evidence suggests that patients, on average, assign different importance to myocardial infarction, major gastrointestinal bleeding and major stroke, the major drivers of the benefit-harm balance of aspirin . Nevertheless, such transparency about the relative importance of outcomes comes with several important advantages. Users of CPGs can understand and replicate how the treatment thresholds were derived and, if they do not agree with certain assumptions (for example, equal importance of myocardial infarction and major gastrointestinal bleeding), they can adjust the result to derive thresholds that would suit their settings (for example, myocardial infarction considered twice as important as major gastrointestinal bleeding). This would also allow the guideline to be interpreted for an individual patient, who may weigh the various outcomes differently than those preferences assumed in the CPG.
The framework for developing risk-stratified treatment recommendation we proposed may be useful for those developing CPGs and to stimulate further research. While much research has been done on how to select and appraise evidence on treatment benefits and harms [43,44] and how to judge the validity of prediction models [37-39], it is less clear how to link risk prediction and treatment evidence, how to select a method for benefit-harm assessment to develop treatment thresholds, and how to include patient preferences. It would be useful to have empirical evidence on how the results of different approaches for linking risk prediction and treatment evidence and for defining treatment thresholds differ and how sensitive they are to assumptions . As for patient preferences, little research has been done to find ways to include stakeholders in the process of selecting important outcomes, or a benefit-harm assessment method that provides the information patients need in order to make decisions [46-48]. The newly founded Patient-Centered Outcomes Research Institute is likely to contribute substantially to the questions raised.
Our study has some weaknesses. We selected guidelines from five major disease categories and from one database and focused on CPGs from the US, Canada and NICE (UK). Thus our results may not be generalizable, but provide an optimistic assessment of CPGs because we included some of the most prominent guidelines in medicine. For the fields of cardiovascular medicine and diabetes, guideline developers have a long tradition of making risk-stratified treatment recommendations. We relied on published reports, which may not reflect the true underlying development process for CPGs. We considered all background documents that were openly accessible but we may have missed some information on the development of risk-stratified treatment recommendations.
We found that the methods for linking risk prediction with treatment evidence are often not reported and that it was unclear for all but one CPG how treatment thresholds were developed. Therefore, current CPGs for major chronic diseases may not support patients and physicians in finding an acceptable benefit-harm balance that reflects profile-specific outcome risks and preferences.
COPD: chronic obstructive pulmonary disease; CPG: clinical practice guideline; NGC: National Guideline Clearinghouse; NICE: National Institute for Health and Clinical Excellence; RCT: randomized controlled trial; USPSTF: U.S. Preventive Services Task Force.
The authors declare that they have no competing interests.
All authors contributed to the conception of the study; TY, DV and MP collected and analyzed the data; all authors interpreted the data; TY and MP drafted the article; all authors revised it critically for important intellectual content; and all authors read and approved the final manuscript.
The pre-publication history for this paper can be accessed here:
PRISMA checklist. The additional file shows the completed PRISMA checklist and shows where the checklist items are reported.
This publication was made possible by Grant Number UL1 RR 025005 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), and NIH Roadmap for Medical Research. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH. CB's effort was supported by the Paul Beeson Career Development Award Program (NIA K23 AG032910, AFAR, The John A. Hartford Foundation, The Atlantic Philanthropies, The Starr Foundation and an anonymous donor). MP had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.