|Home | About | Journals | Submit | Contact Us | Français|
Conceived and designed the experiments: SLN HKH BUB. Analyzed the data: SLN HKH BUB LAO RF. Wrote the paper: SLN. Revised and approved manuscript: SLN HKH BUB LAO RF.
Conflict of interest (COI) of clinical practice guideline (CPG) sponsors and authors is an important potential source of bias in CPG development. The objectives of this study were to describe the COI policies for organizations currently producing a significant number of CPGs, and to determine if these policies meet 2011 Institute of Medicine (IOM) standards.
We identified organizations with five or more guidelines listed in the National Guideline Clearinghouse between January 1, 2009 and November 5, 2010. We obtained the COI policy for each organization from publicly accessible sources, most often the organization's website, and compared those polices to IOM standards related to COI. 37 organizations fulfilled our inclusion criteria, of which 17 (46%) had a COI policy directly related to CPGs. These COI policies varied widely with respect to types of COI addressed, from whom disclosures were collected, monetary thresholds for disclosure, approaches to management, and updating requirements. Not one organization's policy adhered to all seven of the IOM standards that were examined, and nine organizations did not meet a single one of the standards.
COI policies among organizations producing a large number of CPGs currently do not measure up to IOM standards related to COI disclosure and management. CPG developers need to make significant improvements in these policies and their implementation in order to optimize the quality and credibility of their guidelines.
Clinical practice guidelines (CPGs) are “statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options,” according to the Institute of Medicine (IOM) . CPGs can influence the care delivered by a large number of healthcare providers and thus the outcomes of patients , so their quality is critically important. High-quality guidelines have the potential to promote the use of effective clinical services, decrease undesirable practice variation, reduce the use of services that are of minimal or questionable value, increase the use of effective but underused services, and target services to populations most likely to benefit . High-quality guidelines are valid (unbiased), reproducible, clinically applicable to the populations of interest, flexible, clearly presented, developed through a multidisciplinary process, reviewed on a regular schedule, and well-documented . CPGs may improve processes of care , , , however data on the effectiveness of CPGs on health outcomes are sparse and conflicting , , –.
Conflict of interest (COI) is an important potential source of bias in the development of CPGs. A COI is a set of conditions in which professional judgment concerning a primary interest (such as the health and well being of a patient or the validity of research), is unduly influenced by a secondary interest . There are data that suggest an association between author or funder COI and study outcomes – and between COI and systematic review conclusions . Data on the prevalence of industry relationships of guideline sponsors and authors are limited and dated, however –, and evidence of the effect of these relationships on guideline recommendations is confined to case studies . Nonfinancial interests may also be powerful motivators for guideline authors , including the advancement of medical science, career advancement, fulfillment of a desire to do good, opportunity to publish, notoriety, future success in obtaining funding for research, and increased sense of self worth .
Recent IOM reports highlight the importance of COI in healthcare , systematic reviews , and CPGs , and provide standards for disclosure and management of COI. The extent to which current CPG organizational policies align with IOM standards has not yet been reported. The objective of this study was to describe the COI policies of organizations currently producing a large number of CPGs, to compare these policies to the recent IOM standards related to COI in CPG development , and to identify characteristics of organizations that correlate with policies that meet these standards.
We identified organizations that had five or more guidelines listed in the National Guideline Clearinghouse (NGC) (http://www.guideline.gov) that were published between January 1, 2009 and the date of our search (November 5, 2010). We selected this cohort in order to examine the policies of organizations that have the potential for a major effect on physician behavior and patient outcomes. The NGC, funded by the Agency for Healthcare Research and Quality (AHRQ), was established in January of 1999, “to provide physicians and other health professionals, health care providers, health plans, integrated delivery systems, purchasers, and others an accessible mechanism for obtaining objective, detailed information on clinical practice guidelines and to further their dissemination, implementation, and use” . The inclusion criteria for guidelines within the NGC are:
Two authors (HKH, BUB) independently searched for each organization's COI policy on the organization's website and through a general internet search. Only policies that were specific to CPG development were examined further: the policy was required to either include a statement specifically regarding CPGs or be included in a CPG development handbook, and at a minimum address COI for CPG panel members. Policies that covered a variety of organizational activities were not examined further because such general policies did not address adequately address the issues of COI relevant to CPG development. If the guideline developer was US-based, the authors also determined if the CPG organization was signed onto the Council of Medical Specialty Societies' (CMSS) Code for Interactions with Companies, by examining the CMSS website . CMSS, a collaboration of almost 40 US-based medical specialty organizations, issued a revised code in March, 2011 to provide guidance for members on how to minimize and manage conflicts of industry that arise from their relationships with companies that produce drugs, devices, and other healthcare interventions. Underpinned by the principles of independence and transparency, the CMSS code provides specific guidance for the conduct of educational programs, business interactions, receipt of funding support, sponsorship, and advocacy, among others.
We did not include information contained within the CPG or in the summary provided by the NGC as the former was invariably insufficient to assess the organization's policy and the latter was information provided by the organization to the NGC after the guideline was completed. We did not contact organizations for additional information, nor did we try to obtain information from organizations with protected websites for members only, because we wanted to include only information that was publicly available to users of CPGs, and our past experiences trying to obtain unpublished COI policies suggested that such an approach was not likely to be productive.
One author abstracted information on each organization's COI policy into a pre-specified template in Excel®, and those data were checked by a second author. The data abstracted from each policy included when disclosures were obtained, from whom, over what time period, the types of COI (financial or intellectual), and how disclosures were managed. Data were synthesized across guidelines using descriptive statistics and a qualitative synthesis. We used a logistic regression model to explore the association between whether an organization had a CPG-specific COI policy and pre-specified predictors included type of organization, the number of CPGs produced between 2009 and our search date, and whether or not the organization was based in the US.
We compared the COI policy of each guideline organization to the COI-related standards in the 2011 IOM report (Table 1)  for the purpose of describing the degree to which organizations are already meeting these standards and the gaps that need to be addressed. We selected the IOM standards as a comparator for our cohort of guidelines because this report is the most recent, large-scale compendium of evidence and recommendations for guideline development in healthcare that we are aware of. We are aware of extensive efforts across North American guideline groups to implement these standards. We assessed the seven IOM standards that were deemed evaluable at the level of the organizational policy (standards 2.1, 2.2a, 2.2b, 2.3, 2.4a, 2.4c, 2.4d). We did not assess standards that were best assessed at the level of the CPG (standard 1.1) or standards that were unlikely to be included in an organization's COI policy (standard 2.4b, 2.4e).
Three authors (SLN, HKH, BUB) independently made a determination as to whether each organization's policy met each IOM standard and then consensus was achieved through discussion. Inter-rater reliability was assessed by using Kappa statistics for each standard and overall for the seven standards combined.
We then explored pre-specified predictors for organizations fulfilling these IOM standards, using the same variables that we used in our examination of whether each organization had a COI policy or not. The association between meeting each standard or at least one standard and the predictors was assessed using a logistic regression model. The association between meeting any IOM standard and the predictors was assessed using a logistic generalized estimating equation (GEE) model, accounting for correlation among multiple standards within each organization. The association between the proportion of standards met in each organization and the predictors was explored using a linear regression model.
596 CPGs were published in NGC between 2009 and the date of our search in 2010, by 118 organizations listing at least one guideline. Of these organizations, 37 fulfilled our inclusion criteria, publishing 392 guidelines (66% of the total of 596 CPGs) (Table 2). The majority of the 37 organizations (86.5%) published fewer than 20 guidelines, while one organization published more than 40 . Professional (56.7%) and government (29.7%) organizations made up the majority of the 37 organizations.
Seventeen of the 37 organizations (45.9%) had a COI policy that was publicly accessible and applicable to the development of CPGs (Table 3). The remaining organizations either did not have a policy or it was not publicly available (43.2%), or the only identified policy was not explicitly related to CPG development (10.8%). Organizational characteristics were not significantly associated with the presence of a COI policy, including the production of fewer CPGs (p=0.24), governmental compared to a professional organization (p=0.65), and based in the US compared to other countries (p=0.17).
Twelve – of the 17 organizations (71%) with a COI policy implemented their policy within the last 5 years; one organization  had a policy dating back to 2003. Four organizations , – signed onto the CMSS Code of Interactions with Companies , however, only one organization  acknowledged their CMSS affiliation website and within their COI policy.
COI was variably defined across the 17 policies (Table 3), of which 15 included the terms “financial” or “commercial” –. Disclosure of specific financial relationships varied among organizations: the most common types were paid consultancies, research or salary support, patents or royalties, equity or stock, and gifts. Only three organizations , ,  specified a dollar-value threshold for required reporting of a financial interest.
The types of non-financial COI described in the policies also varied (Table 3). Eleven organizations (65% of 17) specifically required that intellectual or other non-financial COI be disclosed , , , , –. Four of these organizations did not provide a definition for non-financial COI, but rather used terms such as “intellectual” or “other related” conflicts , , , . The remaining seven organizations provided more detailed, and very heterogeneous, descriptions , , –, , . One organization required the disclosure of competing beliefs, academic institutions, societies and publications , while another organization required the reporting of substantial career efforts (effort devoted to procedures currently performed in clinical practice) . One organization  used the definition of intellectual COI as proposed by Guyatt and colleagues  where an attachment to a specific viewpoint as developed through academic and other activities could unduly affect an individual's judgment about a specific guideline recommendation.
All organizations required disclosure from all guideline development group members (Table 4). The editorial board (5.8%), systematic review authors (5.8%), expert reviewers (17.6%), and other guideline contributors (5.8%) such as expert advisors and staff were also specifically required to disclose in some policies. Seven organizations (23.5%) required that the disclosure policy extend to relatives or dependents of guideline development group members , –, , , .
Disclosures are collected at various time points across the organizations, (Table 4) most commonly at the time of appointment to a guideline panel or prior to beginning deliberations. Two organizations addressed updating of COI, either throughout the CPG process  or just prior to CPG publication . The reporting period for COI ranged between the prior 12 to 60 months (median 12 months).
Seven organizations (41.2%) limited the disclosure to only those that were relevant to the content of the CPG (Table 2) , , , –, . In most organizations relevance was not defined, however, although three organizations that shared a common COI policy require disclosure for personal payments, “which may either relate to the manufacturer or owner of a product or service being evaluated, in which case it is regarded as ‘specific’ or to the industry or sector from which the product or service comes, in which case it is regarded as ‘non-specific’ ” –.
Eleven organizations (64.7% of 17) delineated strategies to manage COI after disclosure, with considerable variation in these approaches (Table 4). Advisory panels (5.8%), the CPG committee (23.5%), committee chairs (5.8%), or another party (e.g., funder of the CPG) (17.6%) assessed the disclosures. In some cases the same body of individuals assessed and managed the disclosures (17.6%). More commonly, however, a party separate from those involved in the assessment of disclosures devised a management plan: advisory panel (17.6%), the CPG chair (5.8%), or an ethics committee (11.8%). For 47% of organizations, a significant conflict resulted in recusal from the CPG development group or abstention from the final recommendation process. One policy detailed penalties for non-compliance with the organization's policy , while two other organizations stated that non-compliance would lead to disciplinary action , .
None of the 17 organizations with a CPG-specific COI policy met all seven IOM standards specific to COI (Table 5). The policies of eight organizations , , , , –, , , however, met at least one of the seven standards, with five organizations meeting at least four , , –. The standard most frequently met was 2.1 (n=7) and 2.2a (n=8). None of the organization characteristics was statistically significantly associated with meeting any standard, including number of CPGs listed in NGC by that organization in our 2009 to 2010 cohort (p=0.21); professional organization compared to government sponsor (p=0.81); and US-based organization versus other countries (p=0.85). Regression models examining these same predictors for meeting each individual standard, at least one standard, as well as the proportion of standards met for each organization were also not significant (p<0.05).
The agreement among the three assessors for all seven IOM standards across all 17 policies was 94.1% (95% CI, 88.3% to 97.6%). Agreement for specific standards ranged from 76.5% (standard 4a) to 100.0% (standards 2b, 3, 4d). The overall Kappa statistic for inter-rater reliability among the three reviewers was 0.88.
Fewer than half of organizations that have recently listed a large number of guidelines in the NGC have a COI policy directly related to CPGs. Of organizations with such policies, these vary widely with respect to types of COI addressed, from whom disclosures are collected, monetary thresholds for disclosure, approaches to management, and updating requirements. Not one organization's policy adhered to all seven of the IOM 2011 standards applicable to COI policies for CPGs, and approximately half of these organizations do not adhere to a single one of these standards.
There is a need for extensive improvements in COI policies for organizations producing CPGs. Every developer must have a COI policy specific to CPGs: a general COI policy for the organization or a policy that relates to research subjects, for example, is not adequate to address the issues of secondary interests relevant to guideline development. Guidelines involve multiple groups of individuals and processes, and the COI of all of these must be disclosed: authors of the systematic review, the guideline panel members, and other individuals involved in the formulation and approval processes. COI policies for CPGs must also address management of the disclosures, starting with the requirement that the guideline panel chair and co-chair have no relevant COI.
There are important implications of our findings for users of CPGs. Given the evidence that financial relationships of researchers are associated with study findings –,  and that guideline panel composition correlates with panel recommendations , the reader needs to have ready access to accurate and up-to-date disclosures, and be assured that secondary interests have been appropriately managed.
Nonfinancial COI may be an important source of bias, and two-thirds of organizations make some mention of this type of COI in their policies. Definitions are variable, however, and often vague and difficult to interpret, potentially leading to inconsistent and incomplete reporting. The current knowledge base on the impact of nonfinancial COI on decision making is sparse, so that the development of evidence-based guidance on the disclosure and management of nonfinancial COI is difficult at the present time.
The IOM standards related to COI do not address all of the important issues for CPG developers and users. In particular, these standards do not address accessibility of COI policies, the relevance of secondary interests to the primary interest, and the optimal presentation of disclosures. We often encountered a great deal of difficulty locating COI policies: these policies should be readily accessible from the guideline organization's website. Policies on disclosures should address both relevance and an appropriate level of detail. Inadequate guidance within COI policies on these issues may reflect lack of evidence on both the relationship between specific types of conflicts and risk of bias, and what constitutes optimal disclosures for the reader. IOM standards also do not address accountability for accurate disclosures and adherence to the policy within the organization. In other words, adherence to these seven IOM standards does not ensure an optimal COI policy.
There are limitations to our approach. Most importantly, we did not assess if and how these policies were applied during CPG development and publication. Nor did we examine the impact of each organization's policy on their processes, disclosures, recommendations, and the use and interpretation of the guidelines by healthcare providers. Our descriptive work provides the basis for these important next steps in evaluating the nature and effect of COI policies in CPG development.
It is possible that our summary of these organizations' policies is not up to date. Given the current attention to COI policies by biomedical journals ,  and organizations producing CPGs , , the COI policies for the organizations that we examined may have changed between development of the CPGs listed on the NGC website, our examination of the policy, and the present. Our information on specific policies may be incomplete in spite of having two independent persons identifying COI policies due to difficulty locating policies in various locations in linked websites.
We did not contact organizations for additional, unpublished information, nor did we try to obtain information from organizations with protected websites. We felt that if the guideline was in the public domain the COI policy should be there also, given the presumed importance of transparency in COI processes and disclosures. We also encountered challenges in determining if an organization's COI policy was specific to CPGs. Two independent reviewers made this assessment and then came to consensus, but it is possible that some of the organizations that we assessed as not having a CPG-specific policy would consider themselves to have such a policy. We would counter that these policies need to be more transparent and to address the processes and individuals involved in CPGs specifically.
The applicability of our findings to other organizations producing CPGs may be limited. The COI policies of organizations producing a small number of CPGs (less than 5) may differ from the policies that we examined. In addition, we focused only on CPGs published in English and those listed in NGC: the policies of other organizations may differ from our findings. The IOM standards were developed by a US-based group of experts, and thus encompass the American perspective on guideline topics, methodology, available resources, and the standards target U.S. patient populations and health care providers. The evidence base examined in the IOM report encompasses studies from a variety of international settings, and thus the IOM standards are likely applicable to guideline development in the Western world.
The IOM standards  were released after the publication of the CPGs examined herein. Thus our goal in comparing the IOM standards to existing organizational polices was to describe the current status of these policies and to identify areas where improvements are needed and not to criticize organizations for not having met these standards. Our reporting of the gaps in current approaches may aid organizations in planning updates of their COI policies and in comparing policies across organizations.
The IOM standards were not intended to be a quality assessment tool, thus implementing them as such was problematic. Standard 2.1 addresses both financial and nonfinancial COI, thus the two types of COI are not distinguished. Many of the standards are too vague to implement as quality assessment criteria: for example, standard 2.4a (Table 1).
COI policies among organizations producing a large number of CPGs currently do not measure up to IOM standards. Policy-makers, guideline funders, sponsors, and developers, as well as users need to address and demand improvements. Patients and populations need trustworthy CPGs, and the accurate disclosure and subsequent management of COI is essential to achieve that goal.
Competing Interests: SLN has participated in the development systematic reviews for the United States Preventive Services Task Force (USPSTF) and is a member of the GRADE Working Group. BUB is employed by the Oregon Evidence-based Practice Center at Kaiser Permanente Center for Health Research and participates in the development of evidence reviews and methodology for the USPSTF. This does not alter the authors' adherence to all the PLoS ONE policies on sharing data and materials.
Funding: This project was supported by grant number 1R01HS018500 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.