|Home | About | Journals | Submit | Contact Us | Français|
Few studies have directly examined the relationship between commercial support of CME and perceived bias in the content of these activities.
Cross-sectional study of 213 accredited live educational programs organized by a university provider of CME from 2005–2007. We used a standard question from course evaluations to determine the degree to which attendees believed that commercial bias was present. Next, we used binomial regression models to determine the association between course features that may introduce commercial bias and the extent of perceived bias at those CME activities.
The mean response rate for attendee evaluations was 56% (SD 15%). Commercial support covered 20–49% of costs for 45 (21%) educational activities, and >=50% of costs for 46 (22%) activities. Few course participants perceived commercial bias, with a median of 97% (IQR 95%–99%) of respondents stating that the activity they attended was free of commercial bias. There was no association between extent of commercial support and the degree of perceived bias in CME activities. Similarly, perceived bias did not vary for 11 of 12 event characteristics that we hypothesized might introduce commercial bias, or by score on a risk index designed to prospectively assess risk of commercial bias.
Rates of perceived bias were low for the vast majority of CME activities in our sample, and did not differ by the degree of industry support or other event characteristics. Further study is needed to determine whether commercial influence persisted in more subtle forms that were difficult for participants to detect.
Commercial support for accredited continuing medical education (CME) has quintupled over the past decade. By 2006, commercial support for CME totaled $1.2 billion, accounting for 61% of total income for providers of CME.1–3 Over this same period, concern about commercial influence on CME has resulted in substantial efforts to reduce potential bias in accredited educational activities. A handful of institutions have completely forsaken commercial support, but most mainstream organizations, including the Accreditation Council for Continuing Medical Education and the American Association of Medical Colleges, have chosen to preserve commercial support while adopting increasingly more stringent regulations designed to limit the influence of industry funding on the content and objectivity of CME.4
A cornerstone of efforts to limit industry influence in CME is identifying and rejecting proposals for educational activities that pose an unacceptable risk of bias. To accomplish this, many not-for-profit providers of CME such as academic medical centers and professional organizations have adopted screening processes to identify activities at high risk of commercial influence. Historically, many of these screening processes have been based on the personal and collective experience of CME professionals. More recently, attempts have been made to develop standardized instruments to predict bias in CME. The Consortium of Continuing Academic Medical Education (CACME) has developed a standardized rating instrument to prospectively identify activities at high risk of commercial bias. This instrument, which rates activities on the basis of funding source, responsibility for course logistics, relationships between the course director and industry, and other such characteristics was found to have good inter-rater reliability in simulation testing.5 However, to our knowledge there is no published data about the relationship between CME activity characteristics that can be measured prospectively – such as source of funding, conflicts of interest of course chairpersons, or a risk index including these and other factors – and actual or perceived bias in real-world CME activities.6 Thus, little empiric data are available to judge the performance of mechanisms to prospectively identify CME activities at high risk of bias.
To address this question, we studied over 200 CME activities hosted by a major university provider of CME and evaluated the association between activity characteristics and ratings of bias made by course attendees after completing the course. First, we evaluated the extent of perceived bias across the range of CME activities. Next, we sought to determine whether a risk index in current use could prospectively identify activities with higher degree of perceived commercial bias as reported by participants. Finally, we sought to determine whether a variety of characteristics relating to potential commercial influence were associated with perceived bias at those CME activities.
We collected data on 228 directly-sponsored live CME activities hosted by the University of California, San Francisco (UCSF) Office of Continuing Medical Education from 2005 to 2007. Over this time, the UCSF Office of CME followed a highly-structured series of practices for reviewing proposals for CME activities. Applications for first-time activities underwent initial screening by senior staff to determine major sources of conflict of interest. Applications for activities which were considered to have unacceptable risk of commercial influence – for example, those with a single industry supporter - were not considered further.
The remaining applications were then submitted for peer review by faculty members of UCSF’s CME Governing Board. To assist with peer review, each application was separately evaluated by office staff using an internally-developed rating system for assessing an activity’s potential for commercial bias, ultimately scored as “lowest,” “intermediate,” or “highest” (see Box). After full review of each application, peer reviewers designated each activity as rejected, approved, or approved with audit. Courses that were approved with audit, typically those rated as intermediate or highest risk in the pre-peer review, required additional measures before and during the activity to mitigate risk. These measures included discussion with the course chair, advance review of presentation materials, and/or close observation of the CME activities.
At the completion of all CME activities, participants were asked to evaluate several characteristics of the activity, including an overall rating of the course (using a 5-point Likert scale), and a question about whether the activity was free of commercial bias (answered with a yes/no response). Data from individuals’ responses were combined to provide summary statistics for each activity.
Of 228 CME activities, data on perceived commercial bias was not available for 15. Thus, our final analytic cohort comprised 213 CME activities. Compared with our final cohort, the 15 activities without information on perceived bias had similar distribution of risk characteristics and overall quality scores, but had smaller numbers of participants (median number of participants 75 vs. 132, P=.02).
Our main outcome variable was the percentage of respondents at an activity who perceived that the activity was free of commercial bias. Higher numbers indicate less bias; for example, a score of 100% means that all respondents indicated the activity was bias-free. Our primary predictor variable was the UCSF Office of CME’s risk score for commercial bias (rated as “lowest,” “intermediate,” or “highest), as described above and in the Box. In addition, we separately evaluated each event characteristic used to construct this risk index, including factors such as whether the activity chair had ties with industry, whether registration fees were below market value, and so forth.
To evaluate the association between these predictor variables and learner-perceived bias, we employed binomial regression using the “binreg” command in Stata 10.0 in which we modeled the outcome variable as the number of respondents indicating commercial bias out of the total number of respondents. This effectively weights each course in proportion to the number of respondents. We incorporated an overdispersion adjustment to account for intra-activity correlation.7 Using these methods, we separately tested the association between each predictor variable and perceived commercial bias at each event (the outcome variable). In addition, we entered all predictor variables into a regression model to assess whether combinations of risk factors might predict the degree of perceived bias. As a sensitivity analysis, we repeated our core analyses using alternate methods to allow for intra-activity correlation. All methods produced generally similar estimates, and we report only our main analyses.
Finally, we hypothesized that the perception of bias in an activity might be reflected not only in a direct question about bias, but also in the perceived overall quality of the activity. Thus, we conducted additional analyses where our outcome variable was the overall quality of the activity, represented as the mean value of participants’ ratings on a 5-point Likert scale. For these analyses, we used linear regression models weighted by the number of evaluation respondents for each course. Due to the summative nature of the data on quality scores, our models could not precisely account for variance in responses within activities. As a result, these analyses are likely to have increased sensitivity to detect differences in perceived quality among activities with and without certain event characteristics.
This research was exempted from review by the Committee on Human Research at the University of California, San Francisco and by the Research and Development Committee at the San Francisco VA Medical Center.
Characteristics of 213 CME activities studied are shown in Table 1. The median number of participants per course was 132 (interquartile range [IQR] 92 – 194). One-third of courses had no commercial support, with the remaining courses split between varying levels of commercial support. Most activities were considered at “lowest” risk of commercial bias based on the UCSF Office of CME’s risk rating system.
The results of course evaluations are shown in Table 2. The mean response rate was 56% (SD 15%). Few respondents perceived commercial bias, with a median of 97% (IQR 95 – 99%) of respondents stating that the course they attended was free of commercial bias (Figure 1). The perceived overall quality of CME activities was high, with a mean of 4.4 (SD 0.2) on a 5-point Likert scale.
The UCSF Office of CME’s risk score for commercial influence was not associated with perceived bias, as we found no significant difference in perceived bias between CME activities classified as lowest, intermediate, and highest risk (P=.63). Similarly, level of commercial support was not associated with perceived bias (P=0.90). With the exception of the course director being a first-time chair, none of the individual risk factors we assessed were associated with perceived bias (Table 3). Simultaneous inclusion of all of these variables in a single model yielded limited results, with 3% of the total variance in perceived bias between activities explained by these risk factors.
Next, we evaluated several other event characteristics which we did not a priori hypothesize would be associated with perceived bias. Number of registrants was not associated with perceived bias (P=.46). However, the response rate to course evaluations was associated with perceived bias (P<.001), with an adjusted mean “bias-free” rating of 95.8% in activities with below-median response rates vs. 96.9% in those with above-median response rates. There was also a statistical trend for activities in later years to be considered less free of bias (adjusted mean “bias-free” ratings of 97.1% for activities in 2005 vs. 96.1% for activities in 2006 and 2007, P=.08). Together, number of registrants, response rate, and course year explained 11% of the total variance in perceived bias between activities. When all other predictor variables were added to the model, the total explained variance was 15%
Finally, we evaluated whether potential risk factors for commercial bias were associated with the overall quality of the activity. As with perceived bias, neither the risk score nor level of commercial support was associated with the overall quality of the activity. Similarly, the only variable significantly associated with overall quality was the presence of a first-time course chair (mean Likert score 4.23 for activities with first-time chair vs. 4.40 for activities without first-time chair, P=.01). When all risk factors used in the risk score were included in a single model, 9% of the variance in overall quality score was explained by these risk factors.
In this study of 213 educational programs organized through an academic provider of CME, the vast majority of CME activities were perceived to be free of commercial bias. Extent of commercial support, a variety of other event characteristics, and a summative risk index were not associated with the level of perceived bias or the perceived overall quality of CME activities. Stated otherwise, rates of perceived commercial bias were consistently low regardless of the presence or absence of risk factors for commercial bias.
There is a paucity of other research that has directly evaluated the impact of commercial support on the content of CME.6 However, other studies of bias in CME and on relationships between industry and medicine can shed light on potential explanations for our findings. The most direct interpretation of our results is that CME activities in general are free of commercial bias. Under rules of the Accreditation Council for Continuing Medical Education (ACCME), all accredited providers of CME must abide by the Council’s Standards of Commercial Support. These standards require accredited activities to be balanced and that conflicts of interest be disclosed and managed.8,9 However, these regulations cannot prevent all forms of commercial influence, and experts in drug industry promotion have identified widespread opportunities for commercial influence in CME and have documented industry marketing strategies that rely heavily on clinician education to boost drug sales.10–20 Other reports have documented that many physicians believe that industry-supported CME is biased.13,21,22 Finally, many commentators have noted that it is unlikely that industry would expend $1.2 billion per year to support CME if it did not help companies’ bottom line.1–3,11,23,24 Thus, although one interpretation of our findings is that industry support does not lead to bias in CME, other interpretations are possible as well.
One such alternate explanation is that the screening process instituted by the UCSF Office of CME successfully rejected activities with commercial bias that might have been permitted by other CME providers. This screening process is substantially more rigorous than prescribed in the ACCME’s Standards of Commercial Support, for example rejecting proposals with a single industry supporter or those with no cost to the participant. This screening process may have weeded out CME activities at higher risk of commercial bias, leaving only activities where commercial bias was more subtle or entirely absent. Furthermore, activities that were allowed to proceed but flagged as having intermediate or higher risk of bias were subject to mitigation procedures prior to and/or during the activity.
Finally, our findings of low rates of perceived bias may in part be explained by insensitivity of the simple “yes/no” question used to assess learners’ perceptions of bias. Given that commercial influence is often subtle, a single binary question may fail to fully capture the range of learner perceptions of commercial influence.25 Moreover, learner perceptions of bias may not precisely correspond to actual bias in CME activities. It is possible that some suspicious observers may have perceived bias where in fact there was none. However, a wealth of literature in the medical and social sciences suggests that physicians (and people in general) often fail to perceive bias, particularly when these biases are subtle.26–28
Prior work investigating mechanisms of bias in CME has highlighted the importance of subtle, often subconscious and unintentional forms of bias that may be difficult for screening processes to identify and for learners to perceive.1,15,29 Course organizers may focus course curricula more on drug or device-based therapies and less on lifestyle-based treatments, or devote special attention to one aspect of disease management for which a supporter’s drug (or class of drugs) may be commonly used.30 In addition, the medical literature from which evidence-based recommendations are made is subject to commercial influence, as noted in studies which have shown strong and consistent associations between industry support of clinical trials and the outcomes and interpretation of those trials.31,32 Speakers may also play an important role in introducing subtle biases, for example through their choice of topics and presentation of course material.
In each of these cases, bias is introduced not through a direct quid-pro-quo but through subconscious attitudes and feelings of reciprocity that can arise when a speaker or course director has benefitted from industry, for example through funding of an educational or research grant, service on a speakers’ bureau, or receipt of personal educational materials, food, or samples for one’s clinic. Because attitudes and feelings of reciprocity largely occur on a subconscious level, physicians often fail to recognize the ways in which they might be influenced and overestimate their ability to resist commercial influence.23,26 Of note, in 2007 only 20% of speakers at UCSF CME activities reported conflicts of interest such as grant funding or speakers bureau honoraria relevant to their talks; however, this number may be substantially higher elsewhere.
To the extent that influence on the content of CME can often be subtle and occur without the conscious knowledge of course organizers and speakers, bias in CME can be difficult to detect even for the watchful observer.23,33 In addition, in many cases it can be challenging, if not impossible, to determine whether a speaker is emphasizing a certain topic or recommending a certain treatment strategy because of subtle commercial influence on that speaker or because that speaker has a truly independent, scientifically-valid opinion that reaches the same conclusion.22 Paradoxically, CME activities where commercial influence is subtle are likely to be more effective than activities with overt influence, since physicians are more receptive to information that appears objective and may reject information they perceive to be promotional.30
Given the difficulty of disentangling unvarnished educational goals and scientific opinion from subtle forms of commercial bias, there have been increasing calls to fundamentally change the relationship between the healthcare industry and CME providers. In a seminal article, Brennan et al. recommended eliminating industry support of individual CME activities at academic medical centers, and instead suggested a mechanism in which industry donations would be pooled into a central account that would be used to help fund individual CME activities without industry input.29 Several universities have adopted this model, with a handful forsaking all industry funding of their CME programs.34 A recent conference commissioned by the Josiah Macy Foundation went a step farther, recommending the complete elimination of industry support of accredited CME.16 Not surprisingly, these recommendations are controversial, in part because many CME providers are financially dependent on industry, and because physicians are concerned about increases in course registration fees if industry support was substantially reduced or withdrawn.4,13,21,35
Our results could be interpreted as either supporting or contradicting these recommendations. On the one hand, one might conclude that the safeguards offered by the UCSF Office of CME’s screening practices resulted in a series of CME programs with little if any bias. On the other hand, our results are consistent with the observation that commercial bias is difficult to detect, both by organizers and recipients of CME, and that the only way to safely guard against industry influence is to eliminate or limit industry involvement in CME. To resolve this and other questions, more research is needed to systematically investigate the presence and impact of bias in CME, including in-depth explorations of course content and learner perceptions of bias. Of note, definitive research will likely require large sample sizes and sophisticated measures, making it unlikely that a substantial body of new research will be available in the near future. Until then, policy-makers will need to carefully interpret existing data and apply lessons learned from research on other types of interactions between physicians and industry to help inform their judgments.
Our study has several limitations. As noted earlier, our sample of CME activities were organized by a single institution with a distinct set of criteria for vetting proposed activities. It is thus unclear to what extent our findings would be generalizable to other academic and non-academic providers of CME with less stringent review criteria and a different set of administrative procedures and institutional culture. In addition, as noted above the question that we used to determine perceptions of bias was limited to a binary yes/no response, preventing us from capturing subtlety in participant responses. Third, our study evaluated only activities which had passed an initial screening process in which proposals deemed to have an unacceptably high risk of commercial influence were rejected (whereas such activities might be approved by other accredited providers of CME). As a result, our study design prevents us from assessing perceived bias in activities believed to carry the highest risk of industry influence. Finally, our study evaluated a moderate but limited number of CME activities. Nonetheless, it is unlikely that a larger sample of activities would have changed our conclusions, as even if the observed differences in perceived bias between lower vs. higher-risk activities were statistically significant they would not be meaningfully different.
In summary, in this study of 213 CME activities rates of perceived commercial bias were very low, with no differences in perceived bias between activities with and without potential risk factors for commercial influence. These findings suggest that rigorous review criteria effectively screened out activities with explicit commercial bias. Further research will be needed to evaluate the presence of subtle forms of commercial bias and risk factors to predict these biases in CME.
Items derived from ACCME Essential elements - Planning 2.1
Items derived from ACCME Essential elements - Needs Assessment 2.2
Items derived from ACCME Essential elements - Disclosure Standards 3.3
Items derived from ACCME Essential elements - Commercial Support Standards 3.3
Scoring of risk index:
Each item receives one point. The points are summed and scored as follows:
In addition, events co-sponsored with a non-accredited organization, society, or institution are automatically considered “highest risk”
The authors thank John Boscardin, PhD and Kathy Fung, MS for their help with the statistical analyses
Funding for this work was provided by VA Health Services Research and Development Service (CDTA 01-013) and by the National Institutes of Health (K23 AG030999) (both for Dr. Steinman)
Dr. Steinman served as an unpaid expert witness for the plaintiff in United States ex. rel Franklin vs. Pfizer, Inc, litigation which alleged that Pfizer and Parke-David illegally marketed gabapentin (Neurontin) for uses not approved by the FDA, including the use of medical education for marketing purposes. Dr. Steinman also helped to found an online archive of drug industry marketing documents from this and other litigation, including soliciting a gift of start-up funds from the lead lawyer in the gabapentin litigation. In addition, Dr. Steinman contributed unpaid effort to an educational grant funded with monies from an Attorney General Settlement Fund created through an out-of-court settlement of this litigation.
Publisher's Disclaimer: Disclaimers
Dr. Steinman had full access to the entire data for this study and takes full responsibility for the results.
All authors meet criteria for authorship.
Michael A. Steinman, Staff Physician at the San Francisco VA Medical Center and an Assistant Professor of Medicine in the Division of Geriatrics at the University of California, San Francisco.
Christy K. Boscardin, Assistant Adjunct Professor in the Division of General Internal Medicine and the Office of Medical Education at the University of California, San Francisco.
Leslie Aguayo, Administrative Director of the Office of Medical Education at the University of California, San Francisco.
Robert B. Baron, Professor of Medicine and Associate Dean for Graduate and Continuing Medical Education at the University of California, San Francisco.