Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Am Coll Cardiol. Author manuscript; available in PMC 2012 April 5.
Published in final edited form as:
PMCID: PMC3085030

Concordance of Physician Ratings with the AUC for Coronary Revascularization

Paul S. Chan, MD, MSc,* Ralph G. Brindis, MD, MPH, David J. Cohen, MD, MSc,* Philip G. Jones, MSc,* Elizabeth Gialde, RN,* Richard G. Bach, MD,§ Jeptha Curtis, MD,|| Charles F. Bethea, MD,# Marc E. Shelton, MD,** and John A. Spertus, MD, MPH*



Appropriate Use Criteria (AUC) for coronary revascularization have been developed by selected experts.


To compare the consistency in AUC ratings among a broader range of practicing cardiologists and the AUC Technical Panel.


Prior to AUC publication, 85 cardiologists from 10 U.S. institutions assessed the appropriateness of coronary revascularization for 68 indications that had been evaluated by the Technical Panel. Each indication was classified as appropriate, uncertain, or inappropriate, based on the physician group’s median rating. Rates of concordance between the physician group and the AUC Technical Panel (i.e., same appropriateness category assignment) and rates of non-agreement within the physician group (≥25% of panelists’ ratings outside the group’s appropriateness category assessment) were determined.


Overall concordance between the 2 groups was 84%. Among indications classified as appropriate by the Technical Panel, concordance between the 2 groups was excellent (94% [34/36]); however, non-agreement within the physician group was 44% (16/36). Among indications classified as uncertain, there was 73% (16/22) concordance between the 2 groups. Among inappropriate indications, concordance was moderate (70% [7/10]), but non-agreement occurred frequently (70% [7/10]). Moreover, there was substantial variation in appropriateness ratings between individual physicians and the Technical Panel (weighted kappa range: 0.05–0.76).


While there was good concordance in assessments of appropriateness for coronary revascularization between physicians and the AUC Technical Panel, non-agreement within the physician group was common and there was marked variation in ratings between individual physicians and the Technical Panel.

While few debate the potential of coronary artery bypass grafting (CABG) or percutaneous coronary intervention (PCI) to confer substantial mortality and quality of life benefits in selected patients, concerns about potential overuse and underuse of these procedures have emerged. As a result, national societies have developed the Appropriate Use Criteria (AUC) for coronary revascularization to support the efficient and rational use of these procedures.

While the Technical Panel that created the AUC included a diverse range of perspectives, and based their recommendations upon clinical guidelines, it is unknown whether appropriateness assessments might differ among a broader range of cardiologists, including those in community-based settings. Understanding the extent of concordance in appropriateness ratings between the expert panel and a more diverse group of practitioners can validate scenarios for which there is broad consensus, or highlight the need for either more research or better translation of existing knowledge to clinical practice. It can also indicate the likelihood of widespread acceptance and adoption of the AUC. Accordingly, we surveyed cardiologists with a representative sample of the same clinical indications as those rated by the Technical Panel and examined both their concordance and the variation in their distribution of appropriateness ratings with the Technical Panel.


Development of the AUC

The methodology to develop the AUC for coronary revascularization has been previously described.1 Briefly, the Technical Panel included 17 members (4 interventional cardiologists; 8 noninterventional cardiologists; 4 cardiac surgeons; and 1 Payer) who rated the appropriateness of coronary revascularization for 174 distinct clinical indications based upon different combinations of: 1) clinical presentation (acute vs. non-acute), 2) angina severity (asymptomatic or Canadian Cardiovascular Society [CCS] class), 3) extent of ischemia on noninvasive functional testing (low, intermediate, or high risk), 4) high-risk clinical features (e.g., left ventricular dysfunction), 5) extent of anti-ischemic medical therapy, and 6) extent of anatomic disease based on angiography (number of diseased coronary arteries with or without proximal left anterior descending [LAD], left main, or bypass graft disease). Moreover, the Technical Panel evaluated the relative appropriateness of PCI and CABG for 24 additional scenarios.

Based upon the median score of the Technical Panel for each clinical indication (range: 1–9), clinical indications were categorized as ‘Appropriate’ (median: 7–9), ‘Uncertain’ (4–6), or ‘Inappropriate’ (1–3). In addition, rates of non-agreement were determined to assess variations in panelists’ ratings of appropriateness for a given clinical scenario. Non-agreement occurred when ≥25% of the individual panelists’ ratings fell outside the 3-point region for that indication.2 To accomplish this, the ACCF made available to us the final ratings of each Technical Panel member.

Description of Study Survey

In 2008, prior to the release of the AUC, permission was obtained from the ACCF to replicate the appropriateness ratings process in a cohort of clinicians. Two authors (PSC and JAS) reviewed and selected 68 representative indications evaluated by the Technical Panel for this study’s survey (Appendix A). For the survey, one-third of the AUC’s indications were chosen to minimize respondent burden, but the selected indications reflected a diverse range of clinical scenarios.

Select cardiologists from 10 institutions (Appendix B) were electronically mailed the study survey. Because the AUC were yet unpublished, none of the survey participants had access to the Technical Panel’s ratings. Study participants were provided the same explanatory documents, tables and figures, references, and appendices that had been given to the Technical Panel.

Statistical Analysis

For each of the 68 clinical indications, the median and distribution of appropriateness ratings from study participants were determined. From the median value, the physician group’s assignment of appropriateness category was determined for each indication using the method described above.

Concordance of appropriateness category assignments (e.g., appropriate, uncertain, inappropriate) for each clinical indication was compared between the physician group and the Technical Panel with descriptive plots. Summaries of concordance results were tabulated overall and also stratified by whether the Technical Panel had categorized an indication as appropriate, uncertain, or inappropriate. Analyses were then repeated after dividing the physician group by interventional status, years in practice (<15 years vs. ≥15 years) and percent time dedicated to research (<10% vs. ≥10%).

To examine variation in appropriateness assignments, the presence of non-agreement for each indication within the physician group was assessed. In addition, to examine the extent of variation between individual physicians and the Technical Panel, we computed weighted kappa statistics between each individual physician and the Technical Panel for all 68 clinical indications, and examined the distribution of weighted kappas within the physician group.

Finally, we examined which clinical factors (coronary anatomy, extent of ischemia on noninvasive functional testing, severity of symptoms, and intensity of anti-ischemic therapy) predicted non-agreement within the physician group using multivariable hierarchical regression, with indications clustered by physician and physicians clustered by institution. All analyses were conducted with SAS 9.2 (SAS Institute, Inc, Cary, NC) and R version 2.6.2.


A total of 85 physicians from 10 (2 community, 7 university-affiliated, and 1 university-owned) institutions completed the study survey. Among the physician respondents, 44 (51.8%) were interventional and 41 (48.2%) were non-interventional cardiologists. All but 2 (97.6%) were board-certified in cardiology, the median number of years in practice was 14.5 (interquartile range: 7.5–20.0; range: 1.0–43.0), and the median proportion of time spent in research was 5% (interquartile range: 0%-15%; range: 0%-75%). Overall, there was good concordance (84%) in appropriateness category assignments between the physician group and the Technical Panel for the 68 indications. However, rates of non-agreement were 66% in the physician group.

Clinically Appropriate Indications

For the 36 clinical indications identified as appropriate by the Technical Panel, there was excellent concordance, with the physician group rating the indications as appropriate 94% (34/36) of the time (Figure 1, Table 1). Both interventional and non-interventional cardiologists rated the vast majority of these indications as appropriate (Appendix C), and physician ratings did not differ by number of years in practice or percent time dedicated to research (Table 1). There was, however, greater variation (i.e., wider distribution) in the ratings among physicians in this study than in the Technical Panel. Whereas the Technical Panel had non-agreement in 31% (11/36) of the clinical indications categorized as appropriate, the physician group had non-agreement in 44% (16/36) of the indications (Table 2).

Figure 1
Appropriateness Ratings By the Physician Group for 36 Appropriate Indications
Table 1
Agreement Between the Technical Panel and Survey Participants in Ratings of Appropriateness for Coronary Revascularization
Table 2
Rates of Non-agreement in Ratings of Appropriateness Within Each Group

Clinically Uncertain Indications

For the 22 clinical indications rated as uncertain by the Technical Panel, both groups rated the indications as uncertain 73% (16/22) of the time (Table 1). Of the 6 discordant indications, the physician group rated 3 scenarios (12c, 14c, and 57b) as appropriate and 3 (25a, 28a, and 50a) as inappropriate (Figure 2). While there was similar concordance between interventionalists and non-interventionalists with the Technical Panel, physicians with ≥10% time dedicated to research had higher concordance with the Technical Panel than physicians with <10% research time (see Table 1). Finally, rates of non-agreement for scenarios categorized as uncertain were 100% in the physician group (22/22) and 82% (18/22) in the Technical Panel.

Figure 2
Appropriateness Ratings By the Physician Group for 22 Uncertain Indications

Clinically Inappropriate Indications

For the 10 clinical indications identified as inappropriate by the Technical Panel, the physician group assigned an inappropriate classification for only 70% (7/10) of the indications categorized as inappropriate by the Technical Panel (Figure 3, Table 1), with 100% concordance among non-interventional cardiologists and 70% among interventional cardiologists (Appendix D). Importantly, there was significant variation among physician ratings for these clinical scenarios, with non-agreement rates of 70% in the physician group compared with 20% in the Technical Panel (Table 2). Interventionalists and physicians with ≥10% research time had lower rates of non-agreement than their counterparts.

Figure 3
Appropriateness Ratings By the Physician Group for 10 Inappropriate Indications

Predictors of Non-Agreement Among Physicians

In a multivariable model, the presence of a proximal LAD stenosis was associated with higher rates of physician non-agreement (adjusted RR of 1.29 [95% CI: 1.11–1.51]; P=0.001), suggesting greater variability in ratings of appropriateness within the physician group when the proximal LAD was involved (Table 3). In contrast, a high-risk noninvasive study for ischemia (adjusted RR of 0.51 [95% CI: 0.40–0.65]; p<.0.001), maximal intensity (i.e., 2 or more agents) of anti-ischemic therapy (adjusted RR of 0.75 [95% CI, 0.69–0.82] P<.001), and substantial (CCS class III-IV) symptoms (adjusted RR of 0.46 [95% CI, 0.38–0.57]; P<.001) were each associated with significantly lower rates of physician non-agreement.

Table 3
Clinical Predictors of Non-Agreement Within the Physician Group.

Variation Between Individual Physicians and Technical Panel

The distribution of weighted kappa statistics between individual physicians and the Technical Panel for all 68 indications was wide, ranging from 0.05 to 0.76, with the average weighted kappa for all physicians of 0.52 (Figure 4). This suggests marked variation in appropriateness assignments between individual physicians and the Technical Panel, with certain physicians almost never agreeing with the Technical Panel for any of the surveyed indications and no physician achieving uniform concordance with the Technical Panel.

Figure 4
Agreement in Appropriateness Category Assignments Between Individual Physicians and the Technical Panel


In this study, we compared ratings of clinical appropriateness for coronary revascularization between practicing clinicians and members of the AUC Technical Panel. We found there was excellent concordance (94%) between the two groups for clinical indications categorized as appropriate, but only modest concordance (70%) for clinical indications categorized as inappropriate. However, there was wide variation (i.e., non-agreement) in ratings of appropriateness among physicians, with more than 25% of physicians assigning an appropriateness category different from the group as a whole in 2 out of every 3 scenarios. Moreover, there was substantial variation in appropriateness category assignments between individual physicians and the Technical Panel, with some physicians almost never agreeing with the Technical Panel and no physician achieving >80% agreement. Collectively, our findings suggest that, while there is general concordance in ratings of clinical appropriateness between practicing cardiologists and the Technical Panel, there is not uniform agreement between the 2 groups, with markedly different opinions among individual physicians, even after reviewing existing evidence.

An important strength of this study is that the physician ratings were obtained prior to the publication of the ACC’s AUC for Coronary Revascularization in early 2009. This avoided potential contamination of survey results by respondents from the views of the Technical Panel. Importantly, the physician group received similar instructions and access to clinical guidelines for coronary revascularization as the Technical Panel to derive their appropriateness ratings.

Our findings suggest that there was substantial within-group heterogeneity in ratings of appropriateness among the physicians surveyed. More than 1 in 4 physicians rated an indication for coronary revascularization as uncertain or inappropriate for 43% of indications categorized as appropriate by the AUC and rated an indication as uncertain or appropriate for 70% of the inappropriate indications. Notably, physicians’ rates of non-agreement were lower for those indications involving severe CCS Class III-IV angina symptoms and high-risk noninvasive studies for ischemia, where the benefits of coronary revascularization are likely to be greater. Conversely, there was more divergence of opinion in the setting of significant proximal LAD obstruction, suggesting that some clinicians presumably viewed its treatment as life prolonging, while others did not.

The broad range of kappa statistics between individual physicians and the Technical Panel, none of which exceeded 0.76, highlights the variability in current opinions about the roles of coronary revascularization in the care of patients with coronary artery disease. More research and better translation of existing knowledge to clinical practice to clarify the appropriateness of certain clinical indications, especially for those indications where appropriateness was uncertain or where non-agreement existed, is needed. Given this variability in the appropriateness ratings, it is expected that clinicians will vary greatly in their AUC ratings for coronary revascularization. We believe that it will be important to both measure and provide feedback to clinicians about the appropriateness of patients that they treat, while concurrently educating them about the AUC, if more uniform practice is to be achieved.

Our study should be interpreted in the context of the following limitations. Our survey methodology differed somewhat from that of the Technical Panel, as there were not face-to-face meetings to discuss cases for which divergent assessments were made. Second, our survey was conducted among clinical cardiologists from only 10 U.S. institutions, and we did not assess ratings of appropriateness among cardiac surgeons. Third, while our panel was larger and more geographically representative of clinical practices than the Technical Panel, it did not capture all geographical regions of the U.S. Fourth, we did not have sufficient sample size to examine other subgroups of interest nor obtain more detailed data on characteristics of survey respondents.

In conclusion, we found good concordance for ratings of appropriateness for coronary revascularization between a diverse group of cardiologists and the Technical Panel for the AUC for Coronary Revascularization. However, there was substantial variation in ratings of appropriateness between individual physicians and the Technical Panel, as well as non-agreement in appropriateness category assignments among cardiologists for a number of indications. These findings suggest that more research to understand these variations, along with additional education about procedural appropriateness, may help achieve greater uniformity in the appropriate use of coronary revascularization.

Supplementary Material



Funding/Sponsors: none

  • We wish to thank the American College of Cardiology Foundation for making available to us the de-identified ratings of the Technical Panel for the AUC for Coronary Revascularization, without which comparisons of rates of agreement between the Technical Panel and the physician group would not have been possible.
  • Dr. Chan is supported by a Career Development Grant Award (K23HL102224) from the NHLBI.
  • Dr. Spertus has a contract with the ACCF to analyze the NCDR and to develop AUC ratings in the Cath/PCI registry.

List of Abbreviations

Coronary Artery Bypass Grafting
Percutaneous Coronary Intervention
American College of Cardiology Foundation
Canadian Cardiovascular Society
Left Anterior Descending
Coronary Artery Disease
Inter-Quartile Range


No other authors had any disclosures or potential conflicts of interest.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.


1. Patel MR, Dehmer GJ, Hirshfeld JW, Smith PK, Spertus JA. ACCF/SCAI/STS/AATS/AHA/ASNC 2009 Appropriateness Criteria for Coronary Revascularization. J Am Coll Cardiol. 2009;53(6):530–53. [PubMed]
2. Fitch K, Bernstein S, Aguilar M, et al. The RAND/UCLA Appropriateness Method User’s Manual. RAND; Arlington, VA: 2001.