Operator Experience and Outcomes with Carotid Stenting in Medicare Beneficiaries

doi:10.1001/jama.2011.1357

JAMA. Author manuscript; available in PMC 2012 September 28.

Published in final edited form as:

JAMA. 2011 September 28; 306(12): 1338–1343.

doi: 10.1001/jama.2011.1357

PMCID: PMC3208144

NIHMSID: NIHMS322733

Operator Experience and Outcomes with Carotid Stenting in Medicare Beneficiaries

Brahmajee K. Nallamothu, MD, MPH, Hitinder S. Gurm, MD, Henry H. Ting, MD, Philip P. Goodney, MD, Mary A. M. Rogers, PhD, Jeptha P. Curtis, MD, Justin B. Dimick, MD, Eric R. Bates, MD, Harlan M. Krumholz, MD, SM, and John D. Birkmeyer, MD

Center for Healthcare Outcomes and Policy (CHOP) (B.K.N., H.S.G., J.B.D., J.D.B.), Department of Internal Medicine (B.K.N., H.S.G., M.A.R., E.R.B.), and Department of Surgery (J.B.D., J.D.B.), University of Michigan Medical School, Ann Arbor, MI; Veterans Affairs Ann Arbor Health Services Research and Development Center of Excellence (B.K.N.); Division of Cardiovascular Diseases, Mayo Clinic, Rochester, MN (H.H.T.); Section of Vascular Surgery, Dartmouth-Hitchcock Medical Center, Lebanon, NH (P.P.G.); Department of Internal Medicine, Yale University School of Medicine, New Haven, CT (J.P.C., H.M.K.).

Corresponding Author: Brahmajee K. Nallamothu, MD, MPH University of Michigan Cardiovascular Center, SPC 5869 Ann Arbor, Michigan 48109-5869 ; Email: bnallamo/at/umich.edu P: 734.615.3878, F: 734.214.0691

The publisher's final edited version of this article is available at JAMA

Abstract

Context

Although the efficacy of carotid stenting has been established in clinical trials, operator experience and outcomes with the procedure in routine clinical practice are less certain.

Objectives

To correlate outcomes with 2 measures of operator experience: (1) annual volume; and (2) experience at the time of the procedure among new operators who first performed carotid stenting after a national coverage decision by the Centers for Medicare & Medicaid Services (CMS).

Design, Setting, & Patients

Observational study using administrative data on fee-for-service Medicare beneficiaries 65 years or older undergoing carotid stenting between 2005 and 2007.

Main Outcome Measure

30-day mortality stratified by very-low, low, medium, and high annual operator volumes (<6, 6 to 11, 12 to 23, and ≥24 procedures per year) and treatment early versus late during a new operator’s experience (1^st to 11^th procedure and 12^th procedure or higher).

Results

24,701 procedures were performed by 2339 operators. Of these, 11,846 were performed by 1792 new operators who first performed carotid stenting after its national coverage decision. Overall, 30-day mortality was 1.9% and rate of failure to use an embolic protection device was 4.8%. The median annual operator volume in Medicare beneficiaries was 3.0 per year (IQR, 1.4 to 6.5) and 11.6% of operators performed ≥12 procedures per year during the study period. Observed 30-day mortality was higher among patients treated by operators with lower annual volumes (2.5% [95% CI, 2.1%-2.9%], 1.9% [95% CI, 1.6%-2.3%], 1.6% [95% CI, 1.3%-1.9%], and 1.4% [95% CI, 1.1%-1.7%] across the 4 categories; P<0.001) and among patients treated early versus late during a new operator’s experience (2.3% [95% CI, 2.0%-2.7%] and 1.4% [95% CI, 1.1%-1.9%]; P=0.001). After multivariable adjustment, patients treated by very-low volume operators had a higher risk of 30-day mortality when compared with patients treated by high volume operators (adjusted OR, 1.9; 95% CI, 1.4 to 2.7; P<0.001). Similarly, we found a higher risk of 30-day mortality in patients treated early versus late during a new operator’s experience (adjusted OR, 1.7; 95% CI, 1.2 to 2.4; P=0.001).

Conclusion

Among older patients undergoing carotid stenting, lower annual operator volume and early experience were associated with increased 30-day mortality.

Carotid stenting is increasingly being utilized to treat severe carotid atherosclerosis, an important cause of ischemic stroke.¹ Since approval of the first carotid stent system by the Food and Drug Administration (FDA) in 2004, use of carotid stenting has more than doubled in Medicare beneficiaries.² Explanations for the rapid dissemination of carotid stenting into routine clinical practice include its minimally-invasive nature, lack of need for general anesthesia, and an expanded pool of physicians capable of performing it.³ Yet despite its overall promise, the rising use of carotid stenting also raises potential concern. Carotid stenting is a technically demanding procedure and earlier studies have demonstrated a substantial “learning curve” with it.^4,5 Although this has prompted consideration of establishing minimum volume requirements and training criteria for potential operators,^6-8 specific recommendations have varied across professional organizations and none have been widely enforced.⁹ As such, the total number of operators currently performing carotid stenting in routine clinical practice and their overall experiences and outcomes with the procedure remain largely unknown.

In this context, we used national Medicare data to describe recent patterns of utilization and outcomes for carotid stenting in the United States among the elderly – a high-risk group that makes up approximately three-quarters of the patients undergoing this procedure.¹⁰ As with other complex medical and surgical procedures, we hypothesized that low volume operators performing carotid stenting would have worse risk-adjusted outcomes when compared with high volume operators. In addition, we examined whether a learning curve existed among new operators with evidence of first performing carotid stenting after a initial national coverage decision by the Centers for Medicare & Medicaid Services (CMS) in March 2005, hypothesizing that worse outcomes would occur in patients treated earlier during these operators’ experience.

Methods

Data Sources and Study Cohort

From the Centers for Medicare & Medicaid Services (CMS), we obtained data from the Physician Carrier (Part B), Medicare Provider Analysis and Review (MEDPAR) and Denominator files for all patients undergoing carotid stenting from January 1, 2004 to December 31, 2007. Physician Carrier files include data on claims for procedures from non-institutional providers, such as physicians, while MEDPAR files include data on acute-care hospitalizations and Denominator files contain information on eligibility and enrollment. Procedures of carotid stenting were identified from the Physician Carrier files using Healthcare Common Procedure Coding Systems (HCPCS) codes 37215 and 37216, which were assigned starting in 2004. We only included Medicare beneficiaries aged 65 years or older continuously enrolled in fee-for-service programs at least 1 year prior to their procedure. We excluded any patients treated by operators with evidence of first performing carotid stenting in the last 2 quarters of the study period given limited follow-up.

For our analyses examining experience at the time of the procedure, we further restricted our study cohort to patients who underwent carotid stenting by new operators who first performed carotid stenting after March 17, 2005 – the date of its initial national coverage decision by the CMS.¹¹ Any operator performing carotid stenting prior to this date was considered “experienced”, since the CMS previously had covered carotid stenting in Medicare beneficiaries only when it was performed within an FDA-approved study protocol. As such, this operator may have been performing the procedure for several years. In addition, we searched for and excluded any additional operators listed as investigators in the Global Carotid Artery Stent Registry, SAPPHIRE trial, and CREST trial^12-14 – 3 large, multi-center studies of carotid stenting in the United States that had been active prior to the national coverage decision by the CMS.

The Institutional Review Board of the University of Michigan approved this protocol before its initiation. The requirement for informed consent was waived.

Outcomes and Study Variables

The primary outcome of interest for our analysis was 30-day mortality following the procedure, which was determined from the Denominator files. The secondary outcome of interest was failure to receive an embolic protection device during the procedure. Failure to receive an embolic protection device was obtained using the HCPCS codes, which identified patients as either having undergone carotid stenting with or without an embolic protection device. We evaluated the use of embolic protection devices because their use has been linked to better outcomes¹⁵ and is currently required by the CMS for reimbursement.¹⁶

For our analysis, we had 2 independent variables of interest that each reflected separate, but related aspects of operator experience: (1) annual volume and (2) experience at the time of the procedure. Annual operator volume was assessed by first identifying the total number of procedures performed in Medicare beneficiaries during the study period for each operator. We annualized this number by determining the length of time in days from when that operator’s first procedure was performed to the end of the study period, standardizing that relationship to a 12-month time period.

To estimate operator experience at the time of the procedure, we restricted our study cohort to patients treated by new operators who first performed carotid stenting after the national coverage decision by the CMS (see details above). For each patient, we used the case number of carotid stenting that individual was after ranking all procedures by their operator sequentially over the study period (i.e., the operator’s 1^st, 2^nd, 3^rd…or n^th procedure). In instances where more than 1 operator was involved (n=223), we assigned the experience of the operator who had the higher number of procedures performed up until that time. When more than 1 procedure was performed on the same day by the same operator (n=5373), all patients on that day were assigned a mean rank of experience for that day. Finally, we only included the index procedure in analyzing outcomes, although repeat procedures (n=1757) were included in calculating experience.

Additional information on age, gender, and race were obtained using the Denominator files. From the Physician Carrier and MEDPAR files, we identified Elixhauser co-morbidity conditions using ICD-9-CM diagnostic codes from all claims submitted during the 12 months preceding the procedure. We then collapsed these conditions into a co-morbidity score for each patient using a validated point system.¹⁷ We also used these data sources to determine if a patient had been hospitalized or had an outpatient visit where the diagnosis of acute stroke or TIA was recorded in the 180 days prior to carotid stenting. We specifically chose a 180 day time period given that this was the length of time used by the CREST investigators to define symptomatic patients.¹⁴ Similarly, we identified if a patient had undergone carotid endarterectomy in the 1 year prior to carotid stenting. Finally, we determined the specialty of the operator performing carotid stenting using Medicare specialty codes and the 2007 Unique Physician Identification Number (UPIN) Directory.

Statistical Analysis

Descriptive statistics evaluated for differences in baseline characteristics after stratifying patients into 4 categories of annual operator volume: <6, 6 to 11, 12 to 23, and ≥24 per year. We then used multivariable logistic regression models to assess the relationship between the volume categories and 30-day mortality. Models adjusted for: age (65 to 69 years, 70 to 74 years, 75 to 79 years, 80 to 84 years, 85 years or older), gender, race (black, non-black), Elixhauser co-morbidity score, the presence of acute stroke or TIA in the 180 days prior to carotid stenting, the presence of a carotid endarterectomy in the 1 year prior to carotid stenting, and the date of their operator’s first procedure during the study period. To account for the inherent “clustering” within these data – i.e., the same operator performs multiple procedures over time – models were constructed using the framework of multilevel modeling with random intercepts included for individual operators.¹⁸

Similar models were constructed after restricting our study cohort to those patients who underwent carotid stenting by new operators who first performed carotid stenting after the national coverage decision by the CMS. We compared 30-day mortality between patients treated early versus late during their operator’s experience after stratifying them into 2 categories: 1^st to 11^th procedure and 12^th procedure or higher. To understand the extent to which our findings during were being driven by operators who performed a limited number of cases during the study period, we repeated these analyses after further restricting our study cohort to patients treated by new operators who had performed at least 12 procedures during the study period. Finally, additional models were constructed to assess the relationship between both measures of operator experience and failure to receive an embolic protection device.

For each outcome we assessed, we report risk-adjusted odds ratios [OR] and their corresponding 95% confidence intervals. Overall, our final risk-adjusted models for 30-day mortality had reasonable discrimination with C-statistics of 0.75 for those that evaluated annual operator volume and 0.80 for those that assessed operator experience at the time of the procedure. All analyses were performed using SAS version 9.2 (SAS Institute, NC) and Stata version 11.0 (StataCorp, TX). P values of <0.05 were considered significant and all tests were 2-sided.

Results

Study Cohort

We identified 24,701 patients who underwent carotid stenting by 2339 operators. Of these, 11,846 were performed by 1792 operators who first performed carotid stenting after its national coverage decision. Mean age of the study cohort was 76.2 years (std dev, 6.4), 9917 (40.2%) were women, and 1056 (4.3%) were black. Overall, 461 (1.9%) patients died within 30 days of their procedure and 1173 (4.8%) patients failed to receive an embolic protection device. The median annual operator volume in Medicare beneficiaries during the study period was 3.0 per year (interquartile range, 1.4 to 6.5). A total of 639 (27.3%) operators performed ≥6 procedures per year and 272 (11.6%) operators performed ≥12 procedures per year (Figure 1). Table 1 displays characteristics of patients across different categories of annual operator volume. When compared with patients treated by high volume operators, patients treated by very-low volume operators were younger, but more often black , had a higher number of Elixhauser co-morbidity conditions, and were more likely to have had an acute stroke in the 180 days prior to their carotid stenting and carotid endarterectomy in the 1 year prior to their carotid stenting.

Figure 1

Histogram of annual operator volume with carotid stenting across 2339 operators over the study period.

Table 1

Baseline characteristics of patients stratified by operative experience at time of carotid stenting.

Annual Operator Volume and Outcomes

We found higher 30-day mortality in patients treated by operators with lower annual volumes of carotid stenting (Figure 2a). Observed 30-day mortality was 2.5% [95% CI, 2.1%-2.9%], 1.9% [95% CI, 1.6%-2.3%], 1.6% [95% CI, 1.3%-1.9%], and 1.4% [95% CI, 1.1%-1.7%] across the 4 categories (P<0.001). This difference remained statistically significant after multivariable adjustment (Table 2). For example, when compared with patients treated by operators performing ≥24 procedures per year, those treated by operators performing <6 procedures per year had an adjusted OR of 1.9 for 30-day mortality (adjusted OR, 1.9; 95% CI, 1.4-2.7; P<0.001). A similar relationship was noted when we examined rates of failure to receive an embolic protection device. For example, the adjusted OR was 8.1 for failing to receive an embolic protection device between patients treated by operators performing <6 procedures per year and those treated by operators performing ≥24 procedures per year (adjusted OR, 8.1; 95% CI, 4.4-14.9; P<0.001) (Table 2).

Figure 2

Unadjusted outcomes and their 95% confidence intervals in patients by annual operator volume and operator experience at the time of the procedure. P-values for differences across categories are <0.001 for each of the outcomes. (Note: EPD=embolic (more ...)

Table 2

Unadjusted and adjusted odds ratios of outcomes across categories of operator experience

Operator Experience at the Time of a Procedure and Outcomes

We also found higher 30-day mortality in patients treated early versus late during a new operator’s experience (Figure 2b) (2.3% [95% CI, 2.0%-2.7%] and 1.4% [95% CI, 1.1%-1.9%]; P<0.001). This difference also remained statistically significant after multivariable adjustment. For example, when compared with patients who were their operator’s 12^th procedure or higher, those who were among their operator’s first 11 procedures had an adjusted OR of 1.7 for 30-day mortality (adjusted OR, 1.7; 95% CI, 1.2-2.4; P=0.001) (Table 2). These findings remained largely unchanged during sensitivity analyses that restricted the study cohort to those patients treated by new operators who performed at least 12 or more procedures during the study period (adjusted OR, 1.6; 95% CI, 1.1-2.3; P=0.020). Failure to receive an embolic protection device also was more common early during a new operator’s experience. For example, the adjusted OR for failing to receive an embolic protection device was 4.8 between patients who were among their operator’s first 11 procedures when compared with those who were their operator’s 12^th procedure or higher (adjusted OR, 4.8; 95% CI, 3.4-6.8; p<0.001) (Table 2).

Comment

We found a 30-day mortality of nearly 2% among Medicare beneficiaries undergoing carotid stenting. Mortality rates for elderly patients in contemporary clinical trials and registries are closer to 1%.^13,14,19 Although the higher mortality rates we identified are likely being driven to a large extent by an older and less selected population of patients, we identified an additional factor that may be contributing: limited operator experience with carotid stenting as the procedure has disseminated into routine clinical practice. Indeed, we found that fewer than 1 in 8 operators had annual operator volumes of 12 procedures or more during the study period. Furthermore, we noted that patients treated by very-low-volume operators and those treated early during a new operator’s experience had significantly higher 30-day risk-adjusted mortality.

Ensuring that physicians are adequately experienced to perform innovative and technically-complex procedures, like carotid stenting, is not a new challenge.²⁰ It has been seen with several health technologies in the past, most notably following the introduction of laparascopic procedures in the late 1980s.^21,22 Yet, ensuring expertise with carotid stenting also raises concerns that are particularly unique to this procedure. For example, carotid stenting is performed by specialists from diverse fields including cardiology, radiology and surgery. These physicians have widely varying clinical backgrounds and technical skills that make it hard to standardize educational programs. An expanded pool of physicians capable of performing it also may make carotid stenting more difficult to concentrate expertise among a few operators, especially given strong interest in the procedure by all 3 specialties. Of course, making policy decisions about restricting use of carotid stenting to highly-experienced operators is complicated and involves balancing safety concerns with the potential long-term harm of limiting access to an innovative procedure early on during its dissemination.

To a certain extent, the challenge of ensuring adequate expertise among operators has been widely recognized by professional organizations and regulators. For example, professional organizations representing the major specialties involved in carotid stenting have listed specific criteria to guide facilities in credentialing individual operators.^6-8 These include minimum volume requirements suggested for operators, although specific thresholds vary across groups. The FDA has also encouraged educational initiatives, and in particular, the development of dedicated virtual simulation technologies.^23,24 Recent work indicates these initiatives may minimize differences in outcomes between operators with different levels of experience.²⁵ Yet the application of volume requirements in routine clinical practice, the quality of various educational initiatives, and the overall impact of both approaches on outcomes remain largely unknown.

In addition to differences in 30-day risk-adjusted mortality, we also found that failure to use embolic protection devices was more common among patients treated by low volume operators and earlier on during a new operator’s experiences. Although we did not have sufficient clinical or anatomic information to identify why an embolic protection device may not have been used in a particular patient, a failure to receive these devices is a potentially important process measure that needs to better understood. It could be that, as operators are gaining more skill with these devices, they are simultaneously and independently improving other procedural techniques that lead to better outcomes. However, it could also be that operators with more experience are better at selecting patients based on their suitability for embolic protection devices or even deferring carotid stenting when they cannot be used.

Our study should be interpreted in the context of the following limitations. We examined carotid stenting in elderly Medicare beneficiaries. While this age group represents approximately three-quarters of the patients undergoing the procedure in the United States, our results may not be generalizable to younger patients. This also means that our determination of operator experience underestimates “overall” experience for any individual operator, especially if their case-mix of Medicare beneficiaries differs substantially from other operators. As such, inferring a precise number of procedures that will be associated with better outcomes is not possible from this study. However, determining such a number may be less relevant than understanding the overall association between greater operator experience and outcomes with carotid stenting.

A second issue related to Medicare claims data is the potential for residual confounding, particularly given the minimal changes we found between unadjusted and adjusted odds ratios from our models. Because we were unable to account for several clinical and anatomic factors, it may be that patients who were treated by lower-volume operators or early during their operator’s experience are sicker or more complex in unmeasured ways than high-volume operators. While this limits the ability to draw causal inferences from our analysis, the association we identified does point toward the need for further studies to understand potential reasons why outcomes were consistently worse among less experienced operators. Related concerns with using Medicare data include their limited ability to assess additional outcomes of importance (e.g., stroke) or the procedure’s overall appropriateness relative to alternative treatments, such as carotid endarterectomy or even medical therapy.

Third, our analyses examining early versus late experiences with carotid stenting in new operators is likely to have included some operators who performed carotid stenting prior to the date of the initial national coverage decision by the CMS. Carotid stenting has been described dating back to the mid 1990s, although in the past its use was more limited. Yet we suspect that any misclassification of operators, if present, biased our findings toward the null. Finally, we examined the association between these 2 measures of operator experience and outcomes across a large number of physicians. Although our findings represent an “average” effect, studies of the “volume-outcome” relationship and performance improvement in other areas suggest that individual operators develop and maintain their skills at varying rates,²⁶ and it is even possible that this relationship could vary based on their prior experiences with other endovascular procedures.

In conclusion, many physicians have begun performing carotid stenting in Medicare beneficiaries during recent years, although most operators appear to have developed limited experience with the procedure over time. This finding is important since risk-adjusted outcomes following the procedure are worse among very-low volume operators and early during an operator’s experience.

Acknowledgment

We are grateful to Onur Baser, PhD, Li Wang, MS, PhD, and Mingrui Lu, MPH for their roles in preparing data. The authors have no financial conflicts of interest to disclose in regards to this work. This project was supported by a grant from the National Institutes of Health (5R21AG032155-02). The funding agency was not involved in the design and conduct of the study; in data management or analysis; or in manuscript preparation. The views expressed in this article are those of the authors and do not necessarily represent the views of this agency or the Department of Veterans Affairs and the Department of Health and Human Services. Dr. Nallamothu had full access to all of the data in the study and takes responsibility for the integrity of the data and accuracy of the data analysis.

References

1. Davis SM, Donnan GA. Carotid-artery stenting in stroke prevention. N Engl J Med. 2010;363(1):80–82. [PubMed]

2. Goodney PP, Travis LL, Malenka D, et al. Regional variation in carotid artery stenting and endarterectomy in the Medicare population. Circ Cardiovasc Qual Outcomes. 2010;3(1):15–24. [PubMed]

3. Halm EA. The good, the bad, and the about-to-get ugly: national trends in carotid revascularization. Arch Intern Med. 2010;170(14):1225–1227. [PubMed]

4. Lin PH, Bush RL, Peden EK, et al. Carotid artery stenting with neuroprotection: assessing the learning curve and treatment outcome. Am J Surg. 2005;190(6):855–863. [PubMed]

5. Verzini F, Cao P, De Rango P, et al. Appropriateness of learning curve for carotid artery stenting: An analysis of periprocedural complications. J Vasc Surg. 2006;44(6):1205–1211. [PubMed]

6. Rosenfield K, Babb JD, Cates CU, et al. Clinical competence statement on carotid stenting: training and credentialing for carotid stenting--multispecialty consensus recommendations: a report of the SCAI/SVMB/SVS Writing Committee to develop a clinical competence statement on carotid interventions. J Am Coll Cardiol. 2005;45(1):165–174. [PubMed]

7. Connors JJ, Sacks D, Furlan AJ, et al. Training, competency, and credentialing standards for diagnostic cervicocerebral angiography, carotid stenting, and cerebrovascular intervention: a joint statement from the American Academy of Neurology, the American Association of Neurological Surgeons, the American Society of Interventional and Therapeutic Neuroradiology, the American Society of Neuroradiology, the Congress of Neurological Surgeons, the AANS/CNS Cerebrovascular Section, and the Society of Interventional Radiology. Neurology. 2005;64(2):190–198. [PubMed]

8. Bates ER, Babb JD, Casey DE, et al. ACCF/SCAI/SVMB/SIR/ASITN 2007 clinical expert consensus document on carotid stenting: a report of the American College of Cardiology Foundation Task Force on Clinical Expert Consensus Documents (ACCF/SCAI/SVMB/SIR/ASITN Clinical Expert Consensus Document Committee on Carotid Stenting) J Am Coll Cardiol. 2007;49(1):126–170. [PubMed]

9. Clair DG. Evolution of carotid stenting: regulatory issues. Semin Vasc Surg. 2008;21(2):64–68. [PubMed]

10. [Last Accessed January 15, 2011];HCUPnet: Healthcare Cost and Utilization Project. Available at: http://hcupnet.ahrq.gov/

11. [Accessed August 15, 2011];National Coverage Analyses (NCA) Tracking Sheet for Carotid Artery Stenting. Transmittal No. 33. Available at: http://www.cms.gov/transmittals/downloads/R33ncd.pdf.

12. Wholey MH, Wholey M, Mathias K, et al. Global experience in cervical carotid artery stent placement. Catheter Cardiovasc Interv. 2000;50(2):160–167. [PubMed]

13. Yadav JS, Wholey MH, Kuntz RE, et al. Protected carotid-artery stenting versus endarterectomy in high-risk patients. N Engl J Med. 2004;351(15):1493–1501. [PubMed]

14. Brott TG, Hobson RW, Howard G, et al. Stenting versus endarterectomy for treatment of carotid-artery stenosis. N Engl J Med. 2010;363(1):11–23. [PMC free article] [PubMed]

15. Kastrup A, Groschel K, Krapf H, et al. Early Outcome of Carotid Angioplasty and Stenting With and Without Cerebral Protection Devices: A Systematic Review of the Literature. Stroke. 2003;34(3):813–819. [PubMed]

16. [Accessed August 15, 2011];National Coverage Analyses (NCA) Tracking Sheet for Carotid Artery Stenting. Transmittal No. 115. Available at: http://www.cms.gov/transmittals/downloads/R115NCD.pdf.

17. van Walraven C, Austin PC, Jennings A, Quan H, Forster AJ. A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data. Med Care. 2009;47(6):626–633. [PubMed]

18. Ramsay CR, Grant AM, Wallace SA, et al. Statistical assessment of the learning curves of health technologies. Health Technol Assess. 2001;5(12):1–79. [PubMed]

19. Anderson HV, Rosenfield KA, White CJ, et al. Clinical features and outcomes of carotid artery stenting by clinical expert consensus criteria: a report from the CARE registry. Catheter Cardiovasc Interv. 2010;75(4):519–525. [PubMed]

20. Reznick RK, MacRae H. Teaching surgical skills--changes in the wind. N Engl J Med. 2006;355(25):2664–2669. [PubMed]

21. Ramsay CR, Grant AM, Wallace SA, et al. Assessment of the learning curve in health technologies. A systematic review. Int J Technol Assess Health Care. 2000;16(4):1095–1108. [PubMed]

22. See WA, Cooper CS, Fisher RJ. Predictors of laparoscopic complications after formal training in laparoscopic surgery. JAMA. 1993;270(22):2689–2692. [PubMed]

23. Gallagher AG, Cates CU. Approval of virtual reality training for carotid stenting: what this means for procedural-based medicine. JAMA. 2004;292(24):3024–3026. [PubMed]

24. Cates CU, Patel AD, Nicholson WJ. Use of virtual reality simulation for mission rehearsal for carotid stenting. JAMA. 2007;297(3):265–266. [PubMed]

25. Schreiber TL, Strickman N, Davis T, et al. Carotid artery stenting with emboli protection surveillance study: outcomes at 1 year. J Am Coll Cardiol. 2010;56(1):49–57. [PubMed]

26. Grantcharov TP, Reznick RK. Teaching procedural skills. BMJ. 2008;336(7653):1129–1131. [PMC free article] [PubMed]