|Home | About | Journals | Submit | Contact Us | Français|
Appropriateness criteria were applied to outpatient transthoracic echocardiographic (TTE) studies.
Indications were rated as appropriate, inappropriate, or unclassifiable, considering provider-stated indications, previous TTE studies, symptom changes, and patient-stated indications. Clinically important new or unexpected findings were recorded.
Of 368 TTE studies, 206 (56%) were appropriate, 31 (8%) were inappropriate, and 131 (35%) were unclassifiable. Appropriateness was not correlated with patient or provider demographics. In 288 cases with prior TTE studies, there were 92 (32%) important new findings and 63 (22%) unexpected findings, of which 20% were from inappropriately ordered and 31% from unclassifiable TTE studies. Appropriateness was not associated with new (odds ratio, 1.23; 95% confidence interval, 0.48–3.18) or unexpected (odds ratio, 1.15; 95% confidence interval, 0.38–3.52) findings. Provider type and level of training were not correlated with new or unexpected findings.
Many indications for TTE studies were unclassifiable. A high percentage of inappropriately ordered TTE studies yielded important information. Care must be taken in judging the value of TTE studies solely on the basis of appropriateness criteria.
Advances in imaging have unquestionably revolutionized the diagnosis and management of cardiovascular disease. As a result, cardiac imaging has grown exponentially and has been implicated as a factor in the overall increase in health care costs. With constant improvement in technology, there is a critical need for clinicians and payers to promote the appropriate utilization of resources. In 2007, the American College of Cardiology Foundation and the American Society of Echocardiography established appropriateness criteria for transthoracic and transesophageal echocardiography to help guide the efficient use of resources and deliver high-quality, rational medical care.1
These criteria classify indications for echocardiography as appropriate, inappropriate, or uncertain. The goals of applying appropriateness criteria include improving patient outcomes, including survival and health status, as well as reducing the number of unnecessary imaging studies. An “appropriate” imaging study is defined as an one that is generally acceptable as a reasonable approach for an indication, an “inappropriate” study is defined as generally not acceptable and not reasonable, and a study is deemed of “uncertain” appropriateness if it is acceptable or reasonable but more research or patient information is needed. A technical panel presented 59 indications, many of them “purposefully broad.” In the appropriateness criteria for transthoracic and transesophageal echocardiography, 44 indications were deemed appropriate, 14 inappropriate, and 1 uncertain.
We applied the appropriateness criteria to a sample of transthoracic echocardiographic (TTE) studies at our tertiary referral echocardiography laboratory. We hypothesized that there would be a large number of indications not addressed in the appropriateness criteria document (which are termed “unclassifiable”). We also hypothesized that there would be a significant number of technically “inappropriate” TTE studies that nonetheless yielded clinically important findings.
The study was approved by the local institutional review board. Consecutive outpatient TTE studies performed over a 2-month period (October to November 2007) were reviewed. All ordered TTE studies were performed, regardless of the indication. Intake forms were developed to assess patient demographics, risk factors for cardiac disease, referring provider information, and indication and results from previous echocardiographic studies. Indications for studies were classified into categories as listed in Table 1, on the basis of the appropriateness criteria document.1 Patients were briefly interviewed regarding recent changes in symptoms and recent procedures (such as cardiac catheterization, bypass surgery, arrhythmia ablation, and pacemaker placement) and were asked to state why they believed the TTE studies were ordered. Referring provider specialty was obtained by collecting provider names from electronic orders and prescriptions and cross-checking these data with a health system–wide online provider database. Each referring provider was categorized as a general internist, cardiologist, other medical subspecialist, cardiac surgeon, obstetrician/gynecologist, or other surgeon. Information on ordering providers (the providers who actually input the indications for the TTE studies) was extracted from the electronic ordering system. This information could not be obtained, in most instances, from handwritten prescriptions. Ordering providers were categorized as attending physicians, fellows, resident physicians, physician assistant or nurse practitioner, or other nurse.
A level III–certified echocardiographer rated each indication as appropriate, inappropriate, or unclassifiable, according to the published appropriateness criteria, using provider-stated indications, the current and prior echocardiographic reports, and patient interviews and demographics. The only uncertain classification in the appropriateness criteria pertains to transesophageal echocardiography. A separate, blinded level III–certified echocardiographer reviewed a subsample of 50 TTE studies to establish interobserver agreement for appropriateness ratings. The guidelines for applying the appropriateness criteria to each TTE study were intentionally broad. Because it was difficult to know what each provider was thinking at the time of inputting the order, the raters intentionally classified studies as appropriate or unclassifiable unless they clearly met criteria for inappropriateness.
Additionally, the first echocardiographer reviewed each TTE report and the prior echocardiographic report to determine if there were any clinically important findings that constituted a significant change from the prior echocardiographic study or any new findings. These significant changes or new findings were prespecified as findings that should prompt an intervention, including follow-up imaging studies (Table 2). These changes or new findings were compared with information from the demographics, cardiac risk factors, indications for the TTE studies, changes in symptoms, and recent procedures. If no mention of the changes or new findings existed in these sources, the findings were also classified as unexpected.
Logistic regression analysis based on the following variables was used to identify predictors of clinically significant new findings: age (per 10-year increase), race or ethnicity, gender, prior echocardiography, time since prior echocardiography, referring physician category, diabetes mellitus, coronary artery disease, history of stroke, atrial fibrillation, changes in symptoms, and recent procedures. Odds ratios (ORs) with 95% confidence intervals (CIs) were calculated. All tests were 2 sided, and an α value <0.05 was used to define statistical significance. SPSS for Windows version 13.0 (SPSS, Inc., Chicago, IL) was used for statistical analyses.
The sample included 368 TTE studies (see Table 3 for demographics), of which 206 (56%) were rated as appropriate, 31 (8%) were rated as inappropriate, and 131 (35%) were rated as unclassifiable. Interobserver agreement for the subsample (n = 50) was modest (κ = 0.67). The most frequent unclassifiable indications included preoperative risk assessments (n = 17), follow-up several months after pulmonary vein isolation procedures for atrial fibrillation (n = 15), coronary artery disease with no change in symptoms (n = 9), and follow-up of mild-moderate, moderate, or moderate-severe mitral regurgitation (n = 7). No significant association was found between appropriateness category and any of the following: age, ethnicity, gender, coronary disease, diabetes, atrial fibrillation, history of cerebral vascular accident, and recent procedure.
Most of the patients were referred for TTE studies by cardiologists (53%), internists or family practitioners (16%), and other medical subspecialists (14%). Attending physicians input indications for 187 TTE studies (50%), physician assistants or nurse practitioners for 33 (9%), fellows for 16 (4%), resident physicians for 10 (3%), and nurses for 17 (5%). For 105 studies (29%), the levels of training of the ordering providers could not be determined (Table 4). The most common indication categories according to provider type are detailed in Table 5.
In 288 cases (78%), prior TTE studies were available for review at our institution. Clinically important new findings were found in 92 studies (32%). Important findings that were unexpected were found in 63 (22%), including 6 (20%) of the inappropriately ordered TTE studies and 41 (31%) of the unclassifiable studies. Among TTE studies with appropriate indications, (1) valvulopathy, (2) elevated pulmonary pressure, and (3) left ventricular dysfunction were the most common clinically important findings, whether studies were classified as expected or known, unexpected, or new. Among the studies with inappropriate indications, (1) valvulopathy, (2) left ventricular dysfunction, and (3) elevated pulmonary pressure were the most common unexpected and new clinically important findings. Valvulopathy and left ventricular dysfunction were also common expected or known findings among inappropriately ordered TTE studies, along with regional wall motion abnormalities. There were two instances in which a provider-stated indication was rated inappropriate, but the TTE study was found to be appropriate when considering other information.
Appropriateness was not associated with new (OR, 1.23; 95% CI, 0.48–3.18) or unexpected (OR, 1.15; 95% CI, 0.38–3.52) findings. Of all demographics and risk factors in a multivariate analysis that included age (per 10-year increase), race or ethnicity, gender, prior echocardiography, time since prior echocardiography, referring physician category, diabetes mellitus, coronary artery disease, history of stroke, atrial fibrillation, changes in symptoms, and recent procedures, only patient age was independently associated with clinically significant new findings (OR per 10-year increase, 1.20; 95% CI, 1.04–1.39; P =.01).
There was no statistically significant correlation between referring provider type or ordering provider level of training and appropriateness, clinically important results, or unexpected findings.
A recent study of Medicare beneficiaries between 1999 and 2004 demonstrated an annual growth rate in spending on echocardiography of 7.7%, consistent with the overall growth rate in medical services. The authors noted a shift in the volume of echocardiographic studies from hospitals to outpatient offices, with the majority of TTE studies ordered by noncardiologists.2 Although the more appropriate use of echocardiography will not solve the problem of medical cost inflation, it may play an important role in reducing a component of that cost.3
Appropriateness criteria are designed to provide a framework for the rational use of imaging modalities. The application of these criteria for education, quality improvement, and preauthorization may increase system efficiency by reducing the number of inappropriate diagnostic examinations performed. However, the application of appropriateness criteria should not be so narrow as to prevent the performance of clinically useful studies. Thus, the appropriateness criteria document advises that some indications initially deemed inappropriate (and, perhaps, indications for which no appropriateness criteria exist) should still be performed and compensated if the ordering clinician provides “additional documentation to justify payment because of unique circumstances or the clinical profile of the patient.”1 The added time and hassle of requiring and providing such additional documentation would be difficult to capture in analyses of prescreening algorithms and may offset expected gains in efficiency. It is of critical importance that any algorithm used to define the appropriateness of given TTE studies be evaluated for significance (a substantial proportion of ordered exams are inappropriate), applicability (the criteria cover most indications), and validity (patients benefit from appropriate TTE studies and do not benefit from studies defined as inappropriate).
The rate of inappropriate indications found in this study seems surprisingly low for a busy academic laboratory that performs each TTE study ordered, irrespective of indication, but is similar to the 11% inappropriate rate found in a recent study by Ward et al.4 This rate is considerably lower than those found in two studies performed in Italy, albeit using different sets of criteria. One found that 19.6% of TTE studies ordered were inappropriate and 36.8% were of “doubtful” appropriateness, using guidelines developed by the Italian Federation of Cardiology.5 A second study found that only half of TTE orders adhered to the 2003 American College of Cardiology, American Heart Association, and American Society of Echocardiography guidelines for the clinical application of echocardiography.6,7 It should be noted that no echocardiography laboratory is likely to achieve 100% appropriate indications. An acceptable rate of inappropriate studies has not been defined.
In determining the appropriateness of individual TTE studies, we intentionally “gave the benefit of the doubt” to the ordering providers by using all available data to rate appropriateness, not just the indications or diagnostic codes that the providers listed on the order forms or prescriptions. We thus intentionally attempted to assess the appropriateness of the echocardiographic studies, as well as the appropriateness of the orders for the studies (which can be manipulated if ordering providers list false but “appropriate” indications). Although considering all of these factors could have resulted in an artificially low rate of inappropriate indications, we found only two instances in which a provider-stated indication was rated inappropriate but the TTE study was found to be appropriate when considering other data.
In addition, the low rate of inappropriate indications in the current study could have reflected the fact that the majority of TTE studies were ordered by cardiologists. This finding mirrors the results of Barbier et al,6 who found that family physicians and specialists demonstrated no difference in adherence to guidelines in ordering TTE studies. Ward et al,4 however, found that noncardiac specialists were more likely to order inappropriate studies (40% vs 17%). The discrepancies may reflect differences between the studies in applying appropriateness criteria to different sets of information from specific TTE studies and to differences in institution-specific ordering practices. Although it is difficult to determine from the methods sections of the prior studies, it appears that our study may have considered more information about each TTE order (Figure 1, Videos 1 and 2). In addition, the modest interrater reliability in the current study suggests that there are different ways to apply the same criteria to the same information. Of note, our interrater reliability was better than that of a recent study examining the appropriateness of nuclear and stress echocardiographic test orders (κ = 0.56 and κ = 0.27 for orders for patients with prior stress tests).8
We also found no correlation between the levels of training of the ordering providers who actually input the indications for the TTE studies and whether the orders were appropriate. We found that nurses, nurse practitioners, physician assistants, and physicians at different training levels did not differ in the appropriateness of indications they input when ordering TTE studies. Out data suggest that educational interventions to encourage more appropriate test ordering should target all of these groups.
Although few indications were inappropriate, a large number of TTE studies were ordered for reasons not addressed in the appropriateness criteria document (Figure 2, Video 3). The number of unclassifiable indications probably varies significantly among institutions and providers. A large, academic echocardiography laboratory will encounter TTE studies performed in patients undergoing specialized care or receiving novel devices and therapies (eg, pulmonary vein isolation procedures, heart transplantation). Many of these patients, however, will return to routine care by community cardiologists or generalists after their procedures and may undergo follow-up echocardiography ordered by these types of providers. Thus, the number TTE studies ordered for currently unclassifiable indications is likely to grow in all settings. Just as there will always be TTE studies ordered for inappropriate indications, there will always be TTE studies ordered for unclassifiable indications, though the percentage of both may be reduced. The large number of these indications in our study argues for further research to define the appropriateness of these indications, leading to an expansion of the appropriateness criteria. In the meantime, such unclassifiable indications may present a substantial barrier when attempting to apply appropriateness criteria to increase system efficiency, rate quality, educate referring providers, judge suitability for payment, or ration imaging resources.
Transthoracic echocardiography is not inexpensive and requires significant time and expertise. Investigations of the use of screening echocardiography in a variety of asymptomatic populations have demonstrated a relatively low percentage of clinically important results,5–12 suggesting that full-feature echocardiography as a screening modality is not cost effective. Many of the indications listed as inappropriate in the appropriateness criteria document involve echocardiography for asymptomatic patients. Similarly, many inappropriate criteria involve frequent (annual) repeat echocardiographic evaluations of patients without severe pathology who demonstrate no changes in symptoms.1,9–16
However, this study demonstrated that a significant percentage of echocardiographic studies classified as inappropriate by criteria yielded important clinical data, by demonstrating either unexpected changes or previously unknown significant results. The presence of such results was associated with patient age but not other demographic or historical variables. This association with age held true even for patients who had undergone TTE studies <1 year prior, raising the possibility that for certain select populations, follow-up echocardiography may be clinically useful.
Our study did not consider known and unchanged abnormal results to be clinically important, though the lack of progression may have been a significant finding. Because we were unable to determine how a practitioner would apply specific results, we chose a more conservative way to define “clinically important” findings by focusing on new or unsuspected findings. Nonetheless, our data suggest that neither clinical factors nor the appropriateness of an indication provide definitive guidance in capturing the true potential clinical utility of outpatient echocardiography. As stated in the appropriateness criteria document, “the appropriateness criteria for TTE/TEE were not developed to quantify the incremental information that could be obtained by performing the test for reasons beyond those stated in an individual indication.”1 Thus, although an indication may be deemed inappropriate, a study itself may be clinically useful if it uncovers management changing or unexpected findings, potentially leading to improved patient outcomes. Work by Senni et al17 highlighted a corollary of this point in a population at risk for heart failure, demonstrating that the underuse of echocardiography led to poorer outcomes. Prospective endeavors to deny payment for inappropriate TTE studies must consider the factor of unexpected clinically important results and the potential impact on individual and population health, should these results remain undiscovered. Although additional research is needed to define the impact of educational interventions on provider ordering patterns, this specific application of the appropriateness criteria might afford the most benefit while avoiding the problems inherent with preauthorization or rationing interventions.
This study was performed at a single, large, urban academic center, and the results may not be generalizable to other echocardiography laboratories. Patients at tertiary care centers may be more likely to have cardiovascular disease and more likely to undergo echocardiography for unclassifiable indications than at most community-based echocardiography laboratories. Our particular institutional emphasis on electrophysiological procedures yielded a number of atypical indications. In addition, most orders came through an electronic medical record and ordering system that encourages the direct input of TTE indications by physicians, rather than the delegation of this task to other providers. There is no financial incentive for academic staff physicians at our institution to refer patients for echocardiography (other than the financial health of the cardiovascular division). All of these factors might reduce the rate of inappropriateness and potentially increase the percentage of clinically significant findings. We focused on outpatient TTE studies, for which indications are more easily and accurately derived at our institution. The inclusion of inpatient TTE studies might have substantially increased the rate of inappropriately ordered examinations. Our study may not have had sufficient power to detect differences among provider types and levels of training in inputting appropriate indications. The modest interrater reliability in the current study highlights the fact that applying appropriateness criteria to rate a specific study requires a certain amount of interpretation and therefore is inherently subjective. This analysis was performed on data from TTE studies performed <6 months after the appropriateness criteria were published. Although possible, it is unlikely that the criteria were disseminated widely enough to decrease the rate of inappropriately ordered TTE studies.
In this sample of outpatient TTE studies, there was a 8.4% rate of inappropriate indications and a 35% rate of unclassifiable indications. Provider specialty and training level were not specifically associated with inappropriate indications. Clinically significant changes or new results were found in 20% of inappropriately ordered TTE studies and 31% of the unclassifiable TTE studies. The appropriateness of a study was not associated with new or unsuspected important findings.
Research is needed to classify more indications to improve the ability of the appropriateness criteria to evaluate and increase system efficiency in ordering outpatient TTE studies. Care must be taken in applying appropriateness criteria to judge the value of an echocardiographic study, because the strict application of these criteria may lead to the underdetection of important clinical findings.