Economic evaluations allow costs to be included, alongside data on safety and effectiveness, in healthcare policy decisions. As healthcare costs rise, the availability of these economic evaluations becomes increasingly important to the formulation of disease management strategies which are both clinically effective and financially responsible. According to the National Center for Complementary and Alternative Medicine (NCCAM), CAM is ‘a group of diverse medical and healthcare systems, practices and products that are not generally considered part of conventional medicine’.⁹ In not being part of conventional medicine, individual complementary therapies and emerging models of integrative medicine (ie, coordinated access to both conventional and complementary care)—collectively termed as complementary and integrative medicine (CIM)—are often excluded in financial mechanisms commonly available for conventional medicine,² and are rarely included in the range of options considered in the formation of healthcare policy. The availability of economic data could improve the consideration and appropriate inclusion of CIM in strategies to lower overall healthcare costs. In addition, economic outcomes are relevant to the licensure and scope of practice of practitioners, industry investment decisions (eg, the business case for integrative medicine), consumers and future research efforts (ie, through identifying decision-critical parameters for additional research¹⁰).

A number of systematic reviews of economic evaluations of CIM have been published.^11–23 Five of these prior reviews attempted to capture all economic evaluations of CIM therapies across all conditions.^{11 19–21 23} However, it is unclear as to whether all or even the majority of economic evaluations of CIM have been identified by these reviews. The searches are dated; the search strategy in the most recent review only captured articles published through 2007.²³ The databases searched were limited—for example, only one used CINAHL,²¹ and only two others used EMBASE,^{19 23} in addition to Medline and AMED. Finally, these reviews generally used limited search terms to identify CIM studies. All but one only used variations on the terms ‘complementary’ or ‘alternative’ ‘medicine’ or ‘therapy’. Unfortunately, other reviewers have found that these search terms do not capture all CIM studies,^{24 25} which may be a reflection of the difficulty in defining what is and is not CIM.²⁶ The search by Maxion-Bergemann et al¹¹ also added individual therapies as search terms, but only included homeopathy, phytotherapy, traditional Chinese medicine, anthroposophic medicine and neural therapy. No search included ‘integrative medicine’.

Defining a comprehensive search strategy for CIM is challenging.^{24 27–29} There have been a number of efforts to develop a concise definition of CAM.^{26 30} This review used the one developed by the members of the Cochrane CAM Field³¹ and then added the terms ‘integrative’, ‘integrated’ and ‘collaborative’ medicine. The Cochrane CAM definition starts with the NCCAM definition⁹ and then refines it by specifically including all therapies ‘based upon the theories of a medical system outside the Western allopathic medical model’ (eg, traditional Chinese medicine and Reiki), and including others depending on the context and setting of their use. The context of use considers treatment/condition combinations and excludes those ‘currently considered to be standard treatment’, and the setting of use generally includes self-care and therapies delivered by CIM providers, but excludes therapies ‘delivered exclusively by conventionally credentialed medical personnel or exclusively within hospital settings’. Therefore, therapies such as chemotherapy regimens (eg, chronotherapy³²), and therapies requiring surgical implantation (eg, neuroreflexotherapy³³) or the placement of a feeding tube³⁴ were not included even though these therapies appeared in our search. In cases where CIM therapies (eg, biofeedback or hypnosis) were included as part of a package of care (eg, with cognitive behavioural therapy), a judgement was made as to whether the CIM portion of the treatment made up half or more of the overall package of care under study. If so, the package of care was included as CIM. Because more than half of CIM users use multiple CIM therapies,³⁵ studies of packages of therapies and coordinated care were identified as such.

Articles were categorised as full economic evaluations if they compared the costs (inputs) and consequences (economic, clinical and/or humanistic outcomes³⁶) of two or more therapeutic alternatives applied to the same patient population (ref.³⁷, p. 11). Otherwise, they were considered partial evaluations, for example, cost-identification or cost-comparison studies.³⁸ Studies that estimated resource utilisation were included as economic evaluations if the utilisation data were detailed enough to allow monetary valuation.

Two reviewers (PMH and BLP) evaluated all articles for inclusion and extracted all data. Disagreements were resolved by discussion between the two review authors, or, if needed, by the other coauthors. Because the results of economic evaluations can rapidly lose relevance with time, mainly due to changes in practice patterns and cost structures, data were extracted only from the economic evaluations published 2001–2010. Extracted data were entered into an Excel template developed for a previous review.²⁰ The type(s) of CIM evaluated and the target population were captured for all economic evaluations. Various indicators of study quality were captured for all full economic evaluations, and more detailed data and results were captured only for those full economic evaluations that met five quality criteria.

The quality of an economic evaluation can be judged along two general dimensions: (1) whether the study was a quality measure of outcomes for its target population and location—that is, whether it was internally valid; and (2) whether enough information is provided for the study's results to be transferable (‘generalisable’).³⁹ Health outcomes are to some extent considered generalisable across settings; however, because resource availability, practice patterns and relative prices can vary greatly, economic outcomes usually are not.⁴⁰ Therefore, one goal in economic evaluation is to ensure the transferability of study results—that is, to provide enough study detail so that results can be adapted (usually via modelling) to apply to other settings.³⁹ The 35-item BMJ checklist captures components of both dimensions of quality and was applied to all full economic evaluations.⁴¹ We also chose five quality criteria by which to identify a subset of full economic evaluations to highlight as being of most interest to policy makers. These quality criteria are based on recommendations made by the US Panel on Cost Effectiveness in Health and Medicine ⁴² and by well-known experts in the field,³⁷ and focus on the quality of the underlying study (the first type of quality):

Because the prices used to value resources are highly location-specific and setting-specific,^{39 46} we also note, for the articles meeting the above criteria, the presence of a study reporting criterion essential for the transferability of study results (usually via modelling):^{39 40} separate reporting of unit costs from resource use for economic evaluations alongside trials, or from model parameters (eg, transition probabilities) for economic evaluations using models.

Other data extracted for the economic evaluations which meet the five study-quality criteria are: treatment and study duration, primary clinical and economic outcome measures, the setting in which treatment took place, study design and sample size, the type (table 1) and perspective (ie, the point of view used to define costs) of the economic analysis, and incremental cost-effectiveness of the CIM alternative compared to usual care. Incremental cost-effectiveness is reported in 2011 US$ and is calculated from reported results by first converting the study currency to US$ using the Federal Reserve annual exchange rate⁴⁷ for the study's currency year and then inflated to 2011 values using the medical care component of the Consumer Price Index.⁴⁸

As shown in table 2, the 204 economic evaluations published in the past 10 years are spread across a wide range of CIM therapies applied to a number of different study populations. The biggest concentration of full economic evaluations (19 in number) pertained to the use of NCCAM's definition of manipulative and body-based practices (eg, chiropractic, osteopathic manipulation, massage, etc) for the treatment of back pain.^53–72 However, even this subgroup is fairly heterogeneous in terms of the therapy (or therapies) tested and/or the type of back pain treated. Eight of these comparisons involved chiropractic care for back pain; one for chronic,⁵³ one for acute⁵⁷and six for either type.^{59 60 63 64 67 68} Five evaluated spinal manipulation and manual therapy provided by physiotherapists for chronic back pain (one),⁶⁵ acute back pain (two)^{58 69} or either (two).^{56 68} Four involved osteopathic manipulation; one for chronic⁷¹ and one for subacute back pain⁷² and two for musculoskeletal conditions including back pain.^{66 68} Three evaluated massage; two for chronic^{55 62} and one for acute back pain.⁵⁷ The last two studies evaluated a musculoskeletal physician (treatment ‘with a combination of manual therapy, injections, acupuncture and other pain management techniques’) for orthopaedic referrals;⁵⁴ and a Finnish folk medicine practice called ‘bone setting’ for the treatment of patients with chronic back pain.⁶¹

Table 3 shows the results of the application of the 35-item BMJ checklist to the full economic evaluations published 2001–2010.⁴¹ On average, the number of applicable items met by each article stayed fairly constant during this period. However, the application of two key items (ie, the proper use of discounting and the inclusion of sensitivity analysis) and the disclosure of funding sources improved significantly, and reporting of the study time horizon worsened significantly. As expected, the average overall and individual-item percentages were higher for the higher-quality articles (those meeting the five study-quality criteria) and for CUAs of CIM. It is not surprising that CUA's quality is higher. They generally involve more effort than other CEAs and are required or recommended by various national guidelines.^{42 73–75} Nevertheless, it seems as though the quality of CUAs of CIM is generally comparable to, or slightly better than, that seen in CUAs across all medicine, at least in terms of the Tufts quality score, disclosure of funding sources and the five items where comparable data are available.^{76 77}

The number of full evaluations meeting each of the five study-quality criteria are: comparison to usual care 97 (85%), all costs from a recognised perspective 96 (84%), health outcomes from a randomised or matched-control trial 86 (75%), patient-specific data on costs and outcomes 105 (92%) and sensitivity analyses 37 (32%). Sixty-two (54%) of full evaluations met the first four of these and 31 (27%) met all five. A summary of the results of these 31 higher-quality articles (covering 28 different studies) is shown in table 4.^{54 60 62 68 71 78–103} Twenty-two of these articles (19 of the studies) reported resource use (trials) or model parameters (models) separate from unit prices—a minimum measure of study transferability.^{54 62 68 71 78 80–85 87–93 95 100 101 103} For those studies which included a randomised trial, the modified Jadad scores ranged from 2 to 4 on a scale from 0 to 4. The Tufts CEA Registry quality scores for the studies containing a CUA ranged from 4 to 6.5 on a scale from 1 to 7. The percentage of the applicable items on the BMJ checklist met by these studies ranged from 66% to 97%.

Of the 56 comparisons made in these studies, 16 (29%) are cost saving—that is, the added CIM therapy had better health outcomes and lower costs than usual care alone. Cost savings were seen for acupuncture alone (instructional visits with an acupuncturist followed by home self-care by the partner for pregnant women with breech presentations at 33 weeks in terms of reductions in both breech presentation at birth and ceasareans in the Netherlands,⁹¹ and treatment by traditional Chinese medicine-trained licensed acupuncturists in private acupuncture clinics in the UK for low-back pain in terms of quality-adjusted life-years or QALYs from the societal perspective⁸⁵) and in combination with other therapies (along with manual therapy, injections and other pain management for patients referred to an orthopaedic surgeon's office in Scotland who were unlikely to need surgery in terms of both improvements in health-related quality of life and QALYs⁵⁴). Cost savings were also seen for manual therapy delivered by a physiotherapist, who is also a registered manual therapist, for neck pain in terms of perceived recovery, pain, neck disability and QALYs⁸²; for preoperative oral supplementation with arginine and ω-3 fatty acids for patients with gastrointestinal cancer undergoing surgery¹⁰²; for vitamin K₁ supplementation for postmenopausal women with osteopenia and osteoporosis in terms of QALYs¹⁰³; for supplementation with vitamins C and E and β-carotene for cataract prevention⁹⁰; for fish oil supplementation in men with a history of heart attack⁸⁷; for tai chi to prevent hip fractures in nursing home residents⁹⁵ and for naturopathic care offered through a worksite clinic for chronic low-back pain in terms of both reductions in absenteeism and gains in QALYs.⁸⁰

The strengths of this study are the comprehensive search strategy, which revealed a substantial number of published economic evaluations of CIM, the use of two reviewers and the use of multiple measures of study quality. Higher-quality studies were identified and highlighted for policy makers using a simple objective list of quality criteria, which reduced the potential for bias. The weaknesses of this study are similar to those of the other systematic reviews. The reviewers were not blinded to journals and article authors, which may have influenced results. Also, some aspects of what makes a quality economic evaluation could not be judged from what was reported. For example, ideally, pragmatic trials enrol patients typical of normal caseload in typical settings with typically trained and experienced practitioners following them under routine conditions (ref.³⁷, p. 251). Judgements as to whether these criteria were met were not always possible from the reports, and were beyond the scope of this review. Finally, publication bias was not assessed. However, since the major goal of this study was to establish the extent of the published literature on this topic and to highlight the results of the higher-quality studies, it is not clear that publication bias is relevant here.

The number of economic evaluations of CIM found and reviewed by this study far exceeds the numbers found in previous studies.^{11 19–21 23} This study found a total of 338 economic evaluations of CIM published between and including 1979 and 2010; 211 of these were full economic evaluations. White and Ernst¹⁹ identified 34 economic evaluations of CAM published 1987–1999; 11 of which were full economic evaluations. Between 1999 and October 2004, Herman et al²⁰ identified 56 economic evaluations of CAM (39 full evaluations). Between 1994 and May 2004 Hulme and Long²¹ identified 19 full economic evaluations of CAM, and over a similar period (1995–2007) Doran et al²³ found 43 economic evaluations (15 full evaluations). Maxion-Bergemann et al¹¹ identified 5 (1 full) economic evaluations over an unspecified search period. The large number of economic evaluations found in this study reflects the facts that: (1) all evaluations from previous reviews were included; (2) a number of studies have been published since the last search dates of prior reviews and (3) a more extensive search strategy was used. It should be noted that 20% of the articles (68 of 338) in this review were identified through bibliography searches and from expert lists. Therefore, even the application of a long list of search terms to multiple databases does not guarantee that all CIM studies will be identified. However, there is some evidence that the indexing of these articles in medical databases is improving; studies from bibliographies and expert lists made up 32% of found articles published 2000 and before, but only 12% recent articles.

There are several implications of this study for policy makers, clinicians and future researchers. First, there is a large and growing literature of quality economic evaluations in CIM. However, although indexing is improving in databases, finding these studies can require going beyond simple CIM-related search terms. Second, the results of the higher-quality studies indicate a number of highly cost-effective, and even cost saving, CIM therapies. Almost 30% of the 56 cost-effectiveness, cost-utility and cost-benefit comparisons shown in table 4 (18% of the CUA comparisons) were cost saving. Compare this to 9% of 1433 CUA comparisons found to be cost saving in a large review of economic evaluations across all medicine.¹⁰⁶ Third, by meeting the five study-quality criteria, the studies shown in table 4 can each be considered a reasonable indicator of the health and economic impacts of the CIM therapy studied, at least in that population and setting. These studies, especially those showing cost savings, should be considered further for applicability in other settings. This requires the study to be transferable.³⁹ Fortunately, the majority of the higher-quality studies met our measure of study transferability—resource use or model parameters, and unit costs were reported separately.