Search tips
Search criteria 


Logo of bmjoBMJ OpenVisit this articleSubmit a manuscriptReceive email alertsContact usBMJ
BMJ Open. 2012; 2(5): e001046.
Published online 2012 September 3. doi:  10.1136/bmjopen-2012-001046
PMCID: PMC3437424

Are complementary therapies and integrative care cost-effective? A systematic review of economic evaluations



A comprehensive systematic review of economic evaluations of complementary and integrative medicine (CIM) to establish the value of these therapies to health reform efforts.

Data sources

PubMed, CINAHL, AMED, PsychInfo, Web of Science and EMBASE were searched from inception through 2010. In addition, bibliographies of found articles and reviews were searched, and key researchers were contacted.

Eligibility criteria for selecting studies

Studies of CIM were identified using criteria based on those of the Cochrane complementary and alternative medicine group. All studies of CIM reporting economic outcomes were included.

Study appraisal methods

All recent (and likely most cost-relevant) full economic evaluations published 2001–2010 were subjected to several measures of quality. Detailed results of higher-quality studies are reported.


A total of 338 economic evaluations of CIM were identified, of which 204, covering a wide variety of CIM for different populations, were published 2001–2010. A total of 114 of these were full economic evaluations. And 90% of these articles covered studies of single CIM therapies and only one compared usual care to usual care plus access to multiple licensed CIM practitioners. Of the recent full evaluations, 31 (27%) met five study-quality criteria, and 22 of these also met the minimum criterion for study transferability (‘generalisability’). Of the 56 comparisons made in the higher-quality studies, 16 (29%) show a health improvement with cost savings for the CIM therapy versus usual care. Study quality of the cost-utility analyses (CUAs) of CIM was generally comparable to that seen in CUAs across all medicine according to several measures, and the quality of the cost-saving studies was slightly, but not significantly, lower than those showing cost increases (85% vs 88%, p=0.460).


This comprehensive review identified many CIM economic evaluations missed by previous reviews and emerging evidence of cost-effectiveness and possible cost savings in at least a few clinical populations. Recommendations are made for future studies.

Keywords: Complementary Medicine, Health Economics, Statistics & Research Methods

Article summary

Article focus

  • Given the limited nature of previous systematic reviews, what is the extent of evidence on the economic impacts of complementary and integrative medicine (CIM)?
  • What are the range of therapies and populations studied, and the quality of published economic evaluations of CIM?
  • What are the results of the higher-quality, more recent (and likely most cost-relevant) economic evaluations of CIM?

Key messages

  • This study's comprehensive search strategy identified 338 economic evaluations of CIM, including 114 full evaluations published 2001–2010.
  • The cost-utility analyses found were of similar or better quality to those published across all medicine.
  • The higher-quality studies indicate potential cost-effectiveness, and even cost savings across a number of CIM therapies and populations.

Strengths and limitations of this study

  • The strengths of this study are the comprehensive search strategy, the use of two reviewers, the use of multiple measures of study quality and the identification of higher-quality studies, for which results are reported in detail, via an objective short-list of quality criteria, which reduced the potential for bias.
  • The weaknesses of this study are similar to those of the other systematic reviews: reviewers were not blinded to journals and article authors, and some aspects of what makes a quality economic evaluation could not be judged from what was reported.
  • Publication bias was not assessed. However, it is not clear as to whether publication bias is relevant, given the purposes of this review.


Between 1990 and 2007, four nationally representative surveys demonstrated that a third or more of US adults routinely used complementary and alternative medicine (CAM) therapies to treat their principal medical conditions.1–4 Total expenditures for CAM therapies were estimated at US$14 billion in 1990,1 US$27 billion in 19972 and US$34 billion in 2007.4 The 2007 US National Health Inventory Survey found that out-of-pocket expenditures for CAM therapies accounted for 11% of all out-of-pocket healthcare expenditures by Americans.4 Similar use numbers are seen in other countries.5–8 However, despite the popularity of and substantial expenditures on CAM therapies, their cost-effectiveness remains ill-defined and controversial.

Economic evaluations allow costs to be included, alongside data on safety and effectiveness, in healthcare policy decisions. As healthcare costs rise, the availability of these economic evaluations becomes increasingly important to the formulation of disease management strategies which are both clinically effective and financially responsible. According to the National Center for Complementary and Alternative Medicine (NCCAM), CAM is ‘a group of diverse medical and healthcare systems, practices and products that are not generally considered part of conventional medicine’.9 In not being part of conventional medicine, individual complementary therapies and emerging models of integrative medicine (ie, coordinated access to both conventional and complementary care)—collectively termed as complementary and integrative medicine (CIM)—are often excluded in financial mechanisms commonly available for conventional medicine,2 and are rarely included in the range of options considered in the formation of healthcare policy. The availability of economic data could improve the consideration and appropriate inclusion of CIM in strategies to lower overall healthcare costs. In addition, economic outcomes are relevant to the licensure and scope of practice of practitioners, industry investment decisions (eg, the business case for integrative medicine), consumers and future research efforts (ie, through identifying decision-critical parameters for additional research10).

A number of systematic reviews of economic evaluations of CIM have been published.11–23 Five of these prior reviews attempted to capture all economic evaluations of CIM therapies across all conditions.11 19–21 23 However, it is unclear as to whether all or even the majority of economic evaluations of CIM have been identified by these reviews. The searches are dated; the search strategy in the most recent review only captured articles published through 2007.23 The databases searched were limited—for example, only one used CINAHL,21 and only two others used EMBASE,19 23 in addition to Medline and AMED. Finally, these reviews generally used limited search terms to identify CIM studies. All but one only used variations on the terms ‘complementary’ or ‘alternative’ ‘medicine’ or ‘therapy’. Unfortunately, other reviewers have found that these search terms do not capture all CIM studies,24 25 which may be a reflection of the difficulty in defining what is and is not CIM.26 The search by Maxion-Bergemann et al11 also added individual therapies as search terms, but only included homeopathy, phytotherapy, traditional Chinese medicine, anthroposophic medicine and neural therapy. No search included ‘integrative medicine’.

The goal of this paper is to identify, to the extent possible, all published economic evaluations of CIM, describe the types of CIM evaluated and the clinical conditions for which they have been evaluated, and identify the recent (and therefore, most cost-relevant) higher-quality studies and highlight their results for policy makers. We also make recommendations for future economic evaluations of CIM.


Six electronic databases were searched from their inception through December 2010: PubMed, CINAHL, AMED, PsychInfo, Web of Science and EMBASE. To be as comprehensive as possible, a combination of 11 medical subject headings (MeSH) and 39 other search terms were used (box 1). In addition, bibliographies of found articles and reviews were searched, and key researchers in various areas of CIM were contacted for their lists of known studies. Although non-English language articles were collected, they are not analysed in this review.

Box 1

Search terms used for the PubMed search: (Complementary Therapies (medical subject headings (MeSH)), Dietary Supplements (MeSH), Micronutrients (MeSH), Trace Elements (MeSH), Vitamins (MeSH), acupuncture, alternative medicine, ayurvedic medicine, chiropractic, biofeedback, collaborative medicine, complementary and alternative medicine, botanical medicine, complementary medicine, diet, energy medicine, herbal medicine, herbs, homeopathy, hypnosis, integrated medicine, integrative medicine, massage, meditation, mind-body medicine, minerals, naturopathic medicine, naturopathy, nutrients, nutritional supplements, relaxation, spa therapy, traditional Chinese medicine OR vitamins) AND (Cost-Benefit Analysis (MeSH), Cost Control (MeSH), Cost Savings (MeSH), Costs and Cost Analysis (MeSH), Economics (MeSH), economics (Subheading), Insurance (MeSH), cost benefit, cost effectiveness, cost identification, cost minimisation, cost utility, economic evaluation, insurance claims, managed care OR technology assessment). Searches in the other five databases used the same text words and (where available) analogous controlled vocabulary terms. All searches were restricted to human studies.

Defining a comprehensive search strategy for CIM is challenging.24 27–29 There have been a number of efforts to develop a concise definition of CAM.26 30 This review used the one developed by the members of the Cochrane CAM Field31 and then added the terms ‘integrative’, ‘integrated’ and ‘collaborative’ medicine. The Cochrane CAM definition starts with the NCCAM definition9 and then refines it by specifically including all therapies ‘based upon the theories of a medical system outside the Western allopathic medical model’ (eg, traditional Chinese medicine and Reiki), and including others depending on the context and setting of their use. The context of use considers treatment/condition combinations and excludes those ‘currently considered to be standard treatment’, and the setting of use generally includes self-care and therapies delivered by CIM providers, but excludes therapies ‘delivered exclusively by conventionally credentialed medical personnel or exclusively within hospital settings’. Therefore, therapies such as chemotherapy regimens (eg, chronotherapy32), and therapies requiring surgical implantation (eg, neuroreflexotherapy33) or the placement of a feeding tube34 were not included even though these therapies appeared in our search. In cases where CIM therapies (eg, biofeedback or hypnosis) were included as part of a package of care (eg, with cognitive behavioural therapy), a judgement was made as to whether the CIM portion of the treatment made up half or more of the overall package of care under study. If so, the package of care was included as CIM. Because more than half of CIM users use multiple CIM therapies,35 studies of packages of therapies and coordinated care were identified as such.

Articles were categorised as full economic evaluations if they compared the costs (inputs) and consequences (economic, clinical and/or humanistic outcomes36) of two or more therapeutic alternatives applied to the same patient population (ref.37, p. 11). Otherwise, they were considered partial evaluations, for example, cost-identification or cost-comparison studies.38 Studies that estimated resource utilisation were included as economic evaluations if the utilisation data were detailed enough to allow monetary valuation.

Two reviewers (PMH and BLP) evaluated all articles for inclusion and extracted all data. Disagreements were resolved by discussion between the two review authors, or, if needed, by the other coauthors. Because the results of economic evaluations can rapidly lose relevance with time, mainly due to changes in practice patterns and cost structures, data were extracted only from the economic evaluations published 2001–2010. Extracted data were entered into an Excel template developed for a previous review.20 The type(s) of CIM evaluated and the target population were captured for all economic evaluations. Various indicators of study quality were captured for all full economic evaluations, and more detailed data and results were captured only for those full economic evaluations that met five quality criteria.

The quality of an economic evaluation can be judged along two general dimensions: (1) whether the study was a quality measure of outcomes for its target population and location—that is, whether it was internally valid; and (2) whether enough information is provided for the study's results to be transferable (‘generalisable’).39 Health outcomes are to some extent considered generalisable across settings; however, because resource availability, practice patterns and relative prices can vary greatly, economic outcomes usually are not.40 Therefore, one goal in economic evaluation is to ensure the transferability of study results—that is, to provide enough study detail so that results can be adapted (usually via modelling) to apply to other settings.39 The 35-item BMJ checklist captures components of both dimensions of quality and was applied to all full economic evaluations.41 We also chose five quality criteria by which to identify a subset of full economic evaluations to highlight as being of most interest to policy makers. These quality criteria are based on recommendations made by the US Panel on Cost Effectiveness in Health and Medicine 42 and by well-known experts in the field,37 and focus on the quality of the underlying study (the first type of quality):

  • Because cost-effectiveness analysis (CEA) is comparative, to ensure that results are useful to decision makers, one of the alternatives to which the CIM intervention was compared must be some version of commonly available (routine, standard or usual) care.
  • The analysis must explicitly or implicitly use (and include all relevant costs from) at least one recognised perspective—for example, society, third-party payer, hospital or employer.
  • Since ‘an economic evaluation of a healthcare programme is only as good as the effectiveness data it is built upon’ (ref.43, p. 232), health outcomes must be from randomised controlled trials or non-randomised controlled trials using either statistical adjustment or matching to address baseline differences.
  • Since having patient-specific data on both costs and outcomes is an advantage for internal validity,44 resource use must be a measured outcome of the study. Modelling studies utilise the results of other published studies, therefore, are exempt from this criterion.
  • Because uncertainty in an economic evaluation comes not just from sample variation, but also from assumptions made,45 a sensitivity analysis is required.

Because the prices used to value resources are highly location-specific and setting-specific,39 46 we also note, for the articles meeting the above criteria, the presence of a study reporting criterion essential for the transferability of study results (usually via modelling):39 40 separate reporting of unit costs from resource use for economic evaluations alongside trials, or from model parameters (eg, transition probabilities) for economic evaluations using models.

Other data extracted for the economic evaluations which meet the five study-quality criteria are: treatment and study duration, primary clinical and economic outcome measures, the setting in which treatment took place, study design and sample size, the type (table 1) and perspective (ie, the point of view used to define costs) of the economic analysis, and incremental cost-effectiveness of the CIM alternative compared to usual care. Incremental cost-effectiveness is reported in 2011 US$ and is calculated from reported results by first converting the study currency to US$ using the Federal Reserve annual exchange rate47 for the study's currency year and then inflated to 2011 values using the medical care component of the Consumer Price Index.48

Table 1
Types of full economic evaluations

Finally, up to three additional quality measures are included for each of these studies. The Tufts CEA Registry49 quality score is recorded when it was available (note it is only available for cost-utility analyses, CUAs). A Jadad score50 with minor modifications (the two possible points for blinding were replaced with one point for the use of a blinded assessor)51 was calculated for the economic evaluations that included a randomised trial. The percentage of the applicable items from the 35-item BMJ checklist that were met by each article is also reported.41


As shown in figure 1, the database search identified 270 published economic evaluations. An additional 68 articles were added through the bibliography and expert-supplied list search for a total of 338 economic evaluations of CIM. Of these, 204 (60%) were published from 2001 through 2010 (114 full and 90 partial economic evaluations). Of the recent full economic evaluations almost all (103, 90%) examined the effect of one CIM therapy and most of the balance (10, 9%) examined the effect of two or more CIM therapies provided by the same practitioner. Only one looked at the effect of multiple CIM therapies provided by different CIM providers.52 CIM was generally evaluated as an adjunct to usual care.

Figure 1
The flow of records and articles through the systematic review.

As shown in table 2, the 204 economic evaluations published in the past 10 years are spread across a wide range of CIM therapies applied to a number of different study populations. The biggest concentration of full economic evaluations (19 in number) pertained to the use of NCCAM's definition of manipulative and body-based practices (eg, chiropractic, osteopathic manipulation, massage, etc) for the treatment of back pain.53–72 However, even this subgroup is fairly heterogeneous in terms of the therapy (or therapies) tested and/or the type of back pain treated. Eight of these comparisons involved chiropractic care for back pain; one for chronic,53 one for acute57and six for either type.59 60 63 64 67 68 Five evaluated spinal manipulation and manual therapy provided by physiotherapists for chronic back pain (one),65 acute back pain (two)58 69 or either (two).56 68 Four involved osteopathic manipulation; one for chronic71 and one for subacute back pain72 and two for musculoskeletal conditions including back pain.66 68 Three evaluated massage; two for chronic55 62 and one for acute back pain.57 The last two studies evaluated a musculoskeletal physician (treatment ‘with a combination of manual therapy, injections, acupuncture and other pain management techniques’) for orthopaedic referrals;54 and a Finnish folk medicine practice called ‘bone setting’ for the treatment of patients with chronic back pain.61

Table 2
Types of individual complementary and integrative medicine (CIM) therapies studied for various conditions and in various populations: 2001–2010 (reported as the ratio of the total number of economic evaluations to the number of full economic evaluations) ...

Table 3 shows the results of the application of the 35-item BMJ checklist to the full economic evaluations published 2001–2010.41 On average, the number of applicable items met by each article stayed fairly constant during this period. However, the application of two key items (ie, the proper use of discounting and the inclusion of sensitivity analysis) and the disclosure of funding sources improved significantly, and reporting of the study time horizon worsened significantly. As expected, the average overall and individual-item percentages were higher for the higher-quality articles (those meeting the five study-quality criteria) and for CUAs of CIM. It is not surprising that CUA's quality is higher. They generally involve more effort than other CEAs and are required or recommended by various national guidelines.42 73–75 Nevertheless, it seems as though the quality of CUAs of CIM is generally comparable to, or slightly better than, that seen in CUAs across all medicine, at least in terms of the Tufts quality score, disclosure of funding sources and the five items where comparable data are available.76 77

Table 3
Comparison of various quality measures between economic evaluations of complementary and integrative medicine (CIM) and conventional medicine

The number of full evaluations meeting each of the five study-quality criteria are: comparison to usual care 97 (85%), all costs from a recognised perspective 96 (84%), health outcomes from a randomised or matched-control trial 86 (75%), patient-specific data on costs and outcomes 105 (92%) and sensitivity analyses 37 (32%). Sixty-two (54%) of full evaluations met the first four of these and 31 (27%) met all five. A summary of the results of these 31 higher-quality articles (covering 28 different studies) is shown in table 4.54 60 62 68 71 78–103 Twenty-two of these articles (19 of the studies) reported resource use (trials) or model parameters (models) separate from unit prices—a minimum measure of study transferability.54 62 68 71 78 80–85 87–93 95 100 101 103 For those studies which included a randomised trial, the modified Jadad scores ranged from 2 to 4 on a scale from 0 to 4. The Tufts CEA Registry quality scores for the studies containing a CUA ranged from 4 to 6.5 on a scale from 1 to 7. The percentage of the applicable items on the BMJ checklist met by these studies ranged from 66% to 97%.

Table 4
Summary of results of complementary and integrative medicine (CIM) economic evaluations that met five study-quality criteria (31 articles representing 28 studies)

Of the 56 comparisons made in these studies, 16 (29%) are cost saving—that is, the added CIM therapy had better health outcomes and lower costs than usual care alone. Cost savings were seen for acupuncture alone (instructional visits with an acupuncturist followed by home self-care by the partner for pregnant women with breech presentations at 33 weeks in terms of reductions in both breech presentation at birth and ceasareans in the Netherlands,91 and treatment by traditional Chinese medicine-trained licensed acupuncturists in private acupuncture clinics in the UK for low-back pain in terms of quality-adjusted life-years or QALYs from the societal perspective85) and in combination with other therapies (along with manual therapy, injections and other pain management for patients referred to an orthopaedic surgeon's office in Scotland who were unlikely to need surgery in terms of both improvements in health-related quality of life and QALYs54). Cost savings were also seen for manual therapy delivered by a physiotherapist, who is also a registered manual therapist, for neck pain in terms of perceived recovery, pain, neck disability and QALYs82; for preoperative oral supplementation with arginine and ω-3 fatty acids for patients with gastrointestinal cancer undergoing surgery102; for vitamin K1 supplementation for postmenopausal women with osteopenia and osteoporosis in terms of QALYs103; for supplementation with vitamins C and E and β-carotene for cataract prevention90; for fish oil supplementation in men with a history of heart attack87; for tai chi to prevent hip fractures in nursing home residents95 and for naturopathic care offered through a worksite clinic for chronic low-back pain in terms of both reductions in absenteeism and gains in QALYs.80

Of the 28 cost-utility comparisons, one (massage for low-back pain62) was dominated— that is, had worse health outcomes and higher costs than usual care. Five (18%) are cost saving,54 80 82 85 103 5 (18%) have incremental cost-effectiveness ratios (ICERs) between US$0 and US$10 000 per quality-adjusted life-year (QALY),68 71 81 85 97 and 89% had ICERs less than US$50 000/QALY, a threshold often considered to represent the upper limit of society's value for a QALY.104 The cost-saving cost-utility studies were included in the paragraph above (ie, those that mention QALYs). The studies with cost-utility ICERs between US$0 and US$10 000 per QALY were: treatment by traditional Chinese medicine-trained licensed acupuncturists in private acupuncture clinics in the UK for low-back pain.85 hospital-based acupuncture by licensed oriental medical doctors in South Korea for 60-year-old women with first-time acute low-back pain,81 acupuncture from physicians with at least 140 h of training (A-diploma) in Germany for patients with dysmenorrhoea,97 osteopathic spinal manipulation by a general practitioner who is a registered osteopath in the UK for patients with subacute back pain,71 and an exercise programme plus spinal manipulation from a chiropractor, osteopath or physiotherapist at a private or National Health Service (NHS) site in the UK for low-back pain.68 The average percentage of applicable BMJ checklist items met by each study was slightly lower for those studies with at least one cost-saving comparison (85% vs 88%), but the difference was not statistically significant (t test=0.75, p value=0.460).


This comprehensive systematic review identified 338 economic evaluations of CIM; 204 of which were published recently (2001–2010) covering a wide range of CIM therapies for a variety of populations. Although most patients who use CIM use more than one modality35 and despite the attention given to integrative medicine (coordinated access to conventional medicine and CIM),105 this systematic review found only one study that examined the effects of use of multiple CIM practitioners.52 In general, the quality of the recent full economic evaluations has held constant and is in line with what is seen in economic evaluations in conventional medicine. Details of the 31 recent higher-quality full economic evaluations indicate potential cost-effectiveness and cost savings across a variety of CIM therapies applied to different conditions. Owing to the non-generalisable nature of economic evaluations, the cost estimates shown are specific to their study settings.40 However, 22 articles provided at least the minimum information for study transferability. Therefore, their results could be adapted via modelling to determine the economic impact of these interventions in other settings.

The strengths of this study are the comprehensive search strategy, which revealed a substantial number of published economic evaluations of CIM, the use of two reviewers and the use of multiple measures of study quality. Higher-quality studies were identified and highlighted for policy makers using a simple objective list of quality criteria, which reduced the potential for bias. The weaknesses of this study are similar to those of the other systematic reviews. The reviewers were not blinded to journals and article authors, which may have influenced results. Also, some aspects of what makes a quality economic evaluation could not be judged from what was reported. For example, ideally, pragmatic trials enrol patients typical of normal caseload in typical settings with typically trained and experienced practitioners following them under routine conditions (ref.37, p. 251). Judgements as to whether these criteria were met were not always possible from the reports, and were beyond the scope of this review. Finally, publication bias was not assessed. However, since the major goal of this study was to establish the extent of the published literature on this topic and to highlight the results of the higher-quality studies, it is not clear that publication bias is relevant here.

The number of economic evaluations of CIM found and reviewed by this study far exceeds the numbers found in previous studies.11 19–21 23 This study found a total of 338 economic evaluations of CIM published between and including 1979 and 2010; 211 of these were full economic evaluations. White and Ernst19 identified 34 economic evaluations of CAM published 1987–1999; 11 of which were full economic evaluations. Between 1999 and October 2004, Herman et al20 identified 56 economic evaluations of CAM (39 full evaluations). Between 1994 and May 2004 Hulme and Long21 identified 19 full economic evaluations of CAM, and over a similar period (1995–2007) Doran et al23 found 43 economic evaluations (15 full evaluations). Maxion-Bergemann et al11 identified 5 (1 full) economic evaluations over an unspecified search period. The large number of economic evaluations found in this study reflects the facts that: (1) all evaluations from previous reviews were included; (2) a number of studies have been published since the last search dates of prior reviews and (3) a more extensive search strategy was used. It should be noted that 20% of the articles (68 of 338) in this review were identified through bibliography searches and from expert lists. Therefore, even the application of a long list of search terms to multiple databases does not guarantee that all CIM studies will be identified. However, there is some evidence that the indexing of these articles in medical databases is improving; studies from bibliographies and expert lists made up 32% of found articles published 2000 and before, but only 12% recent articles.

There are several implications of this study for policy makers, clinicians and future researchers. First, there is a large and growing literature of quality economic evaluations in CIM. However, although indexing is improving in databases, finding these studies can require going beyond simple CIM-related search terms. Second, the results of the higher-quality studies indicate a number of highly cost-effective, and even cost saving, CIM therapies. Almost 30% of the 56 cost-effectiveness, cost-utility and cost-benefit comparisons shown in table 4 (18% of the CUA comparisons) were cost saving. Compare this to 9% of 1433 CUA comparisons found to be cost saving in a large review of economic evaluations across all medicine.106 Third, by meeting the five study-quality criteria, the studies shown in table 4 can each be considered a reasonable indicator of the health and economic impacts of the CIM therapy studied, at least in that population and setting. These studies, especially those showing cost savings, should be considered further for applicability in other settings. This requires the study to be transferable.39 Fortunately, the majority of the higher-quality studies met our measure of study transferability—resource use or model parameters, and unit costs were reported separately.

Given the substantial number of economic evaluations of CIM found in this comprehensive review, even though it can always be said that more studies are needed, what is actually needed are better-quality studies—both in terms of better study quality (to increase the validity of the results for its targeted population and setting) and better transferability (to increase the usefulness of these results to other decision makers in other settings). Therefore, the following recommendations are made.

  1. That all studies measuring the effectiveness of CIM at least consider also measuring input costs and economic outcomes.
  2. That at least one arm of the study be some version of commonly available (usual) care, and that usual care and all interventions studied be described in sufficient detail that decision makers in other settings can determine what was done and whether the study's usual care comparator is applicable in their setting.
  3. That consideration be given to how CIM is typically used (eg, multiple CIM therapies) or can be used (eg, coordinated integrative care models) when designing studies.
  4. That changes in resource use be reported separately from unit costs in economic evaluations alongside clinical trials and that model parameters and unit costs be clearly reported in decision-analytic modelling studies.
  5. That all economic evaluations contain sensitivity analyses to increase the reliability of results.
  6. That more consideration be given to modelling as a method to estimate economic outcomes for existing effectiveness trial results, and to generalise existing quality economic evaluation results to other jurisdictions.

Supplementary Material

Author's manuscript:
Reviewer comments:


The authors wish to acknowledge and most gratefully thank Sandy Kramer of the University of Arizona Health Sciences Library for her assistance in the development and application of the search strategy and for eliminating duplicates from the search results. We would also like to thank Robert Scholten and P Scott Lapinski of the Harvard Medical School for their assistance with the EMBASE searches.


Contributions: PMH conceived of the idea for the paper, designed the search strategy, reviewed the references found, extracted the data from each included article and is the guarantor for this study. In parallel, BLP also reviewed the references found, extracted data from included articles and worked with PMH to resolve any discrepancies between reviewers. CMW provided practical insight and an international perspective to the design of the paper and interpretation of results. DME participated in the early design of the study, including the data extraction plan, inclusion/exclusion criteria and the interpretation of results. All authors contributed to the drafting and editing of the manuscript.

Funding: The Bernard Osher Foundation supports a portion of DME's time for research in integrative medicine. The Foundation had no control or influence over the design or execution of this study, nor no input into this manuscript.

Competing interests: None.

Provenance and peer review: Not commissioned; externally peer reviewed.

Data sharing statement: The full list of found articles is available in a word document from the corresponding author.


1. Eisenberg DM, Kessler RC, Foster C, et al. Unconventional medicine in the United States: prevalence, costs, and patterns of use. N Engl J Med 1993;328:246–52 [PubMed]
2. Eisenberg DM, Davis RB, Ettner SL, et al. Trends in alternative medicine use in the United States, 1990–1997. JAMA 1998;280:1569–75 [PubMed]
3. Barnes PM, Powell-Griner E, McFann K, et al. Complementary and alternative medicine use among adults: United States, 2002. Advance data from Vital and Health Statistics. Hyattsville, MA: National Center for Health Statistics, 2004 [PubMed]
4. Nahin RL, Barnes PM, Stussman BJ, et al. Costs of complementary and alternative medicine (CAM) and frequency of visits to CAM practitioners: United States, 2007. National Health Statistics Reports. Hyattsville, MA: National Center for Health Statistics, 2009 [PubMed]
5. MacLennan AH, Wilson DH, Taylor AW. The escalating cost and prevalence of alternative medicine. Prev Med 2002;35:166–73 [PubMed]
6. Thomas KJ, Nicholl JP, Coleman P. Use and expenditure on complementary medicine in England: a population based survey. Complement Ther Med 2001;9:2–11 [PubMed]
7. Wolf U, Maxion-Bergemann S, Bornhoft G, et al. Use of complementary medicine in Switzerland. Forsch Komplementmed 2006;13:4–6 [PubMed]
8. Hartel U, Volger E. Use and acceptance of classical natural and alternative medicine in Germany—findings of a representative population-based survey. Forsch Komplementmed 2004;11:327–34 [PubMed]
9. National Center for Complementary and Alternative Medicine What is complementary and alternative medicine (CAM)? National Center for Complementary and Alternative Medicine. Bethesda, Maryland: National Institutes of Health, 2011
10. Claxton K, Posnett J. An economic approach to clinical trial design and research priority-setting. Med Econ 1996;5:513–24 [PubMed]
11. Maxion-Bergemann S, Wolf M, Bornhoft G, et al. Complementary and alternative medicine costs—a systematic literature review. Forsch Komplementmed 2006;13(Suppl 2):42–5 [PubMed]
12. van der Roer N, Goossens MEJB, Evers SMAA, et al. What is the most cost-effective treatment for patients with low back pain? A systematic review. Best Pract Res Clin Rheumatol 2005;19:671–84 [PubMed]
13. Branson RA. Cost comparison of chiropractic and medical treatment of common musculoskeletal disorders: a review of the literature after 1980. Top Clin Chiropractic 1999;6:57–68
14. Solomon DH, Bates DW, Panush RS, et al. Costs, outcomes, and patient satisfaction by provider type for patients with rheumatic and musculoskeletal conditions: a critical review of the literature and proposed methodological standards. Ann Intern Med 1997;127:52–60 [PubMed]
15. Kennedy DA, Hart J, Seely D. Cost-effectiveness of natural health products: a systematic review of randomized clinical trials. Evid-Based Complement Altern Med 2009;6:297–304 [PMC free article] [PubMed]
16. Gamber R, Holland S, Russo DP, et al. Cost-effective osteopathic manipulative medicine: a literature review of cost-effectiveness analyses for osteopathic manipulative treatment. J Am Osteopathic Assoc 2005;105:357–67 [PubMed]
17. Bornhoft G, Wolf U, Von Ammon K, et al. Effectiveness, safety and cost-effectiveness of homeopathy in general practice—summarized health technology assessment. Forsch Komplementmed 2006;13:19–29 [PubMed]
18. Schneider CJ. Cost effectiveness of biofeedback and behavioral medicine treatments: a review of the literature. Biofeedback Self Regul 1987;12:71–92 [PubMed]
19. White AR, Ernst E. Economic analysis of complementary medicine: a systematic review. Complement Ther Med 2000;8:111–18 [PubMed]
20. Herman PM, Craig BM, Caspi O. Is complementary and alternative medicine (CAM) cost-effective? A systematic review. BMC Complement Altern Med 2005;5:11. [PMC free article] [PubMed]
21. Hulme C, Long AF. Square pegs and round holes? A review of economic evaluation in complementary and alternative medicine. J Altern Complement Med 2005;11:179–88 [PubMed]
22. Canter PH, Coon JT, Ernst E. Cost-effectiveness of complementary therapies in the United Kingdom—a systematic review. Evid-Based Complement Altern Med 2006;3:425–32 [PMC free article] [PubMed]
23. Doran CM, Chang DH-T, Kiat H, et al. Review of economic methods used in complementary medicine. J Altern Complement Med 2010;16:591–5 [PubMed]
24. Pilkington K. Searching for CAM evidence: an evaluation of therapy-specific search strategies. J Altern Complement Med 2007;13:451–9 [PubMed]
25. Shekelle PG, Morton SC, Suttorp MJ, et al. Challenges in systematic reviews of complementary and alternative medicine topics. Ann Intern Med 2005;142:1042–7 [PubMed]
26. Wootton JC. Classifying and defining complementary and alternative medicine. J Altern Complement Med 2005;11:777–8 [PubMed]
27. Pilkington K, Richardson J. Exploring the evidence: the challenges of searching for research on acupuncture. J Altern Complement Med 2004;10:587–90 [PubMed]
28. Boddy K, Younger P. What a difference an interface makes: just how reliable are your search results? Focus Altern Complement Ther 2009;14:5–7
29. Murphy LS, Reinsch S, Najm WI, et al. Spinal palpation: the challenges of information retrieval using available databases. J Manipulative Physiol Ther 2003;26:374–82 [PubMed]
30. Furnham A. How the public classify complementary medicine: a factor analytic study. Complement Ther Med 2000;8:82–7 [PubMed]
31. Wieland LS, Manheimer E, Berman BM. Development and classification of an operational definition of complementary and alternative medicine for the Cochrane Collaboration. Altern Ther Health Med 2011;17:50–9 [PMC free article] [PubMed]
32. Focan C. Pharmaco-economic comparative evaluation of combination chronotherapy vs. standard chemotherapy for colorectal cancer. Chronobiol Int 2002;19:289–97 [PubMed]
33. Kovacs FM, Llobera J, Abraira V, et al. Effectiveness and cost-effectiveness analysis of neuroreflexotherapy for subacute and chronic low back pain in routine general practice: a cluster randomized, controlled trial. Spine 2002;27:1149–59 [PubMed]
34. Senkal M, Zumtobel V, Bauer KH, et al. Outcome and cost-effectiveness of perioperative enteral immunonutrition in patients undergoing elective upper gastrointestinal tract surgery: a prospective randomized study. Arch Surg 1999;134:1309–16 [PubMed]
35. Wolsko PM, Eisenberg DM, Davis RB, et al. Insurance coverage, medical conditions, and visits to alternative medicine providers. Arch Intern Med 2002;162:281–7 [PubMed]
36. Gunter MJ. The role of the ECHO model in outcomes research and clinical practice improvement. Am J Manag Care 1999;5:S217–24 [PubMed]
37. Drummond MF, Sculpher MJ, Torrance GW, et al. Methods for the economic evaluation of health care programmes. 3rd edn. Oxford: Oxford University Press, 2005
38. Berger ML, Bingefors K, Hedblom E, et al. Health care cost, quality, and outcomes: ISPOR book of terms. Lawrenceville, NJ: International Society for Pharmacoeconomics and Outcomes Research, 2003
39. Drummond M, Barbieri M, Cook J, et al. Transferability of economic evaluations across jurisdictions: ISPOR good research practices task force report. Value Health 2009;12:409–18 [PubMed]
40. Drummond M, Manca A, Sculpher M. Increasing the generalizability of economic evaluations: recommendations for the design, analysis, and reporting of studies. Int J Technol Assess Health Care 2005;21:165–71 [PubMed]
41. Drummond MF, Jefferson TO., BMJ Economic Evaluation Working Party Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ 1996;313:275–83 [PMC free article] [PubMed]
42. Gold MR, Siegel JE, Russell LB, et al. Cost-effectiveness in health and medicine. New York: Oxford University Press, 1996
43. Drummond MF, O'Brien B, Stoddart GL, et al. Methods for the economic evaluation of health care programmes. 2nd edn. Oxford: Oxford University Press, 1997
44. Marshall DA, Hux M. Design and analysis issues for economic analysis alongside clinical trials. Med Care 2009;47:814–20 [PubMed]
45. Briggs A, Sculpher M, Buxton M. Uncertainty in the economic evaluation of health care technologies: the role of sensitivity analysis. Health Econ 1994;3:95–104 [PubMed]
46. Sculpher MJ, Pang FS, Manca A, et al. Generalisability in economic evaluation studies in healthcare: a review and case studies. Health Technol Assess 2004;8:1–213 [PubMed]
47. Board of Governors of the Federal Reserve System Foreign Exchange Rates – G.5A, 1997–2012
48. Bureau of Labor Statistics Archived consumer price index detailed report information, 2000–2011
49. Center for the Evaluation of Value and Risk in Health Cost-effectiveness analysis registry: Institute for Clinical Research and Health Policy Studies. Tufts Medical Center is in Boston, MA, 2011
50. Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996;17:1–12 [PubMed]
51. White AR, Ernst E. A systematic review of randomized controlled trials of acupuncture for neck pain. Rheumatology 1999;38:143–7 [PubMed]
52. Robinson N, Donaldson J, Watt H. Auditing outcomes and costs of integrated complementary medicine provision–the importance of length of follow up. Complement Ther Clin Pract 2006;12:249–57 [PubMed]
53. Almog G, Lamond PJ, Gosselin G. Effects of chiropractic care on spinal symptomatology among professional drivers. Clin Chiropractic 2004;7:114–19
54. Brown APL, Kennedy ADM, Torgerson DJ, et al. The OMENS trial: opportunistic evaluation of musculo-skeletal physician care among orthopaedic outpatients unlikely to require surgery. Health Bull 2001;59:199–210 [PubMed]
55. Cherkin DC, Eisenberg DM, Sherman KJ, et al. Randomized trial comparing traditional Chinese medical acupuncture, therapeutic massage, and self-care education for chronic low back pain. Arch Intern Med 2001;161:1081–8 [PubMed]
56. Cook C, Cook A, Worrell T. Manual therapy provided by physical therapists in a hospital-based setting: a retrospective analysis. J Manipulative Physiol Ther 2008;35:338–43 [PubMed]
57. Eisenberg DM, Post DE, Davis RB, et al. Addition of choice of complementary therapies to usual care for acute low back pain: a randomized controlled trial. Spine 2007;32:151–8 [PubMed]
58. Fritz JM, Brennan GP, Leaman H. Does the evidence for spinal manipulation translate into better outcomes in routine clinical care for patients with occupational low back pain? A case-control study. Spine J 2006;6:289–95 [PubMed]
59. Grieves B, Menke JM, Pursel KJ. Cost minimization analysis of low back pain claims data for chiropractic vs medicine in a managed care organization. J Manipulative Physiol Ther 2009;32:734–9 [PubMed]
60. Haas M, Sharma R, Stano M. Cost-effectiveness of medical and chiropractic care for acute and chronic low back pain. J Manipulative Physiol Ther 2005;28:555–63 [PubMed]
61. Hemmila HM. Quality of life and cost of care of back pain patients in Finnish general practice. Spine 2002;27:647–53 [PubMed]
62. Hollinghurst S, Sharp D, Ballard K, et al. Randomised controlled trial of Alexander technique lessons, exercise, and massage (ATEAM) for chronic and recurrent back pain: economic evaluation. BMJ 2008;337:a2656. [PMC free article] [PubMed]
63. Hurwitz EL, Morgenstern H, Harber P, et al. The effectiveness of physical modalities among patients with low back pain randomized to chiropractic care: findings from the UCLA low back pain study. J Manipulative Physiol Ther 2002;25:10–20 [PubMed]
64. Kominski GF, Heslin KC, Morgenstern H, et al. Economic evaluation of four treatments for low-back pain: results from a randomized controlled trial. Med Care 2005;43:428–35 [PubMed]
65. Lewis JS, Hewitt JS, Billington L, et al. A randomized clinical trial comparing two physiotherapy interventions for chronic low back pain. Spine 2005;30:711–21 [PubMed]
66. Lipton JA, Meneses P, Martin JB, et al. Improved pain score outcomes achieved through the cooperative and cost-effective use of physical (osteopathic manipulative) medicine in the treatment of outpatient musculoskeletal complaints. Am Acad Osteopathy J 2002;12:26–32
67. Stano M, Haas M, Goldberg B, et al. Chiropractic and medical care costs of low back care: results from a practice-based observational study. Am J Manag Care 2002;8:802–9 [PubMed]
68. UK Beam Trial Team United Kingdom back pain exercise and manipulation (UK BEAM) randomised trial: cost effectiveness of physical treatments for back pain in primary care. BMJ 2004;329:1381. [PMC free article] [PubMed]
69. Whitehurst DGT, Lewis M, Yao GL, et al. A brief pain management program compared with physical therapy for low back pain: results from an economic analysis alongside a randomized clinical trial. Arthritis Care Res 2007;57:466–73 [PubMed]
70. Wilkey A, Gregory M, Byfield D, et al. A comparison between chiropractic management and pain clinic management for chronic low-back pain in a National Health Service outpatient clinic. J Altern Complement Med 2008;14:465–73 [PubMed]
71. Williams NH, Edwards RT, Linck P, et al. Cost-utility analysis of osteopathy in primary care: results from a pragmatic randomized controlled trial. Fam Pract 2004;21:643–50 [PubMed]
72. Williams NH, Wilkinson C, Russell I, et al. Randomized osteopathic manipulation study (ROMANS): pragmatic trial for spinal pain in primary care. Fam Pract 2003;20:662–9 [PubMed]
73. National Institute for Health and Clinical Excellence Assessing cost-effectiveness. The guidelines manual. London: National Health Service, 2009:81–91
74. Commonwealth Department of Health and Ageing Guidelines for the pharmaceutical industry on preparaion of submissions to the pharmaceutical benefits advisory committee. Canberra: Commonwealth of Australia, 2002
75. Glennie JL, Torrance GW, Baladi JF, et al. The revised Canadian guidelines for the economic evaluation of pharmaceuticals. Pharmacoeconomics 1999;15:459–68 [PubMed]
76. Neumann PJ, Greenberg D, Olchanski NV, et al. Growth and quality of the cost-utility literature, 1976–2001. Value Health 2005;8:3–9 [PubMed]
77. Neumann PJ. Costing and perspective in published cost-effectiveness analysis. Med Care 2009;47:S28–32 [PubMed]
78. Black C, Clar C, Henderson R, et al. The clinical effectiveness of glucosamine and chondroitin supplements in slowing or arresting progression of osteoarthritis of the knee: a systematic review and economic evaluation. Health Technol Assess 2009;13:1–148 [PubMed]
79. Franzosi MG, Brunetti M, Marchioli R, et al. Cost-effectiveness analysis of n-3 polyunsaturated fatty acides (PUFA) after myocardial infarction: results from Gruppo Italiano per lo Studio della Sopravvivenza nell'Infarto (GISSI)—Prevenzione Trial. Pharmacoeconomics 2001;19:411–20 [PubMed]
80. Herman PM, Szczurko O, Cooley K, et al. Cost-effectiveness of naturopathic care for chronic low back pain. Altern Ther Health Med 2008;14:32–9 [PubMed]
81. Kim N, Yang B, Lee T, et al. An economic analysis of usual care and acupuncture collaborative treatment on chronic low back pain: a Markov model decision analysis. BMC Complement Altern Med 2010;10:74. [PMC free article] [PubMed]
82. Korthals-de Bos IB, Hoving JL, van Tulder MW, et al. Cost effectiveness of physiotherapy, manual therapy, and general practitioner care for neck pain: economic evaluation alongside a randomised controlled trial. BMJ 2003;326:911. [PMC free article] [PubMed]
83. Lamotte M, Annemans L, Kawalec P, et al. A multi-country health economic evaluation of highly concentrated n-3 polyunsaturated fatty acids in secondary prevention after myocardial infarction. Pharmacoeconomics 2006;24:783–95 [PubMed]
84. Quilici S, Martin M, McGuire A, et al. A cost-effectiveness analysis of n−3 PUFA (Omacor®) treatment in post-MI patients. Int J Clin Pract 2006;60:922–32 [PubMed]
85. Ratcliffe J, Thomas KJ, MacPherson H, et al. A randomised controlled trial of acupuncture care for persistent low back pain: cost effectiveness analysis. BMJ 2006;333:626. [PMC free article] [PubMed]
86. Reinhold T, Witt CM, Jena S, et al. Quality of life and cost-effectiveness of acupuncture treatment in patients with osteoarthritis pain. Eur J Health Econ 2008;9:209–19 [PubMed]
87. Schmier JK, Rachman NJ, Halpern MT. The cost-effectiveness of omega-3 supplements for prevention of secondary coronary events. Manag Care 2006;15:43–50 [PubMed]
88. Stevenson M, Lloyd-Jones M, Papaioannou D. Vitamin K to prevent fractures in older women: systematic review and economic evaluation. Health Technol Assess 2009;13:1–134 [PubMed]
89. Thomas KJ, MacPherson H, Ratcliffe J, et al. Longer term clinical and economic benefits of offering acupuncture care to patients with chronic low back pain. Health Technol Assess 2005; 9:1–109 [PubMed]
90. Trevithick JR, Massel D, Robertson JM, et al. Modeling savings from prophylactic REACT antioxidant use among a cohort initially aged 50–55 years: a Canadian perspective. J Orthomolecular Med 2006;21:212–20
91. van den Berg I, Kaandorp GC, Bosch JL, et al. Cost-effectiveness of breech version by acupuncture-type interventions on BL 67, including moxibustion, for women with a breech foetus at 33 weeks gestation: a modelling approach. Complement Ther Med 2010;18:67–77 [PubMed]
92. Van Tubergen A, Boonen A, Landewe R, et al. Cost effectiveness of combined spa-exercise therapy in ankylosing spondylitis: a randomized controlled trial. Arthritis Rheum 2002;47:459–67 [PubMed]
93. Vickers AJ, Rees RW, Zollman CE, et al. Acupuncture of chronic headache disorders in primary care: randomised controlled trial and economic analysis. Health Technol Assess 2004;8:1–35 [PubMed]
94. Willich SN, Reinhold T, Selim D, et al. Cost-effectiveness of acupuncture treatment in patients with chronic neck pain. Pain 2006;125:107–13 [PubMed]
95. Wilson CJ, Datta SK. Tai chi for the prevention of fractures in a nursing home population: an economic analysis. J Clin Outcomes Manag 2001;8:19–27
96. Witt CM, Jena S, Selim D, et al. Pragmatic randomized trial evaluating the clinical and economic effectiveness of acupuncture for chronic low back pain. Am J Epidemiol 2006;165:487–96 [PubMed]
97. Witt CM, Reinhold T, Brinkhaus B, et al. Acupuncture in patients with dysmenorrhea: a randomized study on clinical effectiveness and cost-effectiveness in usual care. Am J Obstet Gynecol 2008;198:166.e1–8 [PubMed]
98. Witt CM, Reinhold T, Jena S, et al. Cost-effectiveness of acupuncture in women and men with allergic rhinitis: a randomized controlled study in usual care. Am J Epidemiol 2009;169:562–71 [PubMed]
99. Witt CM, Reinhold T, Jena S, et al. Cost-effectiveness of acupuncture treatment in patients with headache. Cephalalgia 2008;28:334–45 [PubMed]
100. Wonderling D, Vickers AJ, Grieve R, et al. Cost effectiveness analysis of a randomised trial of acupuncture for chronic headache in primary care. BMJ 2004;328:747. [PMC free article] [PubMed]
101. Zijlstra TR, Braakman-Jansen LM, Taal E, et al. Cost-effectiveness of spa treatment for fibromyalgia: general health improvement is not for free. Rheumatol 2007;46:1454–9 [PubMed]
102. Braga M, Gianotti L, Vignali A, et al. Hospital resources consumed for surgical morbidity: effects of preoperative arginine and omega-3 fatty acid supplementation on costs. Nutrition 2005; 21:1078–86 [PubMed]
103. Stevenson MD, Jones ML. The cost effectiveness of a randomized controlled trial to establish the relative efficacy of vitamin K1 compared with alendronate. Med Decis Making 2010;31:43–52 [PubMed]
104. Grosse SD. Assessing cost-effectiveness in healthcare: history of the $50,000 per QALY threshold. Expert Rev Pharmacoecon Outcomes Res 2008;8:165–78 [PubMed]
105. Schultz AM, Chao SM, McGinnis JM, editors. , eds. Integrative medicine and the health of the public: a summary of the February 2009 Summit. Washington DC: Institute of Medicine, 2009
106. Bell CM, Urbach DR, Ray JG, et al. Bias in published cost effectiveness studies: systematic review. BMJ 2006;332:699–703 [PMC free article] [PubMed]

Articles from BMJ Open are provided here courtesy of BMJ Publishing Group