A possible, efficient solution to filling some of these gaps is through the use of comparative effectiveness research (CER). CER attempts to generate answers to relevant clinical questions through a systematic review of pertinent published studies, retrospective analysis of patient information available in large databases, or implementation of prospective effectiveness studies (1
). Examples of clinically relevant questions that can be answered by CER include effectiveness and tolerability of older versus newer antiepileptic drugs (AED) and the use of generic AED products compared with branded products. Although data from CER are helpful in answering some clinical questions, this type of research is limited by various issues with validity, confounding, and bias (2–4
The U.S. Congress recognized the potential value of CER by including $1.1 billion in the American Recovery and Reinvestment Act of 2009 to support it (3
). The majority of these funds were directed toward the Agency for Healthcare Research and Quality (AHRQ), which has aggressively initiated a variety of CER projects. However, Congress also realized potential problems with unfettered use of results from CER by specifically prohibiting data from these studies to be used for coverage determinations, provider reimbursement decisions, or other policy decisions for public and private payers. This prohibition does not prevent third-party payers from using these CER data as part of the decision-making process on coverage policies. This approach by Congress was an effort to recognize the value of CER but also to recognize the pitfalls in uniformly applying CER results to policy.
With the proliferation of new AED, a question of increasing importance is how newer AED compare with older AED in the treatment of epilepsy. The lack of definitive, prospective, randomized trials dealing with these questions made these issues a prime target for CER. AHRQ concluded this was an important issue and initiated a CER study in 2010 (5
). The CER on AED aimed to answer four key questions through a systematic review of published literature. The key questions to be addressed were as follows:
- In patients with epilepsy, what is the comparative effectiveness/efficacy of AED on health outcomes (e.g., mortality, hospitalizations, office/emergency department visits, composite endpoint of health resource utilization, health-related quality of life, seizures, secondary seizure injury, status epilepticus, loss of driver's license, loss of employment)?
- In patients with epilepsy, what is the comparative effectiveness/efficacy of AED on intermediate outcomes (e.g., pharmacokinetics, comparative dose of medication needed to control seizures, switchback rates)?
- In patients with epilepsy, what is the comparative impact of AED on serious adverse events (e.g., neurologic adverse effects, hypotension, rash, suicidal ideation, mood and cognition, bone density, and cosmetic adverse effects)?
- In patients with epilepsy, what are the comparative benefits or harms for AED in subgroups of patients (e.g., seizure etiology, gender, ethnicity, patient age, patient pharmacogenetic profile, types of antiepileptic medication) (5)?
The issue of generic substitution of AED was also included in this project. Following a period of public comment and input from an expert panel that included several members of the American Epilepsy Society (AES), minor adjustments to the questions were made and the analyses were launched. Prior to publication, the report was reviewed by another expert panel composed of individuals who are AES members. Results of this CER were published and released in December 2011 (6
Obviously, a CER of this size and magnitude is highly complex and detailed. However, the basic conclusions of the study were as follows:
- Carbamazepine, although associated with more adverse effects, had advantages over newer AED as a class in regards to seizure control.
- Phenytoin and valproate had increased adverse effects compared with newer AED but were equivalent with newer AED in seizure control.
- Although adverse effects were more common with carbamazepine, phenytoin, and valproate compared with newer AED, this did not translate into a difference in the risk of withdrawing from a medication.
- There was insufficient or low strength of evidence to suggest that generic substitution of AED is problematic, even though the short-term use of healthcare resources may increase with substitution (6).
The final AHRQ report acknowledged that the findings are primarily limited by heterogeneity. This included not distinguishing between epilepsy types and putting all AED into older or newer categories (6
). In addition, not every possible endpoint was included in the analysis. Several older AED were not included in the final analysis. However, data tables did include information on the comparison of every older AED with the newer AED as a group. The authors also acknowledge that these results only provide insights into population-wide benefits and harms but cannot be applied to individual patient decisions. Given major gaps in published literature, the report suggests strategies for future studies.
Despite the fact that the limitations of this CER and its conclusions are clearly stated in the AHRQ report, there are serious potential dangers in the release of this study that can negatively impact patient care. A simplistic reading of the conclusions could lead to formulary restrictions requiring the use of carbamazepine, phenytoin, or valproate prior to the use of any newer AED. Anecdotal reports from epilepsy programs indicate some patients are having coverage denied for newer AED and certain dosage forms of older AED, indicating that third-party payers may be implementing various forms of stepped therapy.
With this in mind, the AES Treatments Committee and Board of Directors have evaluated the full report and express great concern about the methodology and thus the results reported in this CER (7
). In addition to the limitations noted by AHRQ, several other items need to be highlighted as problems with this report. Other than a few sections of the report, the level of evidence for conclusions were consistently judged by the AHRQ study team as being of low level and poor quality. Inadequate data prevented careful analysis of important and specific questions, such as differences in response to older or newer AED for various seizure types or consideration of the interplay between specific patient groups and selection of an AED, such as women with epilepsy or children with epilepsy. Many of the endpoints used in the final analysis were not clinically relevant for individual patients, and some of the data were incorrectly reported in the tables. The authors, although acknowledging problems with simply grouping AED into old or new categories, state they believe the differences between AED in these groups are minor and inconsequential for the purposes of this study. In reality, there are major differences in pharmacokinetics, adverse effect profiles, and other properties of these drugs that are more clinically important than their date of entry into the market. A large portion of the effectiveness analysis focused on gabapentin and vigabatrin, two drugs that have either been demonstrated to have lower efficacy or are not used as drugs of choice in treating new-onset epilepsy (6
). Finally, the CER analysis gave very little emphasis to the occurrence of adverse effects with various drugs, especially adverse effects that can be life-threatening such as rash or hepatotoxicity.
The results from this CER are in agreement with more global assessments of comparative effectiveness, including a meta-analysis by Löscher and Schmidt and a CER done by the United Kingdom National Health Service (NHS) (8
). The AHRQ and NHS studies lumped all drugs into an older or newer category. These studies did not specifically differentiate between seizure types or groups of patients. All of these studies, including the AHRQ CER, specifically noted the poor quality of data and limited studies available upon which to base their conclusions.
It is also important to note that findings of this CER differ from results in other CER reports that narrowed the focus of their investigation by specific seizure types or age groups. For example, an attempt to develop evidence-based guidelines for initial monotherapy of seizures concluded there was insufficient data to develop the guideline (10
). Arif and colleagues demonstrated lamotrigine to be the most effective AED in older adults, using a similar study design as the AHRQ CER study (11
). A prospective comparative effectiveness study by Marson et al. comparing carbamazepine, gabapentin, lamotrigine, oxcarbazepine, or topiramate, demonstrated lamotrigine to be more effective than carbamazepine for initial treatment of partial seizures (12
). The Arif and Marson studies clearly raise questions about the advisability of lumping drugs, seizure types, or various ages of patients together in a CER study.
Given the multiple, serious limitations of these results and the methodologic problems as noted by the AHRQ study team and AES review of the report, and major differences in results between this study and previously reported CER studies, the AHRQ data should not be used in making any decisions about individual patient care, formulary restrictions, or reimbursement and coverage policies. Information from this CER can be helpful in determining future research initiatives and clearly demonstrates major gaps present in the scientific support for much of current practice. However, there is insufficient information from the AHRQ CER to support major revisions in approaches to the pharmacotherapy of epilepsy or reimbursement policies for AED. Although CER is a useful tool in helping to find answers to perplexing clinical questions, the decision about which drug is best for an individual patient or groups of patients cannot be decided by CER alone.