This value-of-research analysis suggests that CATIE has important limitations for guiding public policy concerning coverage of antipsychotic medications. It also suggests that future research on this important class of medications is likely to be of immense value—exceeding $300 billion—to people with schizophrenia today and those who will develop it over the next twenty years. Much more of the value of research appears to be generated by uncertainty about the effects of treatments on costs than by uncertainty in outcomes, and very large sample sizes will be needed to address such uncertainty.
CATIE not intended to determine coverage policy
Although we identified a number of important limitations of CATIE from the technical and clinical perspectives, among the most important from a policy perspective is that it was never intended to determine coverage policy. Indeed, the high rate of medication switching in CATIE implies that policy decisions based on CATIE’s findings are relevant only in contexts that allow for effective switching. This is important because if comparative effectiveness research is motivated by attempts to inform payment policy, then it must be able to address these sorts of concerns. This suggests that if comparative effectiveness research is to be used to guide policy decisions, such as changes in coverage policy, then such research should include direct evaluations of policy changes.
Such analyses could include observational studies of policy changes or large-scale social experiments in which patients are randomly assigned to different forms of coverage, of which there are some examples in health care (such as the RAND Health Insurance Experiment). Social experiments could also be designed to work at higher levels—for example, if states are assigned different treatments.
The importance of considering social experiments as an important part of comparative effectiveness research is increased by our findings about the remarkably large sample sizes needed for cost-effectiveness studies, whether assessed by traditional deterministic sample-size calculations or by value-of-information approaches. Traditional clinical trials with samples of the size our results suggest are needed seem quite unlikely to be funded in the current funding environment, even if policymakers accept our conclusion that such studies would produce expected benefits that justify their expense, and they might not be feasible even if funding were not a concern.
Influence of switching behavior on value of research
The idea that switching behavior itself influences the value of research if patients tend to gravitate toward (or away from) preferred treatments over time and the idea that switching behavior may be influenced by research that suggests some treatment is dominant show that merely increasing the size of comparative effectiveness studies such as CATIE is unlikely to provide definitive evidence on the best use of antipsychotic agents in schizophrenia. This point is reinforced further by the need to study the potential for more carefully specified treatment algorithms involving multiple medications (at a point in time or over time) to alter cost and outcomes.
Value of CATIE study; limitations of ours
Although the CATIE study had major limitations, it should be emphasized that analysis performed here would not have been possible without the data it provided. Comparative effectiveness, like most research, is inherently iterative; prior studies inform future ones. Accordingly, our study has a variety of limitations, including being dependent on mortality data collected before the atypicals were available, being limited by the duration of and outcomes data collected by CATIE, and the distributional assumptions we made to analyze those data. Finally, our analysis cannot address the value of algorithms that may improve decisions about which sequences or combinations of drugs should be used when initial treatment with a single drug does not produce a desirable outcome.
Answering these questions will require further investment in research that will need to be justified relative to other areas of research. Building on the tradition of CATIE in establishing the potential of comparative effectiveness research to inform policy making, value-of-research calculations such as those presented here can play an important role in designing such trials, prioritizing them relative to other studies, and building the evidence base to assess the appropriate level of overall spending on comparative effectiveness research.