To adhere to statistical design issues, researchers often specify a small set of measures to serve as the primary outcomes, with another (often larger) set listed as secondary. While it is common practice to collect, analyze and report multiple measures, the efficient and appropriate analysis of multiple outcomes is not fully established. A number of approaches to accounting for multiple outcomes have been proposed [1–3], assessed [4] and reviewed [5]. The most common method for analyzing multiple outcomes is separate testing of each individual outcome, sometimes with but most often without adjustment for multiple testing. Another approach involves combining the multiple outcomes into a single (composite) outcome and performing a single test [6]. A third approach undertakes global testing using simultaneous (joint) tests [7].

The choice of an appropriate method for dealing with multiple outcomes is important because clinical interpretations can be difficult in the presence of multiple conflicting results. Simultaneous or joint models that provide an overall test, with separate reports of individual outcome, can provide useful additional information. Moreover, joint models can be more powerful if some outcomes are missing.

Study design features should be specified prior to patient enrollment and characterized in the study protocol. However, prior work by Al-Marzouki et al [8] and Chan et al [9] found discrepancies in outcomes between published study protocols and clinical trial reports. Reasons for these discrepancies may be related to unanticipated changes while conducting the trial, such as modifications to trial inclusion and exclusion criteria to increase enrollment. Al-Marzouki and colleagues found that there were major differences in 11 out of 37 trials. Overall there was a median of one outcome in published protocols and a median of two outcomes in the published report. Chan et al concluded that reported outcomes are often incomplete and inconsistent with the registered protocols, which potentially yield bias and unreliable results. Specifically, statistically significant outcomes were more likely to be fully reported than non-significant outcomes, indicating that results may be overestimating the benefits of an intervention. Turner and colleagues [10] found a similar trend among antidepressant clinical trials. Another study by Viereck and Boudes [11] found that pharmaceutical industry practices do not generally make clinical trial protocols and results accessible to the general public. The FDA Amendment Act [12] expanded the trial registry established under the FDA Modernization Act to encompass a larger set of protocols involving treatments and devices [13]. Along with the International Committee of Medical Journal Editors [14], these new promulgations are likely to improve reporting.

We assessed the prevalence of multiple outcomes in clinical trials of depression. Major depression and related disorders were chosen because they have a profound personal, social and economic cost [15], and are the focus of a number of prevention and intervention trials. Examples of depression measures include a clinical diagnosis, measures of depressive symptoms (such as the Center for Epidemiologic Studies–Depression Scale [16], the Beck Depression Inventory [17, 18], the Hamilton Depression Rating Scale [19], and the Montgomery-Asberg Depression Rating Scale [20]. The use of multiple outcomes in depression trials is particularly common because disease complexity is multifaceted [7]. No one measure encompasses all aspects of the disorder, and clinicians may be interested in the impact of a new treatment on different domains. As a result, it is naive to force a single outcome. Instead, use of a broad range of clinically relevant measures, along with a procedure to globally assess them is more realistic and useful.

Of the primary outcomes that had a depression component, the most commonly used were the Hamilton Depression Rating Scale (22.9%, n=11), the Montgomery-Asberg Depression Rating Scale (8.3%, n=4), Clinical Global Impression Scale (8.3%, n=4), and clinical diagnosis using the DSM-IV (8.3%, n=4). Table 2 displays the distribution of primary depression outcomes used in more than one study.

Secondary outcomes were also common with a median of three outcomes. This was also heavily skewed with a maximum number of secondary outcomes of 31 (25^th percentile 0, 75^th percentile 6 outcomes). Almost all (94.5%, n=52) of the articles reported more than one primary and secondary outcome. The number of secondary outcomes was significantly larger (p=0.003) for papers reporting only 1 primary outcome (median=5) as compared to those with more than 1 primary outcome (median=0). The median of the total number of outcomes was 7 outcomes (25^th percentile 4, 75^th percentile 10 outcomes). Figure 2 displays boxplots of the distribution of number of primary, secondary and total (sum of primary + secondary) outcomes.

A total of eight articles used a Bonferroni-type adjustment; however five papers reporting this approach applied it to comparisons within multiple treatments and not for multiple outcomes [23–27]. Of the 52 articles with multiple outcomes, only 5.8% (n=3) used a Bonferroni adjustment. While Strong et al [28)] specified only one primary outcome, the seven secondary outcomes analyzed were adjusted by utilizing a modified cutoff for statistical significance at 0.01. Welton et al [29] adjusted for the 41 outcomes by using a Bonferroni corrected alpha level of 0.0001. Lesperance et al [30] clearly specified a single primary outcome and single secondary outcome in the abstract, and included 6 exploratory outcomes later in their analysis. They used a Bonferroni-like multiplicity adjustment by partitioning the experiment-wise alpha level into 0.033 for the primary outcome analysis and 0.017 the secondary outcome analysis. All additional analyses, which they designated as exploratory, used the standard alpha level of 0.05. No articles used a Hochberg [31] or similar procedure. None of the papers reported use of joint testing methods or global tests.

Only two articles used a composite measure as their primary outcome. Raskin et al [32] combined information from four cognitive tests (Verbal Learning and Recall, Symbol Digit Substitution, Two-Digit Cancellation, and Letter-Number Sequencing). The composite cognitive score was a weighted sum (in proportion to the time spent administering the test) that ranged from 0 to 51. Goldberg et al [33] used a measure of either recovery (8 weeks with partial or full remission) or recovering (4 weeks with low level of symptoms).

The median sample size of the reported RCT’s was 200 with a minimum size of 28 and maximum of 7380. Studies with sample sizes less than 100 and greater than or equal to 400 participants had significantly more outcomes reported than did studies with sample sizes between 100 and 400 (p=0.003, see Table 3). There were more papers with total number of participants between 100 and 400 with only 1 outcome than expected.

The CONSORT 2010 [36] statement strengthens the discussion of multiple outcomes, and notes that while a trial may have more than one primary outcome, "having several primary outcomes, however, incurs the problems of interpretation associated with multiplicity of analyses … and is not recommended. " (p. 7).

Nearly half of depression clinical trials published between January 2007 and October 2008 in leading medical and psychiatry journals reported more than one primary outcome, while nearly all reported more than one primary or secondary outcome. The median number of total outcomes (not including our category of tertiary outcomes for side effects or similar) was seven. While depression is a multifaceted disorder that manifests itself in many ways over multiple domains, there is a need to specify what outcomes are being considered and how they will be accounted for in a clear fashion. No single primary outcome is appropriate for all depression studies.

We also found that determining the number of primary and secondary outcomes for many of the articles included in this study was not straightforward, with relatively few clearly and consistently specifying primary and secondary outcomes [e.g. 28, 30, 37].

Separate analyses, with no correction for multiplicity, were the most common method to analyze multiple outcomes. A familiar drawback of this approach is the risk of inflating the Type-I error rate (likelihood of obtaining significant results due to chance). While it is critically important that multiple domains of a disorder are discussed, interpretation of a large number of p-values by a clinical reader can be challenging. While we focused on randomized trials, similar issues arise in observational studies. Failure to account for the multiplicity of comparisons could lead to invalid inferences and spurious conclusions. At the very least, researchers reporting a profusion of results without adjustment should address the internal consistency of their findings [38].

The appropriate use of corrections for multiplicity is not always straightforward [38, 39]. Rothman [40] notes that scientists need to explore multiple leads in the search for better interventions and treatments, and that inappropriate use of multiplicity adjustment may obscure possibly important findings. Nonetheless, inflation of Type I error is a serious concern, and in the setting of randomized trials, this must be accounted for in the trial protocol. Several papers employed a Bonferroni-type correction to address the issue of multiplicity. A particularly creative approach was undertaken by Lesperance et al [30], where the primary outcome (HAM-D) was tested at alpha=0.033 while the secondary outcome (BDI) was tested at 0.017.

A common critique of the Bonferroni method is that it will tend to be conservative when the outcomes are correlated. However, the simulations of Yoon et al [4] indicated that for settings similar to that of the CATIE trial, with 5 outcomes, the Bonferroni adjustment performed adequately when correlations were moderately. For psychiatric studies, it is rare to have highly correlated endpoints.

Further use of more sophisticated approaches to account for multiplicity may be warranted. Joint testing is particularly attractive in this setting [4, 7]. By capitalizing on the correlation of multiple outcomes, these methods are generally more powerful than separate analyses [7] or Bonferroni adjustment [4]. While more complicated than separate testing of multiple outcomes with multiplicity adjustment, these approaches are straightforward to fit in general purpose statistical software [4, 7]. Changes in the scale of research and the use of large data banks to test hypotheses will complicate future evidence-based medicine, and will likely exacerbate these issues [41].

Another troubling problem, unrelated to the multiplicity issue, concerns missing data. When outcome data are only partially observed, separate analyses of the outcomes will lead to the inclusion of different subjects for the analysis of each outcome. The reader is then faced with interpreting treatment effects based on different samples of subjects, as well as assessing assumptions regarding missingness. Joint models are particularly attractive in this setting, since they incorporate partially observed data and pool information across outcomes.

The concordance between the published protocols in registries and the number of published outcomes was also discouragingly low, albeit similar to findings reported for cardiology, rheumatology and gastroenterology RCTs [42]. Although the 2007 FDA Modernization Act now requires investigators and sponsors to submit information for any applicable clinical trial to NIH/NLM, complete adherence to this act will require some time before becoming appearing in published trial results. While we anticipate that more investigators will publish their protocols in a more timely and complete fashion as part of new journal requirements, selective reporting remains a potential problem [8, 9]. Investigators must not "torture their data until they speak" [38] by examining additional outcomes, undertaking unplanned subgroup analyses or similar mischief. The addition of a CONSORT checklist item to note changes in trial outcomes after the trial commences should also help with this issue.

To help improve practice in this area, we suggest that all clinical trial reports:

appendix