|Home | About | Journals | Submit | Contact Us | Français|
Conceived and designed the experiments: TH. Performed the experiments: EH. Analyzed the data: EH JS. Contributed reagents/materials/analysis tools: JS TH. Wrote the paper: EH JS TH.
Investigating differences between means of more than two groups or experimental conditions is a routine research question addressed in biology. In order to assess differences statistically, multiple comparison procedures are applied. The most prominent procedures of this type, the Dunnett and Tukey-Kramer test, control the probability of reporting at least one false positive result when the data are normally distributed and when the sample sizes and variances do not differ between groups. All three assumptions are non-realistic in biological research and any violation leads to an increased number of reported false positive results. Based on a general statistical framework for simultaneous inference and robust covariance estimators we propose a new statistical multiple comparison procedure for assessing multiple means. In contrast to the Dunnett or Tukey-Kramer tests, no assumptions regarding the distribution, sample sizes or variance homogeneity are necessary. The performance of the new procedure is assessed by means of its familywise error rate and power under different distributions. The practical merits are demonstrated by a reanalysis of fatty acid phenotypes of the bacterium Bacillus simplex from the “Evolution Canyons” I and II in Israel. The simulation results show that even under severely varying variances, the procedure controls the number of false positive findings very well. Thus, the here presented procedure works well under biologically realistic scenarios of unbalanced group sizes, non-normality and heteroscedasticity.
Many research projects in Life Sciences employ comparative studies –. For example, biodiversity exploration such as in population genetics measures the properties of individuals belonging to different groups. Often, multiple groups each containing several individuals are compared for traits which may differ only quantitatively but not qualitatively. The scientific hypothesis under test is then most often formulated in terms of mean differences among at least two of these groups. However, choosing an appropriate statistical inference procedure in order to assess mean differences between multiple groups often poses a non-trivial challenge. First, for many statistically less well trained users it is hard to verify to which extent statistical procedures for comparing means are based on theoretical assumptions such as normality or homoscedasticity, i.e. homogeneous or equal variances among all groups. This may lead to misapplication of tests, which is often not even detected by reviewers or editors. Second, for a specific experiment an appropriate statistical procedure might not even be available from the statistical literature. This is the case when the researcher can not assume the variances to be equal under all experimental conditions. All previously suggested parametric procedures for comparisons of means, such as the methods by Tukey  and Dunnett , require homogeneous variances among all groups. Applying these methods under heteroscedasticity, which refers to heterogeneous or unequal variances among all groups, can result in extreme size violations. As a consequence, false positive results will be reported with a probability far higher than , which is the a-priori chosen probability for wrongly rejecting a true null hypothesis. The situation is becoming even worse when unbalanced group sizes and/or non-normally distributed data are present. Unfortunately, unequal variances, non-normal data and unbalanced group sizes are realistic and hardly avoidable situations in biological research. A switch to non-parametric tests is not necessarily an option because even though they do not assume normality, they still assume that the shapes of the distributions are the same in all groups, which implies that variances are equal . Several approaches for global comparison of several means under heteroscedasticity have been reported –. Yet no methods for multiple pairwise comparisons of means in presence of heteroscedasticity and potentially unequal sample sizes in the groups exist so far.
Hothorn et al.  introduced a statistical framework for simultaneous inference in general parametric models, which can be applied to a broad range of parametric models including ANOVA models. Neither homoscedasticity nor normality nor balanced group sizes are assumed, thus allowing for multiple comparisons in balanced and unbalanced models with arbitrary error distribution and hence arbitrary data distribution and variance structure. Pairwise comparisons of means can be tested simultaneously under control of the familywise error rate. The familywise error rate is the probability of falsely rejecting one or more hypothesis (i.e. finding a significant difference among the means of any two groups in the dataset even though there is actually no difference present) and is used as the standard measure for false positive results in multiple testing.
The aim of this paper is to advocate a new statistical method for the comparison of multiple means which does not suffer from increased false positive results the standard procedures will produce under non-normal heteroscedastic errors in unbalanced experimental designs. Asymptotic control of the familywise error rate for this procedure has been shown . To assess the quality of the test under finite sample sizes, we examine the familywise error rate of the test under homoscedasticity as well as under heteroscedasticity for different error distributions in simulations and show that the familywise error rate is controlled. We also present the familywise error rate of procedures assuming homoscedasticity and show that the familywise error rate is not controlled under different forms of heteroscedasticity. In addition, we investigate the test's ability to find significant differences and therefore estimate the test's power, which is the probability of correctly rejecting a false hypothesis. We then reanalyze data from biodiversity research using this new procedure. In this research, the multiple cladogenic splits of evolutionary lineages (putative ecotypes) of the bacterium Bacillus simplex as an adaptational response to the microclimatically heterogeneous environment of “Evolution Canyon”, Israel, are being studied –. In this model population, unbalanced groups with frequently heterogeneous variances in their phenotypic properties are found. We apply the here presented method, which accounts for the existing heteroscedasticity. The analyzes are additionally conducted with methods requiring homogeneous variances. For several comparisons the results differ depending on whether heterogeneous variances are accounted for. When neglecting the heteroscedasticity, in several comparisons significant differences are found although they are actually not present or significant differences are not detected although they are present when the appropriate method is chosen. Results from simulations and the application to biodiversity research show how standard methods for multiple group comparison may fail under biologically realistic scenarios of heteroscedasticity and unbalanced groups, whereas the here presented method appears to be appropriate for such scenarios even in the situation of non-normal data. An implementation of the test procedure is provided in the multcomp package in the open-source-software R. We present R code which can be used to perform multiple comparisons of groups showing heterogeneous variances in the section “Computational Details”.
We consider a one-way ANOVA model
where denotes the th observation in group , is the overall average, denotes the main effect in group and are random errors.
To assess which particular groups differ concerning their means, we are interested in testing Tukey's all pairwise comparisons of group effects
or other post hoc comparisons simultaneously. To apply the inference procedure introduced by Hothorn et al.  these hypotheses have to be specified as general linear hypotheses of the model parameter vector . The general linear hypothesis
is set up by a matrix of linear functions , being the number of all pairwise comparisons. Each row of the matrix corresponds to one of the partial hypotheses . With the matrix of the form
and the right hand side of the hypotheses specified as , the general linear hypothesis corresponds to the partial hypotheses specified in equation (2). Further pairwise comparisons procedures like Dunnett's many-to-one comparisons can be specified by a corresponding matrix .
We assume that an estimate of the parameter vector can be calculated based on observations and that the estimate follows an asymptotic multivariate normal distribution . Additionally, a consistent estimation of the associated covariance matrix is required to be available. With these two assumptions fulfilled, the asymptotic distribution of the linear combinations is available, which is a joint normal distribution . The deviation of the estimates from the null hypothesis is standardized by . The test statistics are defined in terms of these standardized deviations, i.e., which again asymptotically follows a joint normal distribution: with . This distribution holds under heteroscedasticity or unequal sample sizes in the groups and is used as the reference distribution for the simultaneous inference on the comparisons specified in the general linear hypothesis.
The max- test provides the information which of the pairwise comparisons is significant . It is based on , which is the maximum of the absolute values of the standardized test statistics . Under the null hypothesis the distribution function of this statistic is
where is the density function of the distribution . Adjusted single-step -values, which control the familywise error rate, are
for the th partial hypothesis with the components of the observed test statistic . Approximate simultaneous -confidence intervals are given by
where are the square roots the diagonal elements of .
In the derivation of the max- test we only assume that the parameter estimates are asymptotically multivariate normal with a consistent estimate of the associated covariance matrix being available. In an ANOVA model the group effects are generally estimated by the ordinary least squares method. Under homoscedasticity the ordinary least squares parameter estimates are asymptotically normal and the ordinary least squares covariance estimation is a consistent estimation of the true covariance of the parameter estimates. Thus, both assumptions are fulfilled. In presence of unequal variances, the ordinary least squares parameter estimates are still asymptotically normal, while the covariance estimation obtained by the ordinary least squares estimation technique is inconsistent. Hence, a heteroscedastic consistent covariance estimation technique needs to be applied for simultaneous inference on the linear hypotheses. For small samples with a total number of observations up to Long and Ervin suggest to use the covariance estimation HC3 introduced by MacKinnon and White , .
The inference procedure is based on the asymptotic distribution of the test statistic. To assess the quality of the max- test in ANOVA models with finite sample sizes we investigated the familywise error rate and the power of the max- test in rather small samples by simulations. The familywise error rate must not exceed the a-priori defined level , i.e., the probability of rejecting at least one true null hypothesis. If the familywise error rate is controlled, we are additionally interested in the power of the test, which measures the test's ability to find significant differences. For each false comparison the power is the probability of rejecting this false comparison.
We considered unbalanced one-way ANOVA models with groups with equal variances and normal data (A) and with heterogeneous variances with smaller variances in the smaller groups (B, D) and vice versa (C, E) both for normal and non-normal, right-skewed data. For the classical procedures, these special conditions of positive or negative pairing of group sizes and variances typically lead to conservative or liberal results, respectively.
A: and , normal data.
B: and , normal data.
C: and , normal data.
D: and , non-normal data.
E: and , non-normal data.
For all pairwise comparisons of the group effects the familywise error rate and the power properties of the max- test using the covariance estimation HC3 were estimated and compared to the Tukey-Kramer test, which assumes equal variances among all groups.
Total sample sizes of were considered with the observations unbalancedly distributed to the four groups. The number of observations for each group were defined as , leading to . The overall mean was set to and all group effects were chosen equally . The random errors were independently normally distributed with group specific standard deviations . Standard deviations were chosen as in model A, in model B, in model C, in model D and in model E.
Datasets of size were simulated according to the considered models A to E. In each dataset all pairwise comparisons of the group effects were tested simultaneously by the max- test accounting for heteroscedasticity and by the Tukey-Kramer test.
To investigate the power of the tests the effects of groups 2 to 4 ( and ) were kept equal while the effect of the first group was chosen differently. Thus, the pairwise comparisons of with each of the three other effects were false. For each of these false partial hypotheses the power of the max- test and the Tukey-Kramer test were estimated by the proportion of correctly rejected partial hypotheses among 1000 datasets for increasing distances between and . 41 values of distances were considered. The familywise error rate was estimated by the proportion of datasets, in which at least one true partial hypothesis was falsely rejected. The same datasets were used for the analyzes of size and power leading to 41 estimated values of the familywise error rate each based on 1000 datasets. The distribution of the estimated familywise error rate is illustrated by the boxplots in Figure 1, where the boxplot for each setting is calculated from the 41 estimated values.
The B. simplex population from “Evolution Canyons” I and II in Israel has recently developed to a model study of bacterial adaptation and speciation under heterogeneous environmental conditions . These two canyons represent similar ecological sites, at a distance of 40 km, in which the orientation of the sun yields a strong sun-exposed and hot ‘African’ south-facing slope versus a rather cooler and mesic-lush ‘European’ north-facing slope within a distance of only 50–400 m. Phylogenetically, based on DNA sequences, the B. simplex population splits into two major groups GL1 and GL2. Interestingly, within each GL1 and GL2, further phylogenetic groups (or so called ‘putative ecotypes’) were observed which show a clear preference for either slope type , . As a putative ecotype (PE) we regard a phylogenetic lineage whose members are adapted to specific ecological conditions , . Whereas GL2 is composed of only PE1 and PE2, GL1 is made up of multiple PE (PE3–PE9) , . In our quest to understand this characteristic slope type preference of the bacteria, we analyze physiological properties (phenotypes) that might be explanatory, such as temperature stress related phenotypes as a putative evolutionary adaptive response to the different temperatures on both slopes. For example, the physical integrity of the cell membrane at different temperatures is crucial for the cell survival. Here, the fatty acid composition of the cell membrane is of substantial importance. This was the motivation for a recent study on the contents of high- and low-temperature-tolerance-providing fatty acids (FAs) of the B. simplex ecotypes . However, as the methods for the genetic characterization were improved in the meanwhile, leading to a re-shuffling of individuals into different groups (see also Table 3 of the supplemental material of ) and as the former fatty acid data were analyzed using the classical non-robust statistical tools  we take here the opportunity to reanalyze the experiment using the newly developed statistical tools presented in this manuscript. We focus specifically on the multiple ecotypes PE3 to PE9 from GL1 (we exclude PE8, as this ecotype is represented by only two bacterial strains).
Heteroscedasticity among the PE is assessed visually by boxplots, which illustrate the distribution of the FAs for the six PE. Analyzes are conducted both with methods assuming homoscedasticity and with methods accounting for heteroscedasticity to investigate in which way wrong conclusions are drawn when heterogeneous variances are ignored. We compute simultaneous confidence intervals for all pairwise differences of group effects to investigate which pairs of PE differ significantly concerning a specific growth condition of the bacteria . These confidence intervals are calculated by the max- method using the ordinary least squares covariance estimation (assuming homoscedasticity), by the max- method using the heteroscedastic consistent covariance estimation HC3 as well as by the Tukey-Kramer method.
The estimated familywise error rates for all pairwise comparisons of group effects for both the max- test using a heteroscedastic consistent covariance estimation and for the Tukey-Kramer test are illustrated in Figure 1. In the model with equal variances in all groups (model A) the estimated familywise error rate of the max- test is close to the a-priori chosen level of for either covariance estimation. With unequal variances and higher variances in the larger groups for both normal or non-normal data (models B and D), the Tukey-Kramer test is conservative while the estimated familywise error rate of the max- test using the heteroscedastic consistent covariance estimation is close to already for a total sample size of . In the situation with higher variances in the smaller groups for both normal or non-normal data (models C and E), the usage of the Tukey-Kramer test results in serious violations of the familywise error rate. The familywise error rate of the max- test using the consistent covariance estimation is liberal for a total sample size of but close to with increasing total sample size .
Figure 2 shows the power curves of the max- test for models A to C for the three pairwise comparisons of group effects with , when the effects of the first group differs from the remaining effects. Under homoscedasticity (model A) the power of both multiple test procedures is almost identical for equivalent sample size . In model B, the power of the max- test is higher than the power of the Tukey-Kramer test. In model C the probability of discovering a false hypothesis is higher for the Tukey-Kramer test, but yet this test cannot be used because the familywise error rate is not controlled (Figure 1C).
Figure 3 shows the distributions of high- and low-temperature-tolerance-providing FAs in six PE of B. simplex (PE3–PE9) for six different experimental conditions (Figures 3a to 3f). Variances differ considerably between the lineages within each type of experimental conditions. Thus, the validity of the results of the tests neglecting heteroscedasticity might be in question and attention should be drawn to the results of the max- method accounting for heteroscedasticity. Results of the inference procedures assuming homoscedasticity (Tukey-Kramer method and max- method using the ordinary least squares covariance estimation) are presented as well to show the extent of differences in the results (Figure 4).
The simultaneous confidence intervals for all pairwise differences of group effects for all six fatty acids calculated by the methods which assume homoscedasticity (Tukey-Kramer and ordinary max- method) do not alter in any comparison of strains. In contrast, the width of the max- confidence intervals based on the heteroscedastic consistent covariance estimation is noticeably different, either narrower or wider.
Two PE are considered significantly different concerning their fatty acid content, if the associated simultaneous confidence interval does not include the zero. For several comparisons the decision of significant difference depends on the method chosen (simultaneous confidence intervals colored blue). When heterogeneous variances are neglected, a significant difference in the lineages PE3 and PE5 is found concerning the FAs (Figure 4a), which is not present when heteroscedasticity is accounted for. For the other FAs (Figure 4b to 4f) significantly differing lineages of B. simplex are not detected, when heteroscedasticity is ignored.
We described the application of the simultaneous inference procedure proposed by Hothorn et al.  to pairwise comparisons of means. By using an appropriate covariance estimation technique, the method can be used for multiple comparisons in presence of either equal or unequal group variances in balanced or unbalanced designs with arbitrary error distribution.
Simulations showed, that the familywise error rate is bound by the a-priori chosen level of already for relatively small sample sizes in unbalanced designs with both normal or skewed error distributions and different kinds of pairing of group sizes and variance, whereas the Tukey-Kramer test can lead to false positive rates considerably higher than . Even in situations where the Tukey-Kramer test does not lead to inflated false positive rates, the max- test is superior to the Tukey-Kramer test, as it has the higher power to detect existing differences in means.
Thus, the max- test for multiple comparisons of means using the heteroscedastic consistent covariance estimation in presence of unequal variances helps to avoid an increased number of false positive results. The procedure is implemented in the R  add-on package multcomp  utilizing an implementation of the HC3 estimator in package sandwich . A short introduction along with an example is given in the Appendix.
Install the R software from http://CRAN.R-project.org/. Then use the R software to install the packages multcomp and sandwich. The multcomp package in R provides a general implementation of the framework for global and simultaneous inference in parametric models. In this section we present R code which can be used to perform multiple comparisons of groups showing heterogeneous variances. Data has to be in a form with two columns, where the first column contains the grouping variable and the second column contains the quantitative values of the observations. This can be in a .txt, .csv or .Rda file, which can be imported in R by the functions read.table(), read.csv() and load() respectively, or by the R Commander. The example data used in the following correspond to the data underlying Figure 3A and and4A4A and are available in the multcomp package.
The example data fattyacid can be loaded by
It contains the grouping variable (here the putative ecotype PE) in the first column and the fatty acid content (FA) by which the groups are to be compared in the second column:
The following R code performs all-pairwise comparisons of means of the fattyacid data. It can be applied to any other data by replacing fattyacid in the third line by the name of the object containing the data in the two-column way described above, and by replacing the variable names PE and FA by the names of the variables used in the dataset wherever PE and FA appear in the code.
> amod <- aov(FA~PE, data=fattyacid)
> amod_glht <- glht(amod, mcp(PE=“Tukey”), vcov=vcovHC)
Simultaneous Tests for General Linear Hypotheses
Multiple Comparisons of Means: Tukey Contrasts
Fit: aov(formula=FA~PE, data=fattyacid)
Estimate Std. Error t value Pr(>|t|)
PE4 - PE3==0 −0.012820 0.034997 −0.366 0.99905
PE5 - PE3==0 −0.084398 0.033846 −2.494 0.13104
PE6 - PE3==0 0.019286 0.035760 0.539 0.99400
PE7 - PE3==0 −0.010048 0.038006 −0.264 0.99981
PE9 - PE3==0 0.075536 0.035783 2.111 0.28057
PE5 - PE4==0 −0.071579 0.019764 −3.622 0.00600 **
PE6 - PE4==0 0.032105 0.022887 1.403 0.71500
PE7 - PE4==0 0.002772 0.026258 0.106 1.00000
PE9 - PE4==0 0.088355 0.022923 3.854 0.00282 **
PE6 - PE5==0 0.103684 0.021085 4.917 <0.001 ***
PE7 - PE5==0 0.074351 0.024703 3.010 0.03678 *
PE9 - PE5==0 0.159934 0.021124 7.571 <0.001 ***
PE7 - PE6==0 −0.029333 0.027266 −1.076 0.88423
PE9 - PE6==0 0.056250 0.024072 2.337 0.18270
PE9 - PE7==0 0.085583 0.027297 3.135 0.02592 *
— Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Adjusted p values reported – single-step method)
First, a common ANOVA model is fitted by the function aov(). The fitted model amod is then given to the function glht() which sets up the hypotheses to be tested (i.e. the multiple contrasts of means). The argument vcov=vcovHC specifies the use of the heteroscedastic consistent covariance estimation HC3 accounting for the heterogeneous variances. The function vcovHC() and further heteroscedastic consistent sandwich covariance estimation functions are provided in the package sandwich. For multiple comparisons by the max- method in the situation of homogeneous variances the argument vcov=vcovHC of the function glht() has to be omitted.
Adjusted -values assuring that the familywise error rate is not larger than are computed by the summary() function. For each pairwise comparison the adjusted -values are given in the last column of the output (column headed ‘Pr(>|t|)’). An adjusted -value of smaller than the a-priori chosen value of indicates a significant difference of the corresponding group means. We here find six significant differences on the level . Significance is marked by asterisks at the end of the associated row.
Simultaneous confidence intervals for each difference of means can be computed by
Simultaneous Confidence Intervals
Multiple Comparisons of Means: Tukey Contrasts
Fit: aov(formula=FA~PE, data=fattyacid)
95% family-wise confidence level
Estimate lwr upr
PE4 - PE3==0 −0.012820 −0.114083 0.088444
PE5 - PE3==0 −0.084398 −0.182332 0.013535
PE6 - PE3==0 0.019286 −0.084185 0.122756
PE7 - PE3==0 −0.010048 −0.120016 0.099921
PE9 - PE3==0 0.075536 −0.028002 0.179074
PE5 - PE4==0 −0.071579 −0.128765 −0.014393
PE6 - PE4==0 0.032105 −0.034117 0.098328
PE7 - PE4==0 0.002772 −0.073204 0.078748
PE9 - PE4==0 0.088355 0.022027 0.154683
PE6 - PE5==0 0.103684 0.042676 0.164693
PE7 - PE5==0 0.074351 0.002874 0.145828
PE9 - PE5==0 0.159934 0.098812 0.221057
PE7 - PE6==0 −0.029333 −0.108227 0.049560
PE9 - PE6==0 0.056250 −0.013400 0.125900
PE9 - PE7==0 0.085583 0.006601 0.164565
where the entries of the columns headed ‘lwr’ (lower) and ‘upr’ (upper) give a lower and an upper bound for the confidence interval of each contrast.
visualizes the simultaneous confidence intervals.
The given R Code performs Tukey's all pairwise comparisons of means. Dunnett's many-to-one contrasts comparing several groups each with a reference group can be tested by replacing the argument mcp(PE=“Tukey”) by mcp(PE=“Dunnett”) in the function glht(). Arbitrary other multiple contrasts of group means can be described symbolically, e.g. by replacing the argument mcp(PE=“Tukey”) by mcp(PE=c(“PE4 - PE3=0”,
“PE5 - PE3=0”,
“PE9 - PE5=0”)),
for comparisons of means of groups 4 and 3, 5 and 3, and 9 and 5.
Further details to the above listed R code are available at http://CRAN.R-project.org/package=multcomp.
The simulation results can be reproduced using the R transcript file available via
> file.show(system.file(“multcomp_VA.R”, package=“multcomp”))
JS thanks Lea Vaas for drawing his attention to reference .
Competing Interests: The authors have declared that no competing interests exist.
Funding: J.S. received funding from Deutsche Forschungsgemeinschaft, grant number SI 1352/1-2 (http://www.dfg.de). E.H. and T.H. were supported by the Ludwig-Maximilians-Universität München LMUinnovative project “BioMed-S” (http://www.biomed-s.uni-muenchen.de). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.