Overview of module preservation statistics
presents an overview of the module preservation statistics studied in this article. We distinguish between cross-tabulation based and network based preservation statistics. Cross-tabulation based preservation statistics
require independent module detection in the test network and take the module assignments in both reference and test networks as input. Several cross-tabulation based statistics are described in the first section of Supplementary Text S1
. While cross-tabulation approaches are intuitive, they have several disadvantages. To begin with, they are only applicable if the module assignment in the test data results from applying a module detection procedure to the test data. For example, a cross-tabulation based module preservation statistic would be meaningless when modules are defined as gene ontology categories since both reference and test networks contain the same sets of genes. But a non-trivial question is whether the network connections of a module (gene ontology category) in the reference network resemble those of the same module in the test network. To measure the resemblance of network connectivity, we propose several measures based on network statistics. Network terminology is reviewed in and in Methods
Overview of module preservation statistics.
Glossary of network terminology.
Even when modules are defined using a module detection procedure, cross-tabulation based approaches face potential pitfalls. A module found in the reference data set will be deemed non-reproducible in the test data set if no matching module can be identified by the module detection approach in the test data set. Such non-preservation may be called the weak non-preservation: “the module cannot be found using the current parameter settings of the module detection procedure”. On the other hand, one is often interested in strong non-preservation: “the module cannot be found irrespective of the parameter settings of the module detection procedure”. Strong non-preservation is difficult to establish using cross-tabulation approaches that rely on module assignment in the test data set. A second disadvantage of a cross-tabulation based approach is that it requires that for each reference module one finds a matching test module. This may be difficult when a reference module overlaps with several test modules or when the overlaps are small. A third disadvantage is that cross-tabulating module membership between two networks may miss that the fact that the patterns of connectivity between module nodes are highly preserved between the two networks.
Network based statistics
do not require the module assignment in the test network but require the user to input network adjacency matrices (described in Methods
). We distinguish the following 3 types of network based module preservation statistics: 1) density based, 2) separability based, and 3) connectivity based preservation statistics. Density based
preservation statistics can be used to determine whether module nodes remain highly connected in the test network. Separability based
statistics can be used to determine whether network modules remain distinct (separated) from one another in the test network. While numerous measures proposed in the literature combine aspects of density and separability, we keep them separate and provide evidence that density based approaches can be more useful than separability based approaches in determining whether a module is preserved. Connectivity based
preservation statistics can be used to determine whether the connectivity pattern between nodes in the reference network is similar to that in the test network. As detailed in Methods
, several module preservation statistics are similar to previously proposed cluster quality and preservation statistics, while others (e.g. connectivity based statistics) are novel.
reports the required input
for each preservation statistic. Since each preservation statistic is used to evaluate the preservation of modules defined in a reference network, it is clear that each statistic requires the module assignment from the reference data. But the statistics differ with regard to the module assignment in the test data. Only cross-tabulation based statistics require a module assignment in the test data. Network based preservation statistics do not require a test set module assignment. Instead, they require the test set network adjacency matrix (for a general network) or the test data set
of numeric variables (for a correlation network).
We distinguish network statistics by the underlying network. Some preservation statistics are defined for a general network (defined by an adjacency matrix) while others are only defined for a correlation network (constructed on the basis of pairwise correlations between numeric variables). Our applications show that the correlation structure facilitates the definition of particularly powerful module preservation statistics. Preservation statistics 4–11 () can be used for general networks while statistics 12–19 assume correlation networks. Network density and module separability statistics only need the test set adjacency matrix while the connectivity preservation statistics also require the adjacency matrix in the reference data.
It is often not clear whether an observed value of a preservation statistic is higher than expected by chance. As detailed in Methods
, we attach a significance level (permutation test p-value) to observed preservation statistics, by using a permutation test procedure which randomly permutes the module assignment in the test data. Based on the permutation test we are also able to estimate the mean and variance of the preservation statistic under the null hypothesis of no relationship between the module assignments in reference and test data. By standardizing each observed preservation with regard to the mean and variance, we define a
statistic for each preservation statistic. Under certain assumptions, each
statistic (approximately) follows the standard normal distribution if the module is not preserved. The higher the value of a Z statistic, the stronger the evidence that the observed value of the preservation statistic is significantly higher than expected by chance.
Composite preservation statistics and threshold values
Because preservation statistics measure different aspects of module preservation, their results may not always agree. We find it useful to aggregate different module preservation statistics into composite preservation statistics. Composite preservation statistics also facilitate a fast evaluation of many modules in multiple networks. We define several composite statistics.
For correlation networks based on quantitative variables, the
density preservation statistics are summarized by
(Equation 30), the
connectivity based statistics are summarized by
(Equation 31), and all individual
statistics are summarized by
defined as follows
As detailed in the Methods
, our simulations suggest the following thresholds for
there is strong evidence that the module is preserved; if
there is weak to moderate evidence of preservation; if
, there is no evidence that the module preserved. For general networks defined by an adjacency matrix, we find it expedient to summarize the preservation statistics into a summary statistic denoted
Since biologists are often more familiar with p-values as opposed to Z statistics, our R implementation in function modulePreservation also calculates empirical p-values. Analogous to the case of the Z statistics, the p-values of individual preservation statistic are summarized into a descriptive measure called
. The smaller
, the stronger the evidence that the module is preserved. In practice, we observe an almost perfect inverse relationship (Spearman correlation
The Z statistics and permutation test p-values often depend on the module size (i.e. the number of nodes in a module). This fact reflects the intuition that it is more significant to observe that the connectivity patterns among hundreds of nodes are preserved than to observe the same among say only
nodes. Having said this, there will be many situations when the dependence on module size is not desirable, e.g., when preservation statistics of modules of different sizes are to be compared. In this case, we recommend to either focus on the observed values of the individual statistics or alternatively to summarize them using the composite module preservation statistic
(Equation 34). The
is useful for comparing relative preservation among multiple modules: a module with lower median rank tends to exhibit stronger observed preservation statistics than a module with a higher median rank. Since
is based on the observed preservation statistics (as opposed to Z statistics or p-values) we find that it is much less dependent on module size.
Application 1: Preservation of the cholesterol biosynthesis module between mouse tissues
Several studies have explored how co-expression modules change between mouse tissues 
and/or sexes 
. Here we re-analyze gene expression data from the liver, adipose, muscle, and brain tissues of an F2 mouse intercross described in 
. The expression data contain measurements of 17104 genes across the following numbers of microarray samples: 137 (female (F) adipose), 146 (male (M) adipose), 146 (F liver), 145 (M liver), 125 (F muscle), 115 (M muscle), 148 (F brain), and 141 (M brain).
We consider a single module defined by the genes of the gene ontology (GO) term “Cholesterol biosynthetic process” (CBP, GO id GO:0006695 and its GO offspring). Of the 28 genes in the CBP, 24 could be found among our 17104 genes. Cholesterol is synthesized in liver and we used the female liver network as the reference network module. As test networks we considered the CBP co-expression networks in other tissue/sex combinations.
Each circle plot in visualizes the connection strengths (adjacencies) between CBP genes in different mouse tissue/sex combination. The color and width of the lines between pairs of genes reflect the correlations of their gene expression profiles across a set of microarray samples. Before delving into a quantitative analysis, we invite the reader to visually compare the patterns of connections. Clearly, the male and female liver networks look very similar. Because of the ordering of the nodes, the hubs are concentrated on the upper right section of the circle and the right side of the network is more dense. The adipose tissues also show this pattern, albeit much more weakly. On the other hand, the figures for the brain and muscle tissues do not show these patterns. Thus, the figure suggests that the CBP module is more strongly preserved between liver and adipose tissues than between liver and brain or muscle.
Network plot of the module of cholesterol biosynthesis genes in different mouse tissues.
We now turn to a quantitative assessment of this example. We start out by noting that a cross-tabulation based approach
of module preservation is meaningless in this example since the module is a GO category whose genes can trivially be found in each network. However, it is a very meaningful exercise to measure the similarity of the connectivity patterns of the module genes across networks. To provide a quantitative assessment of the connectivity preservation, it is useful to adapt network concepts (also known as network statistics or indices) that are reviewed in Methods
. provides a quantitative assessment of the preservation of the connectivity patterns of the cholesterol biosynthesis module between the female liver network and networks from other sex/tissue combinations. presents the composite summary statistic (
, Equation 1) in each test network. Overall, we find strong evidence of preservation (
, Equation 1) in the male liver network but no evidence (
) of preservation in the female brain and muscle networks. We find that the connectivity of the female liver CBP is most strongly preserved in the male liver network. It is also weakly preserved in adipose tissue but we find no evidence for its preservation in muscle and brain tissues. The summary preservation statistic
measures both aspects of density and of connectivity preservation. We now evaluate which of these aspects are preserved. shows that the module shows strong evidence of density preservation (
) (Equation 30) in the male liver network but negligible density preservation in the other networks. Interestingly, shows that the module has moderate connectivity preservation
(Equation 31) in the adipose networks.
Preservation of GO term cholesterol biosynthetic process across mouse tissues.
measure summarizes the statistical significance of 3 connectivity based preservation statistics. Two of our connectivity measures evaluate whether highly connected intramodular hub nodes in the reference network remain hub nodes in the test network. Preservation of intramodular connectivity reflects the preservation of hub gene status between the reference and test network. One measure of intramodular connectivity is the module eigengene-based connectivity measures
(Equation 17), which is also known as the module membership measure of gene 
. Genes with high values of
are highly correlated with the summary profile of the module (module eigengene defined as the first principal component, see the fifth section in Supplementary Text S1
). A high correlation of
between reference and test network can be visualized using a scatter plot and quantified using the correlation coefficient
. For example, shows that
in the female liver module is highly correlated with that of the male liver network (
). Further, the scatter plots in show that the
measures between liver and adipose networks show strong correlation (preservation):
), while the correlation between
in female liver and the brain and muscle data sets are not significant. This example demonstrates that connectivity preservation measures can uncover a link between CBP in liver and adipose tissues that is missed by density preservation statistics.
We briefly compare the performance of our network based statistics with those from the IGP method 
. The R implementation of the IGP statistic requires that at least 2 modules are being evaluated. To get it to work for this application that involves only a single module, we defined a second module by randomly sampling half of the genes from the rest of the entire network. shows high, nearly constant values of the IGP statistic across networks, which indicates that the CBP module is present in all data sets. Note that the IGP statistic does not allow us to argue that the CBP module in the female liver network is more similar to the CBP module in the male liver than in other networks. This reflects the fact that the IGP statistic, which is a cluster validation statistic, does not measure connectivity preservation.
Application 2: Preservation of human brain modules in chimpanzee brains
Here we study the preservation of co-expression between human and chimpanzee brain gene expression data. The data set consists of 18 human brain and 18 chimpanzee brain microarray samples 
. The samples were taken from 6 regions in the brain; each region is represented by 3 microarray samples. Since we used the same weighted gene co-expression network construction and module identification settings as in the original publication, our human modules are identical to those in 
. Because of the relatively small sample size only few relatively large modules could be detected in the human data. The resulting modules were labeled by colors: turquoise, blue, brown, yellow, green, black, red (see ). Oldham et al
(2006) determined the biological meaning of the modules by examining over-expression of module genes in individual brain regions. For example, heat maps of module expression profiles revealed that the turquoise module contains genes highly expressed in cerebellum, the yellow module contains genes highly expressed in caudate nucleus, the red module contains genes highly expressed in anterior cingulate cortex (ACC) and caudate nucleus, and the black module contains mainly genes expressed in white matter. The blue, brown and green modules contained genes highly expressed in cortex, which is why we refer to these modules as cortical modules. Visual inspection of the module color band below the dendrograms in suggests that most modules show fairly strong preservation. Oldham et al
argued that modules corresponding to evolutionarily older brain regions (turquoise, yellow, red, black) show stronger preservation than the blue and green cortical modules 
. Here we re-analyze these data using module preservation statistics.
Cross-tabulation based comparison of modules (defined as clusters) in human and chimpanzee brain networks.
The most common cross-tabulation approach
starts with a contingency table that reports the number of genes that fall into modules of the human network (corresponding to rows) versus modules of the chimpanzee network (corresponding to columns). The contingency table in shows that there is high agreement between the human and chimpanzee module assignments. The human modules black, brown, red, turquoise, and yellow have well-defined chimpanzee counterparts (labeled by the corresponding colors). On the other hand, the human green cortical module appears not to be preserved in chimpanzee since most of its genes are classified as unassigned (grey color) in the chimpanzee network. Further, the human blue cortical module (360 genes) appears to split into several parts in the chimpanzee network: 27 genes are part of the chimpanzee blue module, 85 genes are part of the chimpanzee brown module, 52 fall in the chimpanzee turquoise module, 155 genes are grey in the chimpanzee network, etc. To arrive at a more quantitative measure of preservation, one may quantify the module overlap or use Fisher's exact test to attach a significance level (p-value) to each module overlap (as detailed in the first section of Supplementary Text S1
). The contingency table in shows that every human module has significant overlap with a chimpanzee module. However, even if the resulting p-value of preservation were not significant, it would be difficult to argue that a module is truly a human-specific module since an alternative module detection strategy in chimpanzee may arrive at a module with more significant overlap. In order to quantify the preservation of human modules in chimpanzee samples more objectively, one needs to consider statistics that do not rely on a particular module assignment in the chimpanzee data.
We now turn to approaches for measuring module preservation that do not require that module detection has been carried out in the test data set. show composite module preservation statistics of human modules in chimpanzee samples. The overall significance of the observed preservation statistics can be assessed using
(Equation 1) that combines multiple preservation
statistics into a single overall measure of preservation, . Note that
shows a strong dependence on module size, which reflects the fact that observing module preservation of a large module is statistically more significant than observing the same for a small module. However, here we want to consider all modules on an equal footing irrespective of module size. Therefore, we focus on the composite statistic
which shows no dependence on module size (). The median rank is useful for comparing relative preservation among modules: a module with lower median rank tends to exhibit stronger observed preservation statistics than a module with a higher median rank. shows that the median ranks of the human brain modules. The median rank of the yellow module is 1, while the median ranks of the blue module is 6, indicating that the yellow module is more strongly preserved than the blue module. Our quantitative results show that modules expressed mainly in evolutionarily more conserved brain areas such as cerebellum (turquoise) and caudate nucleus (yellow and partly red) are more strongly preserved than modules expressed primarily in the cortex that is very different between humans and chimpanzees (green and blue modules). Thus the module preservation results of
, corroborate Oldham's original finding regarding the relative lack of preservation of cortical modules.
Composite preservation statistics of human modules in chimpanzee samples.
Since the modules of this application are defined as clusters, it makes sense to evaluate their preservation using cluster validation statistics. shows that the IGP statistic implemented in the R package clusterRepro 
also shows a strong dependence on module size in this application. The IGP values of all modules are relatively high. However, the permutation p-values (panels C and D) identify the green module as less preserved than the other modules (
, Bonferroni corrected p-value 0.43). show scatter plots between the observed IGP statistic and
, respectively. In this example, where modules are defined as clusters, the IGP statistic has a high positive correlation (
and a moderately large negative correlation (
. The negative correlation is expected since low median ranks indicate high preservation.
While composite statistics summarize the results, it is advisable to understand which properties of a module are preserved (or not preserved). For example, module density based
statistics allow us to determine whether the genes of a module (defined in the reference network) remain densely connected in the test network. As an illustration, we will compare the module preservation statistics for the human yellow module whose genes are primarily expressed in caudate nucleus (an evolutionarily old brain area), and the human blue module whose genes are expressed mostly in the cortex which underwent large evolutionary changes between humans and chimpanzees. In chimpanzees, the mean adjacency of the genes comprising the human yellow module is significantly higher than expected by chance, with a high permutation statistic
. But the corresponding permutation
statistic for the human blue module is only weakly significant,
(see Supplementary Text S2
and Supplementary Table S1
). Thus, the mean adjacency permutation statistic suggests that the blue module is less preserved than the yellow module.
For co-expression modules, one can define an alternative density measure based on the module eigengene (). The higher the proportion of variance explained by the module eigengene (defined in the fifth section in Supplementary Text S1
) in the test set data, the tighter is the module in the test set. The human yellow module exhibits a high proportion of variance explained,
, and the corresponding permutation
. In contrast, for the human blue module we find
and the corresponding permutation
. The permutation statistics again suggest that the yellow module is more preserved than the blue module.
Connectivity-based statistics for evaluating the preservation of the human yellow and blue modules in the chimpanzee network.
Although density based approaches are intuitive, they may fail to detect another form of module preservation, namely the preservation of connectivity patterns
among module genes. For example, network module connectivity preservation can mean that, within a given module
, a pair of genes with a high connection strength (adjacency) in the reference network also exhibits a high connection strength in the test network. This property can be quantified by correlating the pairwise adjacencies or correlations between reference and test networks. For the genes in the human yellow module, the scatter plot in shows pairwise correlations in the human network (
-axis) versus the corresponding correlations in the chimpanzee network (
-axis). The correlation between pairwise correlations (denoted by
and is highly significant,
. The analogous correlation for the blue module, is lower, 0.56, but still highly significant,
, in part because of the higher number of genes in the blue module.
A related but distinct connectivity preservation statistic quantifies whether intramodular hub genes in the reference network remain intramodular hub genes in the test network. Intramodular hub genes are genes that exhibit strong connections to other genes within their module. This property can be quantified by the intramodular connectivity
(Equation 7): hub genes are genes with high
. Intramodular hub genes often play a central role in the module 
. Preservation of intramodular connectivity reflects the preservation of hub gene status between the reference and test network. For example, the intramodular connectivity of the human yellow module is preserved between the human and chimpanzee samples,
(). In contrast, the human blue (cortical) module exhibits a lower correlation (preservation)
value is more significant because of the higher number of genes in the blue module.
Another intramodular connectivity measure is
, which turns out to be highly related with 
. shows that
for the human yellow module is highly preserved in the chimpanzee network (
). The corresponding correlation in the human blue module is lower,
(). In summary, the observed preservation statistics show that the human yellow module (related to the caudate nucleus) is more strongly preserved in the chimpanzee samples than the human blue module (related to the cortex).
Application 3: Preservation of KEGG pathways between human and chimpanzee brains
To further illustrate that modules do not have to be clusters, we now describe an application where modules correspond to KEGG pathways. KEGG (Kyoto Encyclopedia of Genes and Genomes) is a knowledge base for systematic analysis of gene functions, linking genomic information with higher order functional information 
. KEGG also provides graphical representations of cellular processes, such as signal transduction, metabolism, and membrane transport. To illustrate the use of the module preservation approach, we studied the preservation of selected KEGG pathway networks across human and chimpanzee brain correlation networks. While pathways in the KEGG database typically describe networks of proteins, our analysis describes the correlation patterns between mRNA expression levels of the corresponding genes. As before, we define a weighted correlation network adjacency matrix between the genes (described in the third section of Supplementary Text S1
). For the sake of brevity, we focused the analysis on the following 8 signaling pathways: Hedgehog signaling pathway (12 genes in our data sets), apoptosis (24 genes in our data sets), TGF-beta signaling pathway (26 genes), Phosphatidylinositol signaling system (39 genes), Wnt signaling pathway (55 genes), Endocytosis (59 genes), Calcium signaling pathway (78 genes), and MAPK signaling pathway (93 genes). All of these pathways have been shown to play critical roles in normal brain development and function 
. We provide a brief description of the functions of these pathways in Methods
; more detailed description can be found in the KEGG database and in numerous textbooks.
show the composite preservation statistics
. Both statistics indicate that the apoptosis module is the least preserved module. To visualize the lack of preservation, consider the circle plots of apoptosis genes in L, M that show pronounced differences in the connectivity patterns among apoptosis genes. While we caution the reader that additional data are needed to replicate these differences, prior literature points to an evolutionary difference for apoptosis genes. For example, a scan for positively selected genes in the genomes of humans and chimpanzees found that a large number of genes involved in apoptosis show strong evidence for positive selection 
. Further, it has been hypothesized that natural selection for increased cognitive ability in humans led to a reduced level of neuron apoptosis in the human brain 
Composite preservation statistics for KEGG pathways between human and chimp brain networks.
Detailed preservation analysis of KEGG pathways between human and chimp brain networks.
exhibits some dependence on module size. Since we want to compare module preservation irrespective of module size, we focus on the results for the
statistic (). A reviewer of this article hypothesized that gene sets (modules) known to be controlled by coexpression (such as Wnt, TGF-beta, SRF, interferon, lineage specific differentiation markers, and NF kappa B) would show stronger evidence of preservation than gene sets without a priori reason for suspecting such control (calcium signaling, MAPK, apoptosis, chemotaxis, endocytosis). Interestingly, the results for the
statistic largely validate this hypothesis. Specifically, the 4 most highly preserved pathways according to
are Wnt (controlled by coexpression), calcium (not controlled), Hedgehog (controlled), and Phosphatidylinositol (not commented upon). The 4 least preserved pathways are apoptosis (not controlled), TGF-beta (controlled), MAPK (not controlled), endocytosis (not controlled).
Since KEGG pathways are not defined via a clustering procedure it is not clear whether cluster preservation statistics are appropriate for analyzing this example. But to afford a comparison, we also report the findings for the IGP statistic 
. show that IGP identifies Phosphatidilinositol and TGF-beta as the least preserved modules while apoptosis genes are highly preserved. We find no significant relationship between the IGP statistic and our module preservation statistics
(). This example highlights that module preservation statistics can lead to very different results from cluster preservation statistics.
To understand which aspects of the pathways are preserved, one can study the preservation of density statistics () and of connectivity statistics (). According to
, the coexpresssion network formed by apoptosis genes is not preserved. It neither shows evidence of connectivity preservation (
) nor evidence of density preservation (
). The Hedgehog pathway also shows no evidence of density preservation (
) but it shows weak evidence of connectivity preservation (
). The relatively low preservation Z statistics of the Hedgehog pathway may reflect a higher variability due to a small module size (it contains only
genes while the other pathways contain at least 22 genes). To explore this further, we studied the observed preservation statistics, which are less susceptible to network size effects than the corresponding
statistics. The scatter plots in show the correlations
between eigengene based connectivity measures
between the two species. For the Hedgehog pathway, we find that
) which turns out to be higher than that of the TGF-
The lack of preservation of the apoptosis pathway cannot be explained in terms of low module size. shows that it has the lowest observed
This application outlines how module preservation statistics can be used to study the preservation of KEGG pathway networks. The analysis presented here is but a first step towards characterizing molecular pathway preservation between human and chimpanzee brains, and should be extended through more detailed analyses with additional data sets in the future. A limitation of our microarray data is that they measured expression levels in heterogeneous mixtures of cells. KEGG and GO (gene ontology) pathways all essentially describe interactions that take place within cells. So when data have been generated from a heterogeneous mixture of different cell types, it is possible that these relationships are somewhat obscured. It is not obvious that all of the elements of a KEGG pathway should be co-expressed, particularly since the pathways describe protein-protein interactions.
Relationships among module preservation statistics
In , we categorize the statistics according to which aspects of module preservation they measure. For example, we present several seemingly different versions of density and connectivity based preservation statistics. But for correlation network modules, close relationships exist between them as illustrated in . The hierarchical clustering trees in show the correlations between the observed preservation statistics in our real data applications. As input of hierarchical clustering, we used a dissimilarity between the observed preservation statistics, which was defined as one minus the correlation across all studied reference and test data sets. Overall we observe that statistics within one category tend to cluster together. We also observe that separability appears to be weakly related to the density and connectivity preservation statistics. Cross-tabulation statistics correlate strongly with density and connectivity statistics in the study of human and chimpanzee brain data, but the correlation is weak in the study of sex differences in human brain data.
Relationships between module preservation statistics based on applications.
We derive relationships between module preservations statistics in the sixth section of Supplementary Text S1
. In particular, the geometric interpretation of correlation networks 
can be used to describe situations when close relationship exist among the density based preservation statistics (
), among the connectivity based preservation statistics (
), and between the separability statistics (
). These relationships justify aggregating the module preservation statistics into composite preservation statistics such as
(Equation 1) and
Simulation studies and comparisons
To illustrate the utility and performance of the proposed methods, we consider 7 different simulation scenarios that were designed to reflect various correlation network applications. An overview of these simulations can be found in . A more detailed description of the simulation scenarios is provided below.
Design and main results of simulation studies of module preservation.
shows the performance grades of module preservation statistics in the different simulation scenarios. The highest grade of
indicates excellent performance. We find that the proposed composite statistics
) perform very well in distinguishing preserved from non-preserved modules. In contrast, cross-tabulation based statistics only obtain a mean grade of
. Since several simulation scenarios test the ability to detect connectivity preservation (as opposed to density preservation), it is no surprise that on average cluster validation statistics do not perform well in these simulations. For example, the IGP cluster validation statistic () obtains a mean grade of
across the scenarios. But the IGP performs very well (grade 4) when studying the preservation of strongly preserved clusters (scenario 2).
Overview of the performance of various module preservation statistics in our simulation studies.
Comparison of summary preservation statistics to in group proportion.
also shows the performance of individual preservation statistics. Note that density based preservation statistics perform well in scenarios 1 through 5 but fail in scenarios 6 and 7. On the other hand, all connectivity based statistics perform well in scenarios 6 and 7. The relatively poor performance of
is one of the reasons why we did not include it into our composite statistics.
In the following, we describe the different simulation scenarios in more detail.
- In the weak preservation simulation scenario, we simulate a total of 20 module in the reference data. Each of the reference modules contains 200 nodes. But only of the modules are simulated to be preserved in the test network. We call it the weak preservation simulation since the intramodular correlations of preserved modules are relatively low. The intramodular correlations of non-preserved modules are expected to be zero. Note that the summary statistic successfully distinguishes preserved from non-preserved modules (second column of ), with for of the preserved modules. Similarly, the statistic distinguishes preserved from non-preserved modules (third column of ). In comparison, the IGP permutation p-value (fourth column of ) is less successful: only of the preserved modules pass the Bonferroni-corrected threshold; of the modules that pass the threshold, are preserved and are non-preserved. In this simulation we observe a moderate relationship between the observed IGP and , with Pearson correlation .
- In the half-preserved simulation scenario, we simulate modules of varying sizes (between and nodes), labeled 1–10. Modules 1–5 are preserved in the test set, while modules 6–10 are not preserved. All 5 preserved modules have , and all non-preserved modules have . Likewise, separates preserved and non-preserved modules. Permutation p-values of IGP are also successful with respect to the Bonferroni-corrected threshold. In this simulation we observe a strong correlation between IGP and : .
- In the permuted simulation scenario, none of the 10 modules are preserved. Specifically, we simulate modules of varying sizes in the reference set and modules of the same sizes in the test set but there is no relationship between the modules: the module membership is randomly permuted between the networks. The low value of the summary preservation statistic accurately reflects that none of the modules are preserved. In contrast, the IGP permutation p-value for 2 of the 10 modules is lower than the Bonferroni threshold . In this simulation the correlation between IGP and is not significant.
- In the half-permuted simulation scenario, we simulate modules labeled 1–10 in the reference set. Modules 1–5 are preserved in the test set, while modules 6–10 are not. The test set contains modules 6′–10′ of the same sizes as modules 6–10, but their module membership is randomly permuted with respect to the modules 6–10. The summary preservation statistic is quite accurate: all preserved modules have and non-preserved modules have . The observed values of the IGP statistic are highly correlated (, ) with but the IGP permutation p-values do not work well: 2 preserved modules have an IGP p-value above .
- In the intramodular permuted scenario, we simulate modules whose density is preserved but whose intramodular connectivity is not preserved. Specifically, we simulate a total of modules labeled 1–10 in the reference set. The density of modules 1–5 is preserved in the test set but the node labels inside each module are permuted, which entails that their intramodular connectivity patterns is no longer preserved in the test network. For modules 6–10 neither the density nor the connectivity is preserved. Both composite statistics and work well though not as good as in the previous studies. Both composite statistics successfully detect the density preservation. IGP performs quite well: it misclassifies only one non-preserved module as preserved. In this simulation we observe a strong correlation between IGP and : .
- In the pathway simulations scenario, we simulate (preserved) modules whose connectivity patterns are preserved but whose density is not. Further, we simulate modules for which neither connectivity nor density are preserved. In the following description, we refer to the modules from scenario 4 as clusters to distinguish them from the non-cluster modules studied here. The preserved (non-preserved) modules of the pathway scenario are created by randomly selecting nodes from the preserved (non-preserved) clusters in scenario 4. Thus, the preserved modules contain nodes from multiple preserved clusters of scenario 4. Since the pairwise correlations between and within the preserved clusters (of scenario 4) are preserved, the intramodular connectivity patterns of the resulting pathway modules are preserved in the test network. But since nodes from different clusters may have low correlations, the density of the pathway modules tends to be low. The two pathway simulations differ by the module sizes: in the small scenario, modules range from 25 to 100 nodes; in the large scenario, modules range from 100 to 500 nodes. Because module membership is trivially preserved between reference and test networks, cross-tabulation statistics are not applicable. The composite statistics and distinguish preserved from non-preserved modules ( since they also measure aspects of connectivity preservation. By considering individual preservation statistics, we find that all connectivity preservation statistics successfully distinguish preserved from non-preserved modules. As expected, density based statistics and the IGP statistic fail to detect the preservation of the connectivity patterns of the preserved modules () but these statistics correctly indicate that the density is not preserved. Detailed results are provided in Supplementary Text S6.
Additional descriptions of the simulations can be found Supplementary Text S6
and in Supplementary Table S5
. As caveat, we mention that we only considered 7 scenarios that aim to emulate selected situations encountered in co-expression networks. The performance of these preservation statistics may change in other scenarios. A comprehensive evaluation in other scenarios is needed but lies beyond our scope. R software tutorials describing the results of our simulation studies can be found on our web page and will allow the reader to compare different methods using our simulated data.