We proposed a methodology for hit selection in genome-scale RNAi HTS based on a Bayesian framework. The proposed methodology aims to address issues arising from classical approaches including median ± k
MAD. One issue is the adjustment of error rate in multiple hypothesis testing. In this article, we construct Bayesian models to control FDR via the direct posterior probability approach (19
). The second issue is the decision of whether a plate-wise or experiment-wise analysis should be used. The results in both HIV and HCV siRNA screens show that selected hits using an experiment-wise analysis can be dominated by systematic errors in a plate. A plate-wise analysis is more robust in the presence of systematic errors, but can result in false negatives in plates containing a cluster of true hits. Our proposed Bayesian methodology allows data to be shared across the plates of an experiment in a plate-wise analysis, thus obviating the need for choosing either plate-wise or experiment-wise analysis. The third issue is the use of controls: Should the negative control or the majority of sample wells be used as the negative reference when calculating the center and variability of the raw data? How should the information from positive and negative controls with varying degrees of efficacy be incorporated into the hit selection strategy? The choice of prior in our Bayesian methods enables us to incorporate information from various controls. From the posteriors in the proposed models, our Bayesian approach maintains a balance between the contribution from the sample wells and control wells. In addition, outliers are identified and excluded in the estimation of parameters in our Bayesian methods; thus our methods are robust to outliers. Finally, the Bayesian approach provides each siRNA with more than a yes/no answer, but rather an estimate probability of being in each of three groups: activation, inhibition and no effect.
We applied the Bayesian methods to two real genome-scale RNAi screens with different features. The HCV screen did not have a true activation control whereas the HIV screen did. The inhibition controls had data variability larger than the negative control in the HIV screen whereas the inhibition controls had data variability smaller than the negative control in the HCV screen. The strength of positive controls in the two screens also differed from each other. By using the Bayesian methods for hit selection we describe here, we were able to address effectively the issues of hit selection common to both screens and to obtain a reasonable pool of selected hits for both screens despite the differences between the two.
We focus on two Bayesian models for hit selection in RNAi HTS assays. In Model 1, we construct the prior using a negative reference, whereas in Model 2, we construct the prior based on a negative reference, an activation control and an inhibition controls. We apply both Bayesian models for hit selection in an HIV siRNA primary screen and an HCV siRNA primary screen. The applications show that the hit selection results using our proposed methods, especially Model 1, are more reasonable than those using classical methods (). ROC analysis in the case studies shows that Model 1 is more powerful than the commonly used classical approaches. When the positive controls are moderately effective and of high quality, Model 2 is also powerful (B). Simulation studies show that, in an experiment without any enriched plates, in general, Bayesian Model 1 has the best performance especially in detecting hits with weak or moderate effects; while Bayesian Model 2 performs better than the experiment-wise median ± kMAD method and has a good performance in detecting hits with effects equal to or stronger than the positive controls (B1–5). In an experiment with 1 to 9 enriched plates, in general, Bayesian Model 1 has the best performance in detecting hits with weak or moderate effects, while Bayesian Model 2 performs equivalently to classical experiment-wise methods in detecting hits with strong effects (C1–5).
Bayesian Models 1 and 2 allow for three strategies to select hits: (i) fix the number of selected siRNAs, (ii) control a prespecific FDR or (iii) classify an siRNA based on its maximal posterior probability. In a typical primary siRNA HTS screen, one of the goals is usually to select a practical number (usually some number between 500 and 1500) of hits for further investigation. Therefore, there is a motivation to use the first strategy for hit selection in a primary screen. Our Bayesian methods allow selecting a fixed number of siRNAs with the lowest FDR. On the other hand, we cannot simply select a fixed number (say 1000) from every screen and ignore the error rate because different screens may have very different error rates. Thus, the second strategy may be more plausible especially when we want to control some fixed error rate in most screens. Considering both HIV and HCV screens discussed in this article, a reasonable choice of a fixed FDR is somewhere between 0.25 and 0.35, which allows us not only to fix a reasonable number of siRNAs for further analysis but also to control the FDR within reasonable range. The classifying strategy under Model 1 also works effectively in the two screens, which yields an FDR of around 0.30 and a reasonable number of selected hits (i.e. 1352 hits in the HIV screen and 741 hits in the HCV screen).
Model 1 does not use any information about the strength of positive controls. Consequently, the hit selection results using Model 1 are robust to the quality and strength of any positive controls used in the experiment. On the other hand, Model 2 incorporates the strength of positive controls. Thus, hit selection results using Model 2 are highly sensitive to the data quality and strength of positive controls adopted in the experiment. It is not uncommon that the strength of positive controls varies within or between experiments. It is also not uncommon that the data quality of positive controls is poor in some experiments. These facts make Model 1 more favorable than Model 2 for hit selection in RNAi HTS screens. On the other hand, the simulation studies show that Model 2 has a better performance in detecting hits with effects equal to or stronger than the positive controls when there are plates with enriched number of hits (C1–5). If the data quality of positive controls is good and the positive controls are moderately or fairly strongly effective, Model 2 may be more powerful (B). Thus, Model 2 may help us to identify siRNAs with a desired potency if there are high quality and reliable controls with a similar effectiveness in the screen.
In our Bayesian methods, we preestimate some parameters in the priors, resulting in a fast and efficient algorithm. There are two reasons for the preestimation. One is to reduce computational time because it is critical to keep computational time as low as possible when analyzing large data sets from high-throughput screens. The second reason is to focus on selecting hits only in plates with good data quality, because hit selection results are often misleading in plates with poor quality (24–29
). In the Bayesian models described in this article, we use the fact that the unconditional variance of an observed value should be larger than the expectation of its conditional variance (i.e.
) in a plate with good quality. In a plate with poor data quality, the pooled variance from different controls (i.e. estimate of σ2
) could be greater than the variability of sample wells (i.e. estimate of Var
)) in a plate. In such a case, the estimate of τ2
is zero (i.e.
); consequently, the FDR is unavailable in that plate when using the above Bayesian models.
One strategy to handle the situation with
is the use of quality control methods described in (24–29
) to assure all the plates have good quality during the experimental stage of a screen, thus preventing the occurrence of
. Another strategy is to select hits in the plates with
using non-Bayesian methods such as those described in (5–7
) and then to put a warning on all the hits from these plates. The third strategy is to build a Bayesian hierarchical model that puts priors on τ2
and/or other preestimated parameters. For example, we may construct a hierarchical model as follows:
are preestimated as in Model 1. This leads to the posteriors:
Based on these posteriors, we make inference on μi
using direct sampling in the Monte Carlo simulation.
There are two major issues with hierarchical models: one is the long computational time and the other is that plates with good or poor quality are treated in the same way. For example, for the simple hierarchical model described above, we need to run direct sampling on each of 25 000 siRNAs tested. The number of iterations in the Monte Carlo simulation for each siRNA has to be large. Otherwise, due to sampling error, we may obtain incorrect results: i.e. a more effective siRNA may have less chance to be selected as a hit than a less effective siRNA in the same plate. Even if only 2000 iterations are used for each siRNA, this will result in a total of 50 000 000 iterations when analyzing data from an HTS screen. Building more layers in a hierarchical model will result in even longer computational time, especially when there is at least one posterior requiring an indirect sampling in the Monte Carlo simulation (30
). Therefore, in genome-wide RNAi primary screens, we do not recommend the use of Bayesian hierarchical models (especially those requiring the use of indirect sampling) although we may use them in confirmatory screens.
In summary, hit selection in genome-scale RNAi research is important for identifying targets for a new class of therapeutics for treating human diseases. The development of novel effective and powerful analytic methods for hit selection is critical to glean useful information from mounds of data. In this article, we propose a methodology based on a Bayesian framework. The case studies show that our methods effectively address multiple issues in classical methods including error rate issues in multiple hypothesis testing, experiment-wise versus plate-wise analysis and incorporation of information from multiple reliable controls. Both simulation and case studies show that the Bayesian Model 1 we describe is more powerful than classical methods in detecting siRNAs with weak and moderate effects. Our Bayesian Model 2 performs better than classical experiment-wise methods when there are no plates with enriched number of hits and performs equivalently to experiment-wise classical methods in detecting hits with strong effects when there are plates with enriched number of hits in an RNAi HTS experiment.