|Home | About | Journals | Submit | Contact Us | Français|
Automated annotation of protein function is challenging. As the number of sequenced genomes rapidly grows, the overwhelming majority of protein products can only be annotated computationally. If computational predictions are to be relied upon, it is crucial that the accuracy of these methods be high. Here we report the results from the first large-scale community-based Critical Assessment of protein Function Annotation (CAFA) experiment. Fifty-four methods representing the state-of-the-art for protein function prediction were evaluated on a target set of 866 proteins from eleven organisms. Two findings stand out: (i) today’s best protein function prediction algorithms significantly outperformed widely-used first-generation methods, with large gains on all types of targets; and (ii) although the top methods perform well enough to guide experiments, there is significant need for improvement of currently available tools.
The accurate annotation of protein function is key to understanding life at the molecular level and has great biomedical and pharmaceutical implications. However, with its inherent difficulty and expense, experimental characterization of function cannot scale up to the vast amount of sequence data already available.1 The computational annotation of protein function has therefore emerged as a problem at the forefront of computational and molecular biology.
Many solutions have been proposed in the last four decades,2-10 yet the task of computational functional inference in a lab often relies on traditional approaches such as domain identification or finding BLAST11 hits among proteins with experimentally determined function. Recently, the availability of genomic-level sequence information for thousands of species, coupled with massive high-throughput experimental data, has created new opportunities for function prediction. A number of methods have been proposed to exploit these data, including function prediction from amino acid sequence,12-16 inferred evolutionary relationships and genomic context,17-21 protein-protein interaction networks,22-25 protein structure data,26-28 microarrays,29 or a combination of data types.30-34 With the large number of methods available, an unbiased evaluation can provide insight into the ability of different tools to characterize proteins functionally and guide biological experiments. So far, however, a comprehensive assessment incorporating a large and diverse set of target sequences has not been conducted due to practical difficulties in providing an accurately annotated target set.
In this report, we present the results of the first Critical Assessment of protein Function Annotation (CAFA) experiment, a worldwide effort aimed at analyzing and evaluating protein function prediction methods. Although protein function can be described in multiple ways, we focus on classification schemes provided by the Gene Ontology (GO) Consortium.35 Over the course of 15 months, 30 teams associated with 23 research groups participated in the effort, testing 54 function annotation algorithms. These methods were evaluated on a target set of 866 protein sequences from eleven species.
Protein function is a concept that can have different interpretations in different biological contexts. Generally, it describes biochemical, cellular, and phenotypic aspects of the molecular events that involve the protein, including how they interact with the environment (e.g. small compounds or pathogens). From the various classification schemes developed to standardize descriptions of protein function, we chose Molecular Function and Biological Process categories from the Gene Ontology (GO). Each category in GO is a hierarchical set of terms and relationships among them that capture functional information so that it facilitates computation and can be interpreted by humans. GO’s consistency across species and its widespread adoption make it suitable for large-scale computational studies. In CAFA, given a new protein sequence, the task of a protein function prediction method is to provide a set of terms in GO along with the confidence scores associated with each term.
The experiment was organized as follows. A set of 48,298 proteins lacking experimentally validated functional annotation was provided to the community four months before the submission deadline for predictions (Fig. 1). Proteins were annotated by the predicting groups and these annotations were submitted to the assessors. After the submission deadline, GO experimental annotations for those sequences were allowed to accumulate over a period of eleven months. Methods were then evaluated on 866 targets from eleven species that had accumulated functional annotations during the waiting period (Supplementary Table 1). The Swiss-Prot database36 was selected as the gold standard because of its relatively high reliability.37
The selection of proteins was ineluctably biased due to experimentalist and annotator choice during the evaluation timeframe. Thus, the set of targets was first analyzed to establish that it was representative of those sequences experimentally annotated before the submission deadline. In terms of organismal representation, the eukaryotic targets provided reasonable coverage of taxa (Fig. 1). In contrast, the set of prokaryotic targets was heavily biased towards Escherichia coli K-12, with 43 annotated sequences from other organisms. The distribution of terms over the target sequences was representative of the annotations in Swiss-Prot (data not shown); however, we note that in the Molecular Function category a large fraction of target sequences (38%) were associated with “protein binding” as their most specific term. The distribution of term depths over all targets is shown in Supplementary Fig. 1 for both ontologies.
The quality of protein function prediction can be measured in different ways, which reflect differing motivations for understanding function. In some cases, imprecise experimental characterization means that it is not entirely clear if a prediction is correct or incorrect. For CAFA, we principally report a simple metric, the maximum F-measure (Fmax; Online Methods), which considers predictions across the full spectrum from high to low sensitivity. This approach, however, has limitations, such as penalizing specific predictions as discussed in Discussion. We note that the choice of evaluation metric differentially impacts different prediction methods, depending upon their application objectives.
Top predictor performance, based on maximum F-measure and calculated over all targets, is shown in Fig. 2 (the precision/recall curves are shown in Supplementary Fig. 2). All methods are compared with two baseline tools: (i) BLAST, where all GO terms of an experimentally annotated sequence (template) from Swiss-Prot were transferred to the target sequence such that the scores equaled pairwise sequence identity between the template and the target (terms with multiple hits retained the highest score) and (ii) Naïve, where each GO term for each target was scored with the relative frequency of this term in Swiss-Prot over all annotated proteins (Online Methods). We also evaluated the quality of PSI-BLAST predictions but found that it did not provide any advantage over BLAST: specifically, for Molecular Function; and for Biological Process. We believe that the improved ability of PSI-BLAST to identify remote homologs has been canceled out by its re-ranking of close hits.
There is a significant performance difference in the ability to predict the two GO categories (Molecular Function vs. Biological Process). This can be partly explained by the topological differences between the ontologies (number of terms: 8728 vs. 18982; branching factor: 5.9 vs. 6.4; maximum depth: 11 vs. 10; number of leaf terms: 7003 vs. 8125, respectively). However, more fundamentally, terms in the Biological Process ontology are associated with a more abstract level of function. Such terms are less likely to be predictable solely from amino acid sequence, which was the data source used by most methods in this experiment and may critically depend on the cellular and organismal context.
We divided the target sequences into easy and difficult. A target was considered easy if it had a 60% or higher sequence identity with any experimentally annotated protein. The threshold of 60% was manually chosen after plotting the distribution of sequence identities between targets and annotated proteins (Supplementary Fig. 4). This resulted in 188 easy and 343 difficult targets in the Molecular Function category and 247 easy and 340 difficult targets in the Biological Process category. Supplementary Fig. 5 shows the precision/recall curves for both categories. Perhaps unsurprisingly, BLAST outperformed Naïve in the easy target category, while their performance was similar for the difficult targets. More importantly, however, because of the similar performance among top-ranked predictors over easy and difficult targets, the sequence identity-based classification of targets does not seem to accurately reflect the uncertainty associated with a protein’s true function (except for BLAST). This outcome is surprising and may be caused by the ability of the methods to compensate for the differences in sequence similarity of the best hit by exploiting multiple sequence hits as well as other data sources.
Supplementary Fig. 6 shows prediction performance for the eukaryotic and prokaryotic targets. Performance is generally similar in the Molecular Function category, with prokaryotic targets exhibiting higher prediction accuracy in the Biological Process category. We believe this is because most prokaryotic targets came from E. coli for which reliable experimental data are available, whereas the data for eukaryotic targets come from sources with highly variable coverage and quality. It is important to note that the particular calculation of precision and recall (see Online Methods) has adversely impacted methods that predicted only on eukaryotic targets (BMRF, ConFunc, GOstruct, Tian Lab) and resulted in lower overall performance for these methods. Detailed results for eukaryotic and prokaryotic targets, as well as several individual organisms are shown in Supplementary Figs. 6-7.
We further separated targets into sequences containing a single domain vs. sequences containing multiple protein domains, with domains defined according to Pfam-A classification38 (targets without any Pfam-A hits were grouped together with single domain proteins). Multi-domain proteins were generally longer; however, they were not associated with more functional terms than single-domain proteins. By analyzing the performance of the top ten methods in each category, we found that although the overall accuracy was higher on single-domain proteins, results were significant only in the Molecular Function category and for eukaryotic targets (P = 1.4·10−5; n = 10; paired t-test; Fig. 3). While generally not surprising, the higher performance on single-domain proteins further emphasizes the need for developing methods that can optimally combine sequence information from multiple domains along with other information to produce a relatively small set of predicted terms.
The ability of methods to predict individual GO terms was assessed by calculating the area under the ROC curve (AUC; Online Methods). To more confidently assess the performance in predicting individual terms, we only considered terms for which at least 15 targets were annotated. Average AUC values were then calculated from the top-five performing models in each ontology, excluding those models that only provide single-score predictions.
Using the above criteria we were able to calculate average AUC values for 28 Molecular Function and 223 Biological Process terms (Supplementary Table 2). We found a clear distinction between the average AUC of Molecular Function terms generally associated with catalytic and transporter activity, and those associated with binding. In general, the prediction of terms associated with binding showed lower AUC values, even though proteins were biased towards being annotated with binding terms. Among the Biological Process terms, we found, as expected, low AUC values associated with less specific terms such as “locomotion”, “cellular process”, and “response to stress”. We also found that prediction of terms associated with cell adhesion, metabolic process, transcription, and the regulation of gene expression showed high performance. We tested whether high predictor AUC on individual terms was due to high levels of sequence similarity among sequences experimentally annotated with those terms and found a moderate level of correlation (data not shown).
Here we illustrate some challenges associated with computational protein function prediction. We provide a detailed analysis of the human mitochondrial polynucleotide phosphorylase 1 (hPNPase; PNPT1), a large (783aa) protein with seven Pfam domains (Fig. 4A). Human PNPase is characterized by several experimentally determined functions, making it an attractive target to evaluate the performance of prediction methods. hPNPase belongs to a family of exoribonulceases, which hydrolyze single-stranded RNA in the 3’-to-5’ direction. In complex with other components of the mitochondrial degradasome it mediates the translocation of small RNAs into the mictochondrial matrix.39 It is also proposed to be involved in several biological processes including cell-cycle arrest,40 cellular senescence, and response to oxidative stress.41
Due to its involvement in several molecular functions and biological processes, the comprehensive and accurate listing of functions of hPNPase is a challenging task. Furthermore, while polynucleotide phosphorylase 1 is prevalent in bacteria and eukarya, it has accumulated several lineage-specific functions. Specifically, while bacterial and chloroplast PNPase have demonstrated exoribonuclease and polyadenylation activities, hPNPase functions predominantly as an RNA importer,39 with exoribonuclease activity shown only in vitro.42 Finally, hPNPase is a mitochondrial protein found in the inter-membrane matrix. Taken together with its involvement in the rRNA import process, this suggests the need to predict cellular compartment as part of a comprehensive understanding of function.
Figure 4B shows the experimental GO term annotation of hPNPase as well as the terms predicted by a representative set of the top-ten performing methods. Within the Molecular Function terms, none of the methods predicted poly(U/G) RNA binding43 or micro RNA binding. However, most methods that did predict function correctly predicted 3’-5’ exoribonuclease activity and polyribonucletide nucelotidyltransferase activity. It should be noted that poly(U/G) binding and micro RNA binding are not common throughout the PNPase lineage. This may be the reason why none of the programs predicted these terms.
In the Biological Processes category, the most prominent function of hPNPase in the literature is the import of nuclear 5S rRNA into the mitochondrion;39 indeed, it is hypothesized that this is the reason for hPNPase’s location in the inter-membrane matrix. However, this function, along with other important terms, such as cellular senescence, was not predicted by any of the top-performing methods at the optimal threshold levels. Generally, the biological process predictions were highly non-specific for most models. In sum, the multi-domain architecture of hPNPase, its pleiotropy, and the different functions it assumes in different taxa all contribute to the challenge of correctly predicting its function.
Protein function is difficult to predict for several reasons. First, function is studied from various aspects and at multiple levels; e.g. it describes the biochemical events involving the protein, and also how each protein affects pathways, cells, tissues, and the entire organism. Second, protein function and its experimental characterization are context-dependent: a particular experiment is unlikely to determine a protein’s entire functional repertoire under all conditions (e.g. temperature, pH, presence of interacting partners). Third, proteins are often multifunctional44 and promiscuous;45 in fact, 30% of the experimentally annotated proteins in Swiss-Prot have more than one leaf term in the Molecular Function ontology and 60% in the Biological Process ontology.16 Fourth, in addition to being incomplete, available functional annotations are error prone due to experiment interpretation or curation issues.37, 46 Finally, current efforts largely map protein function to gene names, thus confounding the functions of potentially diverse isoforms. Despite these challenges, the CAFA experiment revealed progress in automated function annotation over the past decade.
The first generation of function prediction methods performed a simple function transfer via pairwise sequence similarity; i.e. the most-similar annotated hit was used as the basis of function prediction.47 Several studies have been aimed at characterizing performance of these methods.3, 16, 48 The CAFA experiment provides evidence that the best algorithms universally outperform simple functional transfer. The experiment also showed that BLAST is largely ineffective at predicting functional terms related to the Biological Process ontology. This is possibly due to homologs assuming different biological roles in different tissues and organisms.49
The methods evaluated in CAFA exploited a variety of biological and computational concepts. Most methods exploited sequence alignments with an underlying hypothesis that sequence similarity is correlated with functional similarity. Recent studies have shown that this correlation is weak when applied to pairs of proteins16 and that domain assignments are not sufficient to resolve function.50 Therefore, the main challenge for the alignment-based methods was to devise ways of combining multiple hits or identified domains into a single prediction score. A number of methods exploited data beyond sequence similarity, e.g. types of evolutionary relationships, protein structure, protein-protein interactions, or gene expression data. The challenge for these methods was finding ways to integrate disparate data sources and properly handle incomplete and noisy data. For example, the protein-protein interaction network for yeast is nearly complete (although noisy), while the sets of available interactions for A. thaliana and X. laevis are rather sparse (but less noisy, given a smaller fraction of high-throughput data). Finally, some methods used literature mining, which could also be related to the task of retrieving the correct function rather than predicting it from the set of textual descriptions about a protein. As information retrieval is still a challenging research problem, it was useful to evaluate performance accuracy of the methods that exploited literature search.
On the computational side, most methods exploited machine learning principles, i.e. they typically found combinations of sequence-based or other features that correlated with a specific function in a training set of experimentally annotated proteins. While these methods automate the task of learning and inference, they also require experience in selecting classification models (e.g. a support vector machine), learning parameters, features, or the training data that would result in good performance. In addition, the sets of rules according to which these methods score new proteins may be difficult to interpret. Despite the added layer of complexity, machine learning generally played a positive role in increasing prediction accuracy. Thus, it may be expected that top-performing methods in the future will be based on well-founded principles of statistical learning and inference.
With few exceptions the same methods that performed well for the Molecular Function category also performed well in the Biological Process category; however, their overall performance in the latter category was inferior. We believe that this is because homologs may perform their biochemical roles in different pathways, and prediction methods are less able to discern those differences at this time. Because sequence similarity is less predictive of the biological roles of proteins, a key to improving the prediction of a protein’s biological function will depend on our ability to generate better-quality systems data and to develop computational tools that exploit them.
The choice of evaluation metrics was another interesting aspect of the experiment. We decided to use simple and easily interpretable metrics (Online Methods), although simple measures based on precision and recall have limitations in this domain. First, such metrics are sensitive to problems related to the non-uniform distribution of proteins over GO terms due to giving all terms equal weight. Second, proteins are weighted equally regardless of the depth of their experimental annotation, i.e. a correct prediction on a protein annotated with a shallow term (and its ancestors) is considered as good as a correct prediction on a protein annotated with a deep term. Third, a method that only reports high confidence deep annotations for a small number of proteins will be penalized (in terms of recall) compared to a method that annotates all proteins with frequently occurring general terms. Finally, in some cases, it is not clear whether to consider a prediction correct or erroneous; with our current approach, we consider only the experimental annotation and more general predictions to be correct. As such, correct and highly specific predictions will be penalized if the protein has been experimentally annotated only in a more generic way. For those reasons, we encourage the development of a diverse set of metrics to understand better the strengths and weaknesses of function prediction in different application contexts.
The CAFA experiment was designed to enable the community to periodically reassess the performance of computational methods as experimental evidence further accumulates. In addition, the large set of targets released to the community provided us with prediction scores for most proteins across multiple methods. If the experiment is repeated, we expect to be able to evaluate future methods against those that deposited predictions in the first CAFA experiment and therefore monitor progress in the field over time.
While the CAFA experiment has certainly seen positive outcomes, it is also clear that there is significant room for the improvement of protein function prediction. In the Molecular Function category, performance may be considered accurate. However, in the Biological Process category, the overall performance of the top scoring methods was below our expectations. This was true for any subset of targets. Another area in need of improvement is the availability of tools that can easily be used by experimental scientists and that could be maintained and upgraded on a regular basis. As the community moves beyond the initial algorithm development stage, there is a need to provide standalone tools (similar to the BLAST package) capable of predicting protein function at several different levels.
Given its significance, intellectual challenge, and the growing need for accurate functional annotations, protein function prediction is likely to remain an active and growing research field. As the quality of data improves and the number of experimentally annotated proteins grows, we expect that computational prediction will become more accurate. Based on the CAFA experiment, it seems that the most powerful methods will be those that will devise principled ways to integrate a variety of experimental evidence and weight different data appropriately and separately for each functional term. Novel ideas and approaches are necessary as well.
The CAFA experiment was conceived in the fall of 2009. The Organizing, Steering and Assessment Committees were designated by March 2010. During the same period a feasibility study was conducted to determine the rate at which experimental annotations accumulated in Swiss-Prot between 2007 and 2010. We concluded that a period of six months or more would result in annotations of at least 300-500 proteins, which would be sufficient for statistically reliable comparisons between algorithms. The experiment was announced in July 2010 and subsequently heavily advertised. The set of targets was announced on September 15, 2010 with a prediction submission deadline of January 18, 2011 (Fig. 1).
Predictors were asked to submit predictions for each target along with scores ranging between 0 and 1, which would indicate the strength of the prediction (ideally, posterior probabilities). To reduce the amount of data to be submitted, no more than 1,000 term annotations were allowed for each target. Prediction algorithms were also associated with keywords from a pre-determined set, which were used to provide insight into the types of approaches that performed well. A list of all participating teams, principal investigators, and methods is provided in Supplementary Table 3.
Initial comparative evaluation of models was conducted in July 2011 during the Automated Function Prediction (AFP) Special Interest Group (SIG) meeting associated with ISMB 2011 conference. This study provides the analysis on a set of targets from the Swiss-Prot database from December 14, 2011.
A set of 48,298 target amino acid sequences was announced in September 2010. Because our feasibility study showed that only a handful of species were steadily accumulating experimental annotations, target proteins were predominantly selected from those species. The targets contained all the sequences in Swiss-Prot from seven eukaryotic and eleven prokaryotic species that were not associated with any experimental GO terms. A protein was considered experimentally annotated if it was associated with GO terms having EXP, IDA, IMP, IGI, IEP, TAS, or IC evidence codes. An additional set of targets was announced consisting of 1,301 enzymes from multiple species and metagenomic studies that were the focus of the Enzyme Function Initiative project.51
January 18, 2011 was set as the deadline for the submission of function predictions. To exclude targets that had accumulated annotations prior to the submission deadline, annotated proteins were obtained from the January version of Swiss-Prot, GO,35 and UniProt-GOA52 databases. We refer to those sets of proteins as Swiss-Prot(t0), GO(t0) and GOA(t0), respectively.
The evaluation set of target proteins was later determined by downloading a newer version of the Swiss-Prot database, denoted as Swiss-Prot(t). The set of target proteins for the CAFA experiment was then selected using the following scheme:
Note that this experiment was designed to allow for reassessing algorithm performance at some later point in time.
Algorithms were evaluated in two scenarios: (i) protein-centric and (ii) term-centric. These two types of evaluations were chosen to address the following related questions: (i) what is the function of a particular protein and (ii) what are the proteins associated with a particular functional term.
The main evaluation metric in CAFA was the precision/recall curve. For a given target protein i and some decision threshold t [0,1], the precision and recall were calculated as
where f is a functional term in the ontology, Ti is a set of experimentally determined (true) nodes for protein i, and Pi(t) is a set of predicted terms for protein i with score greater than or equal to t. Note that f ranges over the entire ontology (separately for Molecular Function and Biological Process), excluding the root. Function I(·) is the standard indicator function. For a fixed threshold t, a point in the precision/recall space is then created by averaging precision and recall across targets. Precision at threshold t is calculated as
where m(t) is the number of proteins on which at least one prediction was made above threshold t. On the other hand, recall is calculated over all n proteins in a target set, i.e.
regardless of the prediction threshold. The maximum ratio between m(t) and n (over all thresholds t) is referred to as the prediction coverage. If a particular algorithm only outputs a fixed score (e.g. 1), its performance will be described by a single point in the precision/recall space, instead of by a curve.
For submissions with unpropagated functional annotations, the organizers recursively propagated all scores towards the root of the ontology such that each parent term received the highest score among its children. The annotations were propagated regardless of the type of relationship between terms. We note that it may be useful to associate different weights with different ontological terms and therefore reward algorithms that are better at predicting more difficult or less frequent terms. However, for simplicity, in our main evaluation, each term was associated with an equal weight of 1 (weighted precision/recall curves are shown in Supplementary Fig. 8).
The main appeal of the precision/recall evaluation stems from its interpretability; i.e. if for a particular threshold a method has precision of 0.7 at recall of 0.5, this indicates that on average 70% of the predicted terms will be correct and that about 50% of the true annotations will be revealed for a previously unseen protein. On the other hand, a limitation of this evaluation method is that the terms are not independent due to ontological relationships, and that the unequal level of specificity of functional terms at the same depth in the ontology was not taken into account.
To provide a single number for comparisons between methods, we calculated the F-measure (a harmonic mean between precision and recall) for each threshold and calculated its maximum value over all thresholds. More specifically, we used
For each functional term f, we calculated the area under the ROC curve (AUC) using a sliding threshold approach. The ROC curve is a plot of sensitivity (or recall) for a given false positive rate (or 1 – specificity). The sensitivity and specificity for a particular functional term f and threshold t were calculated as
where Pi(t) is the set of predicted terms for protein i with score greater than or equal to threshold t, and Ti is the set of true terms for protein i. Once the sensitivity and specificity for a particular functional term were determined over all proteins for different values of the prediction threshold, the AUC was calculated using the trapezoid rule. The AUC has a useful probabilistic interpretation: given a randomly selected protein associated with functional term f and a randomly selected protein not associated with f, the AUC is the probability that the former protein will receive a higher score than the latter protein.53
In addition to the methods implemented by the community, we used two additional methods as baselines. The first such method is based on BLAST11 hits to the database of proteins with experimentally annotated functions (roughly 37,000 proteins). The score for a particular term was calculated as the maximum sequence identity between the target protein and any protein experimentally annotated with that term. More specifically, if a particular protein was hit with the local sequence identity 75%, all its functional terms were transferred to the target sequence with the score of 0.75. If a term was hit with multiple sequence identity scores, the highest one was retained. BLAST was selected as a baseline method because of its ubiquitous use. We note that the same method was tested using the BLAST bit scores, which resulted in slightly better performance. In addition to BLAST, we also tested PSI-BLAST11 where the profiles were created using the most recent nr database and −j 3 −h 0.0001 parameters. These profiles were then searched against a database of experimentally annotated proteins with E-values used to rank the hits. The second baseline method, referred to as Naïve, used the prior probability of each term in the database of experimentally annotated proteins as the prediction score for that term. If a term “protein binding” occurs with relative frequency 0.25, each target protein was associated with score 0.25 for that term. Thus, the Naïve method assigned the same predictions to all targets.
We gratefully acknowledge Inbal Landsberg-Halperin for coining the term “CAFA”, Tracey Theriault for the initial graphical design of Figure 1, Gadi Schuster for illuminating discussions on hPNPase, as well as Andrea Facchinetti, Riccardo Velasco, Elisa Cilia, David A. Lee, Pankaj Vats, Ruma Banerjee, and Avinash Bayaskar for participation in various individual projects. Finally, we thank the three anonymous reviewers for their constructive comments and criticisms that improved the presentation and quality of this study.
The Automated Function Prediction Special Interest Group meeting at the ISMB 2011 conference was supported by the National Institutes of Health grant R13 HG006079-01A1 (PR) and Office of Science (BER), U.S. Department of Energy grant DE-SC0006807TDD (IF). Individual projects were partially supported by the following awards: NSF DBI-0644017 (PR), Marie Curie International Outgoing Fellowship PIOF-QA-2009-237751 (SR), NSF ABI-0965768 (AB-H), PRIN 2009 project 009WXT45Y Italian Ministry for University and Research MIUR (RC), NIH GM093123 (JC), FP7 "Infrastructures" project TransPLANT Award 283496 (A-J vD), BBSRC grant BB/G022771/1 (JG), UK Biotechnology and Biological Sciences Research Council (DTJ), Marie Curie Intra European Fellowship Award PIEF-GA-2009-237292 (DTJ), Bioinformatics Division, Department of Information Technology, MCIT, Govt. of India (RJ), NIH GM075004 (DK), NIH GM097528 (DK), NSF DMS0800568 (DK), NIH GM079656 (OL), NIH GM066099 (OL), NSF CCF 0905536 (OL), NSF DBI 1062455 (OL), EU, BBSRC and NIH Awards (CO), NSERC Discovery Award #298292-2009 (HS), Discovery Accelerator Award #380478-2009 (HS), CFI New Opportunities Award 10437 (HS), and Ontario’s Early Researcher Award #ER07-04-085 (HS), Biotechnology and Biological Sciences Research Council Award BB/F020481/1 (MW and MJS), Netherlands Genomics Initiative (YK, CtB), NIH LM00945102, NSF DBI-0965616, NSF DBI-0965768, NICTA (KV), NIH R01 GM071749 (SEB), DOE BER KP110201 (SEB), Alexander von Humboldt-Foundation (BR), NIH LM009722 (SDM), NIH HG004028 (SDM), NSF ABI-1146960 (IF).
Author contributionsPR and IF conceived the CAFA experiment, supervised the project, and wrote most of the manuscript. SDM participated in the design of and supervised the method assessment. WTC performed the analysis of feasibility of the experiment, most of the target and performance analysis, and contributed to writing. PR and WTC designed and produced figures. TRO developed the web interface, including the portal for submission and the storage of predictions. TRO and TW verified the assessment code and participated in analysis. AMS designed and performed the analysis of targets. AB, ML, PCB, SEB, CO, and BR steered the CAFA experiment, provided critical guidance, and participated in writing. The remaining authors participated in the experiment, provided writing and data for their methods, as well as contributed comments on the manuscript.