EVA currently addresses the following protein structure prediction categories (Table ): comparative modelling (EVA-CM); inter-residue contact prediction (EVA-con); secondary structure prediction (EVA-sec); and threading (EVA-FR). In the following, we sketch the measures for accuracy employed for each category. Note that the detailed definitions of the scores are available through the EVA web sites.
Prediction methods evaluated by EVA
Implements a small number of criteria—arranged hierarchically from coarser to finer—that measure the accuracy of a comparative model. The assessed aspects of a model include fold type, alignment, whole structure, core structure, loops and side-chains. Final ranking is reported using the ‘pairwise’ comparison of prediction servers (3
). From May 2000 to January 2003, predictions were collected from five different servers, resulting in 20
957 submitted models for 9050 different PDB chains. On average, 2.3 models were predicted per chain.
Evaluates inter-residue contact/distance predictions. A number of servers predict contacts directly, using neural networks of different kinds trained on contact maps (10
). There are also predictions of contacts based on assembled structures (12
). The current evaluation criteria implemented in EVA-con include: (i) accuracy—the number of the correctly predicted contacts divided by the total number of predicted contacts (13
); (ii) improvement over random—the calculated accuracy divided by the random accuracy (13
); (iii) distance distribution of the predicted contacts—the weighted harmonic average difference between the predicted contact distance distribution and the all-pairs distance distribution (14
); and (iv) delta evaluation—the percentage of correctly predicted contacts that are within a certain number (delta) of residues of the experimental contact, measured along the sequence (15
). EVA-con may also be used to evaluate ab initio
, fold recognition and comparative modelling servers by transforming models into intra-molecular contacts between the corresponding C-beta atoms (C-alpha for Gly) with a 8
Evaluates protein secondary structure predictions. Secondary structures are assigned from 3D structures through DSSP (16
) and STRIDE (17
). EVA-sec measures accuracy by: (i) per-residue accuracy (18
)—percentage of residues correctly predicted in one of the three states (helix, strand or other); (ii) per-segment accuracy (18
)—average overlap between segments (methods that get most of the segment cores right are generally more useful than those that get some of the entire segments right); and (iii) accuracy of predicting structural class—percentage of proteins correctly predicted in one of the following classes: all-alpha, all-beta, alpha/beta and others (20
). Rankings are presented using both the ‘common subset’ and ‘pairwise’ comparison approaches.
Currently evaluates models only for novel sequences (i.e. proteins for which PSI-BLAST searches do not reveal similarity to a known structure). Since there is no single measure that can comprehensively assess the quality of threading models, EVA-FR employs an array of alignment dependent and alignment independent measures (22
). For most of the measures used, two aspects of server performance are considered: (i) the ability to produce good models for each target (rank analysis); and (ii) the ability to assign reliable scores to its models, measured through Receiver Operator Characteristics curves (ROC; note this aspect is often referred to with ‘fold recognition’). Methods are ranked through both the ‘common subset’ and ‘pairwise’ comparison approaches.