There are a total of 40 new test cases. They are listed in , along with the existing cases from Benchmark 2.0. Six of these test cases are identical or homologous to CAPRI targets, indicated in the legend of . To assign difficulty levels of the test cases, we used the degree of conformational changes, as measured by Interface Cα-RMSD (I-RMSD18
) and fraction of non-native residue contacts (fnon-nat16
), of the unbound structures fitted onto the bound structures. Specifically, the rigid-body cases are cases with I-RMSD ≤ 1.5 Å and fnon-nat
≤ 0.4, the difficult cases are cases with I-RMSD > 2.2 Å, and the medium cases are all remaining cases (i.e., with 1.5 Å < I-RMSD ≤ 2.2 Å, or I-RMSD < 1.5 Å and fnon-nat
> 0.4). We used Cα RMSD instead of backbone RMSD (the latter is used in the CAPRI evaluation) because we have been using Cα RMSD since the creation of the Benchmark, which predates CAPRI.
We use this difficulty classification to quantify the extent of conformational change around the binding interface, which broadly affects most docking methods. For Benchmark 2.0 we assigned difficulty level based on the number of possible high-quality hit predictions (as measured by the CAPRI criteria16
) attainable using rigid-body docking on a grid. To remove possible bias due to this method and to simplify the classification, we opted to utilize the I-RMSD and fnon-nat
metrics for the new cases, selecting cutoffs to maintain consistency among the new cases and those from Benchmark 2.0. Besides conformational changes, other factors such as the size and hydrophobic/electrostatic composition of the interface, as well as the available experimental data on the complex, can also affect the difficulty of a test case.17,18
In total, Benchmark 3.0 has 88 rigid cases, 19 medium cases and 17 difficult cases. There are two difficult cases with large hinge movement (1E4K and 1IRA). provides the average values of the three classes in terms of I-RMSD and fnon-nat. We have also included the statistics of the fraction of native residue contacts (fnat16) even though it does not provide additional values to the above two metrics, because it is used in CAPRI evaluation.
Statistics on the Three Difficulty Groups in Benchmark 3.0
In addition to the difficulty assessment, we have classified the new test cases into three biochemical categories: Enzyme-Inhibitor (E; 9 cases), Antigen-Antibody (A; 3 cases) and Others (O; 28 cases), as with previous Benchmark versions.2,3
This information is provided in . We corrected the category assignments for two Benchmark 2.0 cases (1IJK and 1FQ1) from O to E.