Below is an
official ranking table published by the CASP12
contact prediction assessors in their paper.
This table shows the rankings of 6 different metrics and the assessors chose to rank the predictors by the 2nd column.
Our method RaptorX-Contact has overall
the best rank, although it was not fully
implemented when participating in CASP12 (May 1, 2016 -- July 20, 2016). A more
mature implementation of our method has much better performance than that used
in CASP12. See our PLoS CB and CASP12
papers for details. In this table, Full List means that only the submitted
contacts with probability score > 0.5 are considered, but this is logically
flawed since the predicted probability scores can be manipulated (without
changing the predicted contact ranking order) so that all the predicted contacts
would be considered. The CASP12 assessors also proposed a logically flawed
metric F1(prob). Here is the explanation
why F1(prob) and Full List are flawed.
We also
generated a ranking list in terms of the total F1 score when top L/5 long- and
medium-range contacts are evaluated. Ranking does not change much when top L/2
contacts are evaluated. However, it does not necessarily mean that F1 is the
best metric to rank contact predictors since a predicted contact map with a
higher F1 score may not lead to a better 3D modeling. Ultimately, we need to
evaluate how much a predicted contact map can help with 3D modeling, but this
is very challenging since there is no a single criterion to choose top
predicted contacts to assist folding.
|
Group Name |
Rank |
F1 |
Precision |
Recall |
|
RaptorX-Contact |
1 |
12.386 |
55.831 |
7.029 |
|
MetaPSICOV
|
2 |
10.919 |
51.307 |
6.155 |
|
iFold_1 |
3 |
10.917 |
50.916 |
6.153 |
|
MULTICOM-CONSTRUCT |
4 |
10.847 |
50.449 |
6.121 |
|
Pcons-net
|
5 |
10.810 |
49.536 |
6.113 |
|
RBO-Epsilon |
6 |
10.736 |
48.601 |
6.081 |
|
FALCON_COLORS |
7 |
10.387 |
47.253 |
5.880 |
|
Yang-Server |
8 |
10.186 |
46.460 |
5.763 |
|
Deepfold-Contact
|
9 |
10.003 |
46.442 |
5.644 |
|
PconsC31
|
10 |
9.734 |
45.728 |
5.483 |
|
IGBteam
|
11 |
9.428 |
45.596 |
5.288 |
|
MULTICOM-CLUSTER |
12 |
9.114 |
42.661 |
5.138 |
|
naive
|
13 |
9.016 |
42.009 |
5.085 |
|
raghavagps
|
14 |
9.005 |
40.269 |
5.110 |
|
Shen-Group
|
15 |
8.984 |
40.938 |
5.085 |
|
AkbAR
|
16 |
8.901 |
41.417 |
5.021 |
|
MULTICOM-NOVEL |
17 |
8.235 |
36.630 |
4.680 |
|
Zhang_Contact |
18 |
8.111 |
38.259 |
4.570 |
|
PLCT
|
19 |
7.690 |
35.133 |
4.356 |
|
PconsC2
|
20 |
7.347 |
34.976 |
4.132 |
|
Distill
|
21 |
5.889 |
27.762 |
3.315 |
|
FLOUDAS_SERVER |
22 |
5.371 |
24.255 |
3.041 |
|
BG2
|
23 |
5.073 |
25.475 |
2.831 |
|
BAKER_GREMLIN |
24 |
5.072 |
25.451 |
2.831 |
|
myprotein-me |
25 |
4.442 |
21.699 |
2.485 |
|
ZHOU-SPARKS-X |
26 |
3.982 |
19.030 |
2.236 |
|
Wang4
|
27 |
3.547 |
17.193 |
1.988 |
|
Wang2
|
28 |
2.664 |
11.725 |
1.517 |
|
FLOUDAS
|
29 |
2.566 |
11.617 |
1.452 |
|
RRCpred
|
30 |
2.196 |
10.051 |
1.243 |
|
Wang3
|
31 |
2.163 |
9.359 |
1.235 |
|
Wang1 |
32 |
1.825 |
7.733 |
1.046 |
|
KScons
|
33 |
0.505 |
2.521 |
0.281 |
|
FONT |
34 |
0.160 |
0.857 |
0.088 |