Berkeley PHOG: PhyloFacts Orthology Group Prediction Web Server Supplement 2: Detailed Description of Experiments

Ruchira Datta, Bushra Samad, Christoph Neyer and Kimmen Sjölander

Ortholog detection is essential in functional annotation of genomes, with applications to phylogenetic tree construction, prediction of protein-protein interaction and other bioinformatics tasks. The PHOG web server employs a novel algorithm to identify orthologs based on phylogenetic analysis. To assess the accuracy of PHOG, we used a set of human sequences and their predicted orthologs in three model organisms -- Mus musculus (mouse), Danio rerio (zebrafish) and Drosophila melanogaster (fruit fly) -- from the TreeFam-A resource as a gold standard benchmark. TreeFam-A uses a sophisticated ortholog-identification protocol (including tree reconciliation and manual curation) providing for a high-accuracy dataset. Mouse, zebrafish and fruit fly were selected since they had been targeted for analysis by both OrthoMCL-DB and InParanoid and represented a range of evolutionary distances.

Dataset selection: We chose a set of 100 human sequences from TreeFam-A meeting the following requirements:

Each sequence had to have orthologs in the TreeFam-A curated seed families from at least two of the three target species: mouse, zebrafish and Drosophila melanogaster.

No pair of sequences in the set could have a BLAST E-value < 1 (using a database size of 100,000) or share a common PFAM domain. This ensured that the dataset was filtered to remove potential homologs, so that method performance on this dataset should generalize.

We chose 100 human sequences as follows. We considered all human sequences for which TreeFam-A contained orthologs in at least two of the three target species: mouse, zebrafish, and fruitfly. This yielded a set of 353 human sequences. We then ran BLAST for each sequence in this set against the entire set, setting the database length to 100,000, and removed all sequences for which any other sequence in the set had a BLAST e-value <=1. This yielded a reduced set of 106 sequences. To further ensure that no pair of sequences were homologous, we searched for common PFAM domains by scoring each sequence against the PFAM-A library using hmmpfam (HMMs were downloaded from PFAM on January 13^th, 2009). We found that four of the sequences had no significant PFAM hits (based on an e-value <= 0.001 and length >= 30); these four sequences were eliminated. Two pairs of sequences shared PFAM domains; we eliminated one member of each pair. This yielded a set of 100 sequences, containing a total of 138 PFAM domains. See Table S2-1.

Ensuring coverage: To ensure that each sequence in the test set was included in PhyloFacts, we identified the region of each sequence corresponding to each PFAM domain and clustered homologs and constructed a phylogenetic tree using the PhyloBuilder software (using global-local mode for domain-based homology and 5 subfamily HMM iterations). Users can submit their own sequences to this pipeline at the PhyloBuilder webserver at http://phylogenomics.berkeley.edu/phylobuilder.

Assessing method performance: For each human sequence and for each method, we identified predicted orthologs in mouse, zebrafish and Drosophila melanogaster, and compared these predicted orthologs against the orthologs identified by TreeFam-A.

We used TreeFam Release 6.0, InParanoid Version 6.1, OrthoMCL Version 2, and PHOG as of March 1st, 2009. For each of these methods, we used the precomputed predictions that were available on the respective websites. Ensembl Release 52 was used to cross-reference the identifiers used by some of the methods, along with the cross-reference provided by TreeFam. PHOG predictions were made on the basis of a total of 455 trees containing the human sequences based on trees in the PhyloFacts resource as of March 1st, 2009.

Cross-referencing database identifiers: In order to compare the ortholog predictions made by the various methods, we had to bring all the identifiers to the same form. The form we used was the Ensembl gene identifier. We used the gene_stable_id, transcript, transcript_stable_id, translation, and translation_stable_id tables provided by Ensembl to convert Ensembl transcript or protein identifiers to gene identifiers. We also used the stable_id_event table provided by Ensembl to convert outdated Ensembl identifiers to their present forms. We used the ens_xref table provided by TreeFam to convert UniProt accessions to Ensembl identifiers.

Statistical measures of performance: For each method, we computed the recall (or sensitivity) as the fraction of TreeFam-A orthology pairs found.

For reference, recall and precision are defined as follows:

Recall = TP/(TP+FN)

Precision = TP /(TP+FP)

where a True Positive (TP) is an orthology pair included in TreeFam-A that is also predicted by a method, a False Negative (FN) is an orthology pair included in TreeFam-A that is not predicted by a method (i.e., it is missed by the method), and a False Positive (FP) is an orthology pair predicted by a method that is not included in TreeFam-A.

The precision measure reported in the main body of the paper, reported as P1 in Supplement 3, is most stringent. This measure labels all predicted orthologs (by InParanoid, OrthoMCL or PHOG) not found by TreeFam-A as False Positives. In other words, if TreeFam did not identify an ortholog, and OrthoMCL did, OrthoMCL was wrong, and the computed precision will decrease.

We also computed a less stringent measure of precision, reported as P2 in Supplement 3, based on a different definition of False Positive. This measure only calls a predicted orthology pair an error in the case where TreeFam-A selected a different sequence from the same species. In other words, if for a particular human sequence TreeFam-A selects no ortholog from mouse, but OrthoMCL does, OrthoMCL might possibly be correct. That orthology pair does not contribute to the OrthoMCL recall, but does not hurt its precision. On the other hand, if TreeFam-A selects an ortholog from mouse (M1) but OrthoMCL selects a different ortholog from mouse (M2), the (H,M2) pair is called a false positive.

Results: We present the results in the Figures S2-1 through S2-4 below. The point labeled PHOG-T(M) on each curve describes the performance at threshold value 0.09375, where the precision and recall for the human-mouse ortholog predictions are approximately equal. The point labeled PHOG-T(Z) on each curve describes the performance at threshold value 0.296875, where the precision and recall for the human-zebrafish ortholog predictions are approximately equal. The point labeled PHOG-T(D) on each curve describes the performance at threshold value 0.9375, where the precision and recall for the human-fruitfly ortholog predictions are approximately equal. Detailed results are provided in Supplement 3.