Anomalous networks under the multispecies coalescent: theory and prevalence
Authors:
- 19 February 2024
- journal article
- research article
- Published by Springer Nature in Journal of Mathematical Biology
- Vol. 88 (3) , 1-36
Abstract
Reticulations in a phylogenetic network represent processes such as gene flow, admixture, recombination and hybrid speciation. Extending definitions from the tree setting, an anomalous network is one in which some unrooted tree topology displayed in the network appears in gene trees with a lower frequency than a tree not displayed in the network. We investigate anomalous networks under the Network Multispecies Coalescent Model with possible correlated inheritance at reticulations. Focusing on subsets of 4 taxa, we describe a new algorithm to calculate quartet concordance factors on networks of any level, faster than previous algorithms because of its focus on 4 taxa. We then study topological properties required for a 4-taxon network to be anomalous, uncovering the key role of \(3_2\)-cycles: cycles of 3 edges parent to a sister group of 2 taxa. Under the model of common inheritance, that is, when each gene tree coalesces within a species tree displayed in the network, we prove that 4-taxon networks are never anomalous. Under independent and various levels of correlated inheritance, we use simulations under realistic parameters to quantify the prevalence of anomalous 4-taxon networks, finding that truly anomalous networks are rare. At the same time, however, we find a significant fraction of networks close enough to the anomaly zone to appear anomalous, when considering the quartet concordance factors observed from a few hundred genes. These apparent anomalies may challenge network inference methods.
Keywords
All Related Versions
Funding Information
- National Science Foundation (1902892, 2023239, 2051760)
- National Institute of General Medical Sciences (P20GM103395)
- Simons Foundation (735923LPI)
- Wisconsin Alumni Research Foundation (H. I. Romnes faculty fellowship)
This publication has 51 references indexed in Scilit:
- Ancient Admixture in Human HistoryGenetics, 2012
- UNEXPECTEDLY MANY EXTINCT HOMININSEvolution, 2012
- The Probability of a Gene Tree Topology within a Phylogenetic Network with Applications to Hybridization DetectionPLOS Genetics, 2012
- Estimating hybridization in the presence of coalescence using phylogenetic intraspecific samplingBMC Ecology and Evolution, 2011
- BUCKy: Gene tree/species tree reconciliation with Bayesian concordance analysisBioinformatics, 2010
- Identifying the rooted species tree from the distribution of unrooted gene trees under the coalescentJournal of Mathematical Biology, 2010
- Extended Newick: it is time for a standard representation of phylogenetic networksBMC Bioinformatics, 2008
- Discordance of Species Trees with Their Most Likely Gene TreesPLOS Genetics, 2006
- GENE TREE DISTRIBUTIONS UNDER THE COALESCENT PROCESSEvolution, 2005
- Gene Trees in Species TreesSystematic Biology, 1997