Disagreement in FST estimators: A case study from sex chromosomes.

Sewall Wright developed FST for describing population differentiation and it has since been extended to many novel applications, including the detection of homomorphic sex chromosomes. However, there has been confusion regarding the expected estimate of FST for a fixed difference between the X- and Y-chromosome when comparing males and females. Here, we attempt to resolve this confusion by contrasting two common FST estimators and explain why they yield different estimates when applied to the case of sex chromosomes. We show that this difference is true for many allele frequencies, but the situation characterized by fixed differences between the X- and Y-chromosome is among the most extreme. To avoid additional confusion, we recommend that all authors using FST clearly state which estimator of FST their work uses.