Evolutionary biologists often predict the outcome of natural selection on an allele by measuring its effects on lifetime survival and reproduction of individual carriers. However, alleles affecting traits like sex, evolvability, and cooperation can cause fitness effects that depend heavily on differences in the environmental, social, and genetic context of individuals carrying the allele. This variability makes it difficult to summarize the evolutionary fate of an allele based solely on its effects on any one individual. Attempts to average over this variability can sometimes salvage the concept of fitness. In other cases evolutionary outcomes can only be predicted by considering the entire genealogy of an allele, thus limiting the utility of individual fitness altogether. We describe a number of intriguing new evolutionary phenomena that have emerged in studies that explicitly model long-term lineage dynamics and discuss implications for the evolution of infectious diseases.