Leisure time activity is often positioned as a key factor in child development, yet we know relatively little about the causal significance of various specific activities or the magnitude of their effects. Here, we couple individual fixed effects and instrumental variable approaches in trying to determine whether specific forms of leisure contribute to gains in test performance over time. We merge a restricted access version of the Panel Study of Income Dynamics (PSID) Child Development Supplement (CDS), longitudinally collected from 1997 to 2007, with a database of over three million county-day observations of sunlight. We use this proxy for weather to instrument for the variation in physical, outdoor, sedentary, and screen time behaviors based on CDS time diaries. We find evidence that physical and outdoor activity positively influence math performance, while sedentary behavior and screen time exhibit the opposite effect. Moreover, the effect sizes range from a quarter to three quarters of a standard deviation per additional daily hour of activity, rendering them meaningful in a real world sense. Our stratified results indicate that children from less educated mothers and girls seem to be most sensitive to the effects of active and passive forms of leisure. We conclude with a descriptive examination of the trend lines between our data and the new 2014 CDS cohort, providing relevant contemporary context for our findings.