As exemplified by the Ebbinghaus illusion, the perceived size of an object can be significantly biased by its surrounding context. The phenomenon is experienced by humans as well as other species, hence likely evolutionarily adaptive. Here, we examined the heritability of the Ebbinghaus illusion using a combination of the classic twin method and multichannel functional near-infrared spectroscopy. Results show that genes account for over 50% of the variance in the strength of the experienced illusion. Interestingly, activations evoked by the Ebbinghaus stimuli in the early visual cortex are explained by genetic factors whereas those in the posterior temporal cortex are explained by environmental factors. In parallel, the feedforward functional connectivity between the occipital cortex and the temporal cortex is modulated by genetic effects whereas the feedback functional connectivity is entirely shaped by environment, despite both being significantly correlated with the strength of the experienced illusion. These findings demonstrate that genetic and environmental factors work in tandem to shape the context-dependent visual size illusion, and shed new light on the links among genes, environment, brain, and subjective experience.