Experience sampling (ESM), diary, ecological momentary assessment (EMA), ambulatory monitoring, and related methods are part of a research tradition aimed at capturing the ongoing stream of individuals' behavior in real-world situations. By design, these approaches prioritize ecological validity. In this paper, we examine how the purported ecological validity these study designs provide may be compromised during data analysis. After briefly outlining the benefits of EMA-type designs, we highlight some of the design issues that threaten ecological validity, illustrate how the typical multilevel analysis of EMA-type data can compromise generalizability to "real-life", and consider how unobtrusive monitoring and person-specific analysis may provide for more precise descriptions of individuals' actual human ecology.