Effective teaching critically relies upon effective evaluation because evaluation is required to gain understanding of existing abilities and, in turn, determine learning outcomes. Methods of effective evaluation are surprisingly elusive in many fields and this limits our understanding of which training methods are truly effective. In the present article, issues of effective evaluation are discussed in the context of "brain training," an exciting but much criticized field. Problems in test validity in the field of brain training, parallel those in many other fields; such as deficiencies in test reliability, teaching to the test, expectation effects, as well as statistical rigor. Here we review these issues and discuss how commonalities between the goals of evaluation and adaptive training procedures suggest a new paradigm that synthesizes evaluation and training. We suggest that continuous evaluation, where testing is integrated into the training, may provide a path towards greater reliability of skill evaluation, through longitudinal sampling, and validity, through better alignment of evaluation activities in respect to learning objectives.