Using longitudinal progress test data to determine the effect size of learning in undergraduate medical education - a retrospective, single-center, mixed model analysis of progress testing results.

Medical education research focuses on the development of efficient learning methods promoting the acquisition of student's knowledge and competencies. Evaluation of any modification of educational approaches needs to be evaluated accordingly and a reliable effect size needs to be reached. Our aim is to provide a methodological basis to calculate effect sizes from longitudinal progress test data that can be used as reference values in further research. We used longitudinally collected progress test data and evaluated the increasing knowledge of medical students from the first to the fifth academic year. Students were asked to participate in the progress test, which consists of 200 multiple-choice questions in single best answer format with an additional 'don't know' option. All available individual test scores of all progress tests (n = 10) administered between April 2012 and October 2017 were analyzed. Due to the large amount of missing test results, e.g., from students at the beginning of their studies, a linear mixed model was fitted to include all collected data. In total, we analyzed 6324 test scores provided by 2587 medical students. Mean score for medical knowledge (% correct answers) increases from 16.6% (SD: 10.8%) to 51.0% (SD: 15.7%, overall effects size using linear mixed models d = 1.55). Medical students showed a learning effect of d = 0.54 (total gain: 6.9%) between the 1st and 2nd, d = 0.88 (total gain: 12.0%) between the 2nd and 3rd, d = 0.60 (total gain: 7.9%) between the 3rd and 4th and d = 0.58 (total gain: 7.9%) between the 4th and 5th study year. We demonstrated that incomplete data from longitudinally collected progress tests can be used to acquire reliable effect size estimates. The demonstrated effects size between d = 0.53-0.9 by study year may help researchers to design studies in medical education.