Capturing individual differences in sentence processing: How reliable is the self-paced reading task?

Advances in research on language processing have originally come from group-level comparisons, but there is now a growing interest in individual differences. To investigate individual differences, tasks that have shown robust group-level differences are often used with the implicit assumption that they will also be reliable when used as an individual differences measure. Here, we examined whether one of the primary tasks used in psycholinguistic research on language processing, the self-paced reading task, can reliably measure individual differences in relative clause processing. We replicated the well-established effects of relative clauses at the group level, with object relative clauses being more difficult to process than subject relative clauses. However, when using difference scores, the reliability of the size of the relative clause effect was close to zero because the self-paced reading times for the different relative clause types were highly correlated within individuals. Nonetheless, we found that the self-paced reading task can be used to reliably capture individual differences in overall reading speed as well as key sentence regions when the two types of relative clause sentences are considered separately. Our results indicate that both the reliability and validity of different sentence regions need to be assessed to determine whether and when self-paced reading can be used to examine individual differences in language processing.