Robust neural tracking of linguistic speech representations using a convolutional neural network.

When listening to continuous speech, populations of neurons in the brain track different features of the signal. Neural tracking can be measured by relating the electroencephalography (EEG) and the speech signal. Recent studies have shown a significant contribution of linguistic features over acoustic neural tracking using linear models. Linear models cannot model the nonlinear dynamics of the brain. We introduce a convolutional neural network (CNN) that relates EEG to linguistic features using phoneme or word onsets as a control and has the capacity to model non-linear relations.
Approach. We integrate phoneme- and word-based linguistic features (phoneme surprisal, cohort entropy, word surprisal, and word frequency) in our nonlinear CNN model and investigate if they carry additional information on top of lexical features (phoneme and word onsets). We compare the results to a linear decoder and a linear CNN and evaluate the impact of the model's architecture, the presence of linguistic features, and the training paradigm on a match-mismatch task's performance.
Main results. For the non-linear CNN, we found a significant contribution of cohort entropy over phoneme onsets, and of word surprisal and word frequency over word onsets. The training paradigm and architecture have a significant impact on the performance, and the non-linear CNN outperforms the linear baselines on the match-mismatch task. 
Significance. Measuring coding of linguistic features in the brain is important for auditory neuroscience research and applications that involve objectively measuring speech understanding. With linear models this is measurable, but the effects are very small. The proposed non-linear CNN model yields larger effect sizes and therefore could show effects that would be otherwise unmeasurable, and may in the future lead to improved within-subject measures and shorter recording durations.