City Research Online

Abstract

We report a controlled study investigating the effect of visual information (i.e., seeing the speaker) on spoken language comprehension. We compare the ERP signature (N400) associated with each word in audio-only and audio-visual presentations of the same verbal stimuli. We assess the extent to which surprisal measures (which quantify the predictability of words in their lexical context) are generated on the basis of different types of language models (specifically n-gram and Transformer models) predict N400 responses for each word. Our results indicate that cognitive effort differs significantly between multimodal and unimodal settings. In addition, our findings suggest that while Transformer-based models, which have access to a larger lexical context, provide a better fit in the audio-only setting, 2-gram language models are more effective in the multimodal setting. This highlights the significant impact of local lexical context on cognitive processing in a multimodal environment.

Publication Type:	Conference or Workshop Item (Paper)
Additional Information:	©2023 The Author(s). This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY)
Subjects:	P Language and Literature > PB Modern European Languages Q Science > QA Mathematics > QA75 Electronic computers. Computer science R Medicine > RC Internal medicine
Departments:	School of Science & Technology School of Science & Technology > Computer Science
SWORD Depositor:	Symplectic Administrator

Export

Downloads