Audiovisual Perception of Emotion and Speech in Hearing Individuals and Cochlear Implant Users

听力正常者和人工耳蜗使用者对情绪和言语的视听感知

• 批准号: 501994399
• 负责人: Professor Dr. Christian Dobel
• 金额: --
• 依托单位: Institut für Anglistik und Amerikanistik
• 依托单位国家: 德国
• 项目类别: Priority Programmes
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/501994399?language=en
• 关键词: Audiovisual; Perception; Emotion; Speech; Hearing

The ability to communicate via auditory spoken language is taken as a benchmark for success of cochlear implants (CIs), but this disregards the important role visual cues play in communication. The relevance of socio-emotional signals and their importance for quality of life with a CI (Luo, Kern, & Pulling, 2018; Schorr, Roth, & Fox, 2009) calls for research on visual benefits to communication. Recruiting models of communication via the face and voice (Young, Frühholz, & Schweinberger, 2020), we consider that deafness can elicit crossmodal cortical plasticity, such that visual stimuli can activate auditory cortex areas. Even after adaptation to a CI, initial findings suggest a particularly strong contribution of visual information to the perception of speech and speaker gender. Better understanding of these phenomena at the functional and brain level is required to promote efficient interventions improving communication, and ultimately life quality. Here we focus on postlingually deaf adult CI users and propose four studies (S1-S4). In S1, we conduct a systematic review to determine the current state of knowledge regarding the role of visual information (face or manual gesture) for emotion recognition and speech perception from voices, in hearing adults and CI users. In S2, we explore in a behavioral experiment with dynamic time-synchronized audiovisual stimuli whether CI users benefit more from congruent facial expressions when recognizing vocal emotions than do hearing adults – and whether this holds even when controlling for overall auditory-only performance levels. Importantly, we use voice morphing technology, rather than noise, to equate performance levels. In S3, we study brain correlates of audiovisual integration (AVI) in event-related potentials (ERPs) to audiovisual (AV) emotional stimuli. We focus on the ability of congruent AV stimuli to speed up neural processing, and investigate relationships between individual neural markers of AVI and behavioral performance in emotion recognition. In S4, we study the degree to which perceptual training with caricatured vocal emotions can improve auditory and audiovisual emotion recognition in adult CI users. We assess relationships between emotion recognition abilities and reported quality of life in all studies. The project builds on successful previous research funded by the DFG on Voice Perception and Audiovisual integration in the identification of speaker and speech, and on our long-standing collaboration with the Cochlear Implant Rehabilitation Centre in Thuringia. We hope this work will contribute to models of the cognitive and brain mechanisms underlying multimodal perception in human communication. We propose that better understanding of the mechanisms by which visual facial signals support CI users will provide important information that can be used to optimize both linguistic and socio-emotional communication, and ultimately quality of life.

通过听觉口语进行交流的能力被视为人工耳蜗 (CI) 成功的基准，但这忽视了视觉线索在交流中的重要作用以及社会情感信号的相关性及其对生活质量的重要性。 CI (Luo, Kern, & Pulling, 2018; Schorr, Roth, & Fox, 2009) 呼吁研究视觉对沟通的好处（年轻的、 Frühholz 和 Schweinberger，2020），我们认为耳聋可以引发跨模式皮质可塑性，因此视觉刺激可以激活听觉皮层区域，即使在适应 CI 后，初步研究结果表明视觉信息对言语感知的贡献特别大。需要在功能和大脑层面更好地理解这些现象，以促进有效的干预措施，改善沟通，并最终改善生活质量。在这里，我们重点关注语后聋哑成人 CI 用户，并提出四项研究。 (S1-S4)。在 S1 中，我们进行了系统回顾，以确定视觉信息（面部或手势）对于听力成人和 CI 用户的情绪识别和语音感知的作用的当前知识状态。 ，我们在动态时间同步视听刺激的行为实验中探讨了 CI 用户在识别声音情绪时是否比听力正常的成年人更能从一致的面部表情中获益，以及即使在控制整体听觉表现水平时，这一点是否成立。使用语音在 S3 中，我们研究事件相关电位 (ERP) 中的视听整合 (AVI) 与视听 (AV) 情绪刺激的大脑关联性。为了加速神经处理，并研究 AVI 的个体神经标记与情绪识别中的行为表现之间的关系，在 S4 中，我们研究了带有讽刺性声音情绪的感知训练可以改善听觉和视听的程度。我们在所有研究中评估了情绪识别能力与报告的生活质量之间的关系，该项目建立在 DFG 先前资助的关于说话者和语音识别中的语音感知和视听集成的成功研究的基础上。我们与图林根州人工耳蜗康复中心的长期合作希望这项工作能够有助于建立人类交流中多模式感知的认知和大脑机制模型，我们建议更好地理解视觉面部信号支持 CI 的机制。用户将提供可用于优化语言和社会情感交流以及最终生活质量的重要信息。