CRCNS US-Spain Research Proposal: Collaborative Research: Tracking and modeling the neurobiology of multilingual speech recognition

CRCNS 美国-西班牙研究提案：合作研究：跟踪和建模多语言语音识别的神经生物学

• 批准号: 2207770
• 负责人: James Magnuson
• 金额: $ 46.73万
• 依托单位: University of Connecticut
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2025-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2207770&HistoricalAwards=false
• 关键词: CRCNS; US; Spain; Research; Proposal

More than half of the world's population speaks two or more languages fluently. Speaking more than one language allows you to communicate and interact with individuals in other countries and cultures through conversation and reading. It also has economic benefits, and possibly even cognitive and health benefits. Scientifically, there are many open questions about bilingualism for education (how can we best train people in new languages?), health (how can you best treat a bilingual person with language difficulty after a brain injury?), and technology (how can we make speech recognition systems as flexible as multilingual humans?). To better understand multilingual language processing, researchers will record neural responses to speech in people who know either one, two, or three languages, while they listen to the languages they know. The project is a transatlantic collaborative effort with US researchers partnering with Spanish researchers. High school, undergraduate, graduate, and postdoctoral trainees will receive training in cutting-edge computational and cognitive neuroscience and psycholinguistics, including data science and modeling skills transferable to a range of academic and non-academic STEM careers.One of the fundamental questions that researchers will explore is how two or more languages co-exist within a single language system and how each is represented in the brain. Some prior research suggests there is a deep continuous coactivation of all the languages a person knows even when they are in a single language context, while other research suggests that under many circumstances, only the language relevant in the moment is activated. The project will use the tools of computational neuroscience to develop cognitive theories and implemented models of bilingual and trilingual language processing, which the research team will compare to neuroimaging data with high temporal resolution (magnetoencephalography or MEG). MEG will be collected while monolingual, bilingual, and trilingual individuals process speech from languages they know under conditions designed to promote attention to a single language (isolated words or continuous speech from only 1 language) or two languages (random interleaving of isolated words from 2 languages, or more ecological 'code-switching' between 2 languages). Researchers will use a state-of-the-art neural network model of human speech recognition developed with previous NSF support. They will use continuous speech tracking to relate neural activity to both theoretically-generated hypotheses regarding potential impacts of language co-activation and the behavior and internal activity of neural network models. In this way, researchers will be able to compare human brain responses and neural network model responses to statistical predictions of the expectation level for each successive speech sound (consonant or vowel) and word during presentation of continuous speech. Comparing different models to neural responses will help researchers address fundamental questions, such as whether all the languages a person knows are active whenever they hear any language, and whether this is helpful or causes interference. This research promises to deepen our understanding of multilingual language development and processing in the human brain.A companion project is being funded by the State Research Agency, Spain (AEI).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

世界上一半以上的人口能流利地讲两种或多种语言。说不止一种语言可以让您通过对话和阅读与其他国家和文化的人进行交流和互动。它还具有经济效益，甚至可能对认知和健康有益。从科学角度来看，关于双语教育（我们如何才能最好地训练新语言的人？）、健康（如何才能最好地治疗脑损伤后有语言困难的双语者？）和技术（我们如何才能最好地训练双语者），存在许多悬而未决的问题。让语音识别系统像多语言人类一样灵活？）。为了更好地理解多语言语言处理，研究人员将记录懂得一种、两种或三种语言的人在听他们懂得的语言时对语音的神经反应。该项目是美国研究人员与西班牙研究人员合作的跨大西洋合作项目。高中、本科生、研究生和博士后学员将接受尖端计算和认知神经科学和心理语言学方面的培训，包括可转移到一系列学术和非学术 STEM 职业的数据科学和建模技能。研究人员面临的基本问题之一将探索两种或多种语言如何在单一语言系统中共存，以及每种语言如何在大脑中表示。一些先前的研究表明，即使一个人处于单一语言环境中，他所知道的所有语言也会深度持续地协同激活，而其他研究表明，在许多情况下，只有当前相关的语言才会被激活。该项目将利用计算神经科学工具来开发认知理论并实施双语和三语语言处理模型，研究团队将其与高时间分辨率的神经影像数据（脑磁图或 MEG）进行比较。 MEG 将在单语、双语和三语个体在旨在促进对单一语言（仅来自一种语言的孤立词或连续语音）或两种语言（来自 2 种语言的孤立词的随机交错）的条件下处理他们所知道的语言的语音时收集。语言，或者两种语言之间更生态的“代码转换”）。研究人员将使用在 NSF 先前支持下开发的最先进的人类语音识别神经网络模型。他们将使用连续语音跟踪将神经活动与理论上生成的关于语言共同激活的潜在影响以及神经网络模型的行为和内部活动的假设联系起来。通过这种方式，研究人员将能够将人脑反应和神经网络模型反应与连续语音呈现期间每个连续语音（辅音或元音）和单词的期望水平的统计预测进行比较。将不同模型与神经反应进行比较将有助于研究人员解决基本问题，例如一个人听到任何语言时所知道的所有语言是否都会活跃，以及这是否有帮助或造成干扰。这项研究有望加深我们对人脑中多语言语言发展和处理的理解。西班牙国家研究机构 (AEI) 正在资助一个配套项目。该奖项反映了 NSF 的法定使命，经评估认为值得支持利用基金会的智力优势和更广泛的影响审查标准。