Enhancing research on speech and deep learning through holistic acoustic analysis

通过整体声学分析加强语音和深度学习研究

• 批准号: 2219843
• 负责人: Matthew Goldrick
• 金额: $ 100万
• 依托单位: Northwestern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-08-15 至 2026-07-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2219843&HistoricalAwards=false
• 关键词: Enhancing; research; speech; deep; learning; Enhancing; research; speech; deep; learning

You can guess a lot about a person from the way they pronounce words. Remarkably, human listeners can tell if it is likely that talkers learned English as a first language or a second language, or if the talkers might have a brain injury that makes it difficult for them to speak. Such intuitions rely on human listeners’ holistic pattern recognition abilities; these allow us to perceive the important, meaningful, yet subtle differences between pronunciations. However, the methods scientists currently use to measure speech objectively – based on a small number of properties of speech sounds – fail to capture these differences, hampering our ability to use speech to learn about the mind and brain. This project brings together speech scientists, computer scientists, and neuroscientists to test a radically different approach to this problem. Machine learning will be used to discover a new method for quantifying differences between spoken utterances based on holistic pattern recognition. This will be tested against new and existing data from bilingual speakers. If successful, this will yield a fully general method that can be applied to speech from any language or any domain of language usage, allowing scientists to capitalize on the wealth of information in speech to develop powerful new insights into the mind and brain. Improved detection of subtle problems with pronunciation, such as occurs with Alzheimer’s disease, will advance our understanding of the brain mechanisms that humans use to produce speech. The results of this testing will also allow computer scientists to advance our understanding of how machine learning algorithms process sounds, driving improvements in the algorithms and supporting applications in any area of speech and language technology that relies on spoken language processing. Speech variability across talkers provides a treasure trove of information for cognitive neuroscientists, leading to important insights into the cognitive mechanisms underlying language processing and potentially providing early signs of brain dysfunction. Current studies of speech are hamstrung by analyses that require preselecting specific temporal scales and acoustic dimensions. We propose a radically different approach: using unsupervised deep learning to discover a representational space for analysis of acoustic variation. To test this highly general approach, this method will be compared to current state-of-the art methods for analyzing individual variation in bilingual speech. This includes using the acoustic variation in second language speech to predict intelligibility and to detect difficulties in code-switching, particularly the challenges faced by individuals with Alzheimer’s Disease. The results will inform development of deep learning and cognitive neuroscience. The machine learning algorithm is fully general; it can be applied to speech from any language or any domain of language usage, expanding the range of populations and contexts that can be served by speech technology or studied by cognitive neuroscientists. The project’s integrative approach will allow computer scientists to advance our understanding of the extent to which modern deep learning architectures do or do not approximate human speech processing and allow cognitive neuroscientists to further our understanding of how meaningful acoustic distinctions are represented in speech perception and production. human speech representation. This project is funded by the Integrative Strategies for Understanding Neural and Cognitive Systems (NCS) program, which is jointly supported by the Directorates for Computer and Information Science and Engineering (CISE), Education and Human Resources (EHR), Engineering (ENG), and Social, Behavioral, and Economic Sciences (SBE).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

您可以从一个人发音的方式来猜测很多人。值得注意的是，人类的听众可以判断说话者是否可能学到英语是一种语言或第二语言，或者说话者是否会遭受脑损伤，这使他们很难说话。这种直觉依赖于人类听众的整体模式识别能力；这些使我们能够感知发音之间的重要，有意义但微妙的差异。但是，科学家目前使用的方法基于语音的少量属性来客观地衡量语音 - 无法捕获这些差异，从而阻碍了我们使用语音学习思想和大脑的能力。该项目汇集了言语科学家，计算机科学家和神经科学家，以测试针对此问题的截然不同的方法。机器学习将用于发现一种新方法，以量化基于整体模式识别的讲话话语之间的差异。这将根据双语扬声器的新数据和现有数据进行测试。如果成功的话，这将产生一种完全通用的方法，可以应用于任何语言或任何语言使用领域的语音，使科学家能够利用语音中的大量信息，以发展对思想和大脑的强大新见解。改善了对发音的微妙问题的检测，例如阿尔茨海默氏病发生的，将促进我们对人类用来产生言语的大脑机制的理解。该测试的结果还将允许计算机科学家提高我们对机器学习算法如何处理声音，推动算法改进的理解，并支持任何依赖口语处理的语音和语言技术领域的应用程序。谈话者之间的语音变异性为认知神经科学家提供了一大批信息，从而引发了对语言处理基础的认知机制的重要见解，并可能提供了脑功能障碍的早期迹象。当前的语音研究通过需要预览特定的临时量表和声学维度的分析进行障碍。我们提出了一种根本不同的方法：使用无监督的深度学习来发现代表性的空间来分析声学变异。为了测试这种高度通用的方法，将将这种方法与当前的最新方法进行比较，以分析双语语音中的个体变化。这包括使用第二语言语音中的声学差异来预测可理解性并发现代码转换中的困难，尤其是阿尔茨海默氏病患者面临的挑战。结果将为深度学习和认知神经科学的发展提供信息。机器学习算法是完全普遍的。它可以应用于任何语言或任何语言使用领域的语音，扩大语音技术或认知神经科学家可以使用的人群和上下文的范围。该项目的综合方法将使计算机科学家能够促进我们对现代深度学习体系结构进行或不近似人类语音处理的程度的理解，并允许认知神经科学家进一步了解我们对言语感知和生产中如何表示有意义的声学区分。人类言语表示。该项目由理解神经和认知系统（NCS）计划的综合策略提供资金，该计划由计算机，信息科学与工程局（CISE），教育与人力资源（EHR），工程（ENG），社会，行为，行为和经济科学（SBE）的诚实授予的诚实传统的授权宣传，共同支持该局。更广泛的影响审查标准。