Collaborative Research: Estimating Articulatory Constriction Place and Timing from Speech Acoustics

合作研究：从语音声学估计发音收缩位置和时间

• 批准号: 2343847
• 负责人: Mark Tiede
• 金额: $ 15.4万
• 依托单位: Yale University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-04-01 至 2025-11-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2343847&HistoricalAwards=false
• 关键词: Collaborative; Research; Estimating; Articulatory; Constriction

This collaborative project focuses on a new approach for using speech recordings to study speaker pronunciation habits--that is, the way speakers systematically coordinate the articulatory movements of their lips, jaw, tongue, glottis and soft palate to produce words and sentences. These articulatory habits differ between individuals, and across languages and dialects of the same language, accounting for many aspects of foreign accent, speech disorders and speaking style. Whereas previous studies of these habits have required specialized equipment for the immediate observation of articulator movements, the aim of this project is to develop and improve a tool for "speech inversion"--that is, a tool that can accurately recover articulatory movements directly from the acoustic speech signal using machine learning methods. To date, the tool developed by the project team has successfully recovered movements of the tongue and lips; the current project extends the tool’s functionality to encompass nasality (soft palate) and voicing (glottis). Training and validation of the extended system will proceed using a newly collected corpus of acoustic and articulatory data drawn from speakers of American English. This corpus, comprising co-collected audio, nasal, voicing, and articulatory movement, will serve as 'ground truth' for training and assessing the capabilities of the fully trained speech inversion system. As a further test, we will test it against ground truth data from speakers of languages with patterns of articulatory habits known to differ from English.The goal of this project is to develop and refine a Speech Inversion Tool that 'reads' acoustic recordings of speech and 'recovers' details of the magnitude and timing of articulatory movements. The project aims to accomplish this goal by training specialized Neural Network models to relate features of the acoustic signal to separately acquired ground-truth nasal vs. oral outflow signals and concurrent electroglottography. Training data derives from native speakers of English; validation and tests for generalization include productions of speakers of Canadian French and Russian. When successfully validated, the resulting speech inversion tool will be useful for identifying medical issues that affect speech movement organization, such as the well-known disruption of oral/laryngeal timing in speakers with dysarthria. In addition, incorporating estimates of articulation may also aid in the tracking of changes resulting from medical conditions such as depression and schizophrenia. More generally, the ability to rapidly and easily analyze articulatory movements obtained from audio recordings alone has the potential substantially improve Automated Speech Recognition (ASR) systems, and to assist scholars, forensic scientists, and clinical professionals studying the speech of communities under field conditions in rural or under-resourced areas, and to help in the documentation of endangered languages.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该合作项目重点关注一种使用语音录音来研究说话者发音习惯的新方法，即说话者最终协调嘴唇、下巴、舌头、声门和软腭的发音运动以产生不同的单词和句子习惯的方式。个体之间、跨语言和同一语言的方言，解释了外国口音、言语障碍和说话风格的许多方面，特别是以前对这些习惯的研究需要专门的设备来立即观察发音器的运动，其目的是。这个项目是开发和改进“语音反转”工具——即利用机器学习方法直接从声学语音信号中准确恢复发音运动的工具。迄今为止，该项目团队开发的工具已成功恢复了发音运动。舌头和嘴唇；当前项目扩展了该工具的功能，以涵盖鼻音（软腭）和发声（声门）。扩展系统的训练和验证将使用新收集的美式英语使用者的声学和发音数据语料库进行。 。这语料库由共同收集的音频、鼻音、发声和发音运动组成，将作为训练和评估经过充分训练的语音反演系统的能力的“地面实况”。作为进一步的测试，我们将根据地面实况数据对其进行测试。来自已知与英语不同的发音习惯模式的语言的使用者。该项目的目标是开发和完善一种语音反转工具，该工具可以“读取”语音录音并“恢复”语音幅度和时间的细节发音的该项目旨在通过训练专门的神经网络模型将声学信号的特征与单独获取的鼻腔与口腔流出信号以及来自英语母语者的并发电声门描记术相关联来实现这一目标。当成功验证后，所产生的语音倒转工具将有助于识别影响言语运动组织的医学问题，例如众所周知的说话者的口腔/喉部计时中断。此外，结合发音障碍的估计也可能有助于跟踪抑郁症和精神分裂症等疾病引起的变化。更一般地说，快速、轻松地分析单独获得的音频记录的发音运动的能力有可能显着提高自动化。语音识别 (ASR) 系统，协助学者、法医科学家和临床专业人员在农村或资源贫乏地区的实地条件下研究社区的语音，并帮助记录濒危语言。授予 NSF 的法定使命，并通过评估反映使用基金会的智力优点和更广泛的影响审查标准，被认为值得支持。