Deep Learning Based Complex Spectral Mapping for Multi-Channel Speaker Separation and Speech Enhancement

基于深度学习的复杂频谱映射，用于多通道说话人分离和语音增强

基本信息

批准号：
2125074
负责人：
Eric Fosler-Lussier
金额：
$ 39.06万
依托单位：
Ohio State University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2021
资助国家：
美国
起止时间：
2021-08-01 至 2024-07-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2125074&HistoricalAwards=false
关键词：
Deep Learning Based Complex Spectral

项目摘要

Despite tremendous advances in deep learning based speech separation and automatic speech recognition, a major challenge remains how to separate concurrent speakers and recognize their speech in the presence of room reverberation and background noise. This project will develop a multi-channel complex spectral mapping approach to multi-talker speaker separation and speech enhancement in order to improve speech recognition performance in such conditions. The proposed approach trains deep neural networks to predict the real and imaginary parts of individual talkers from the multi-channel input in the complex domain. After overlapped speakers are separated into simultaneous streams, sequential grouping will be performed for speaker diarization, which is the task of grouping the speech utterances of the same talker over intervals with the utterances of other speakers and pauses. Proposed speaker diarization will integrate spatial and spectral speaker features, which will be extracted through multi-channel speaker localization and single-channel speaker embedding. Recurrent neural networks will be trained to perform classification for the purpose of speaker diarization, which can handle an arbitrary number of speakers in a meeting. The proposed separation system will be evaluated using open, multi-channel speaker separation datasets that contain both room reverberation and background noise. The results from this project are expected to substantially elevate the performance of continuous speaker separation, as well as speaker diarization, in adverse acoustic environments, helping to close the performance gap between recognizing single-talker speech and recognizing multi-talker speech.The overall goal of this project is to develop a deep learning system that can continuously separate individual speakers in a conversational or meeting setting and accurately recognize the utterances of these speakers. Building on recent advances on simultaneous grouping to separate and enhance overlapped speakers in a talker-independent fashion, the project is mainly focused on speaker diarization, which aims to group the speech utterances of the same speaker across time. To achieve speaker diarization, deep learning based sequential grouping will be performed and it will integrate spatial and spectral speaker characteristics. Through sequential organization, simultaneous streams will be grouped with earlier-separated speaker streams to form sequential streams, each of which corresponds to all the utterances of the same speaker up to the current time. Speaker localization and classification will be investigated to make sequential grouping capable of creating new sequential streams and handling an arbitrary number of speakers in a meeting scenario. With the added spatial dimension, the proposed diarization approach provides a solution to the question of who spoke when and where, significantly expanding the traditional scope of who spoke when. The proposed separation system will be evaluated using multi-channel speaker separation datasets that contain highly overlapped speech in recorded conversations, as well as room reverberation and background noise present in real environments. The main evaluation metric will be word error rate in automatic speech recognition. The performance of speaker diarization will be measured using diarization error rate.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

尽管基于深度学习的语音分离和自动语音识别方面取得了巨大进步，但主要的挑战仍然是如何将并发的演讲者分开并在房间混响和背景噪音的存在下认识他们的言语。该项目将开发一种多通道复杂的光谱映射方法，以对多对话者说话者的分离和语音增强，以提高这种情况下的语音识别性能。所提出的方法训练深层神经网络，以从复杂域中的多通道输入中预测单个说话者的真实和虚构部分。在将重叠的说话者分成同时流中后，将进行顺序分组以进行诊断，这是将同一说话者的语音发言的任务与其他说话者的话语和其他说话者和暂停的话语一起分组。拟议的扬声器诊断将整合空间和光谱扬声器的特征，这些功能将通过多通道扬声器定位和单渠道扬声器嵌入来提取。重复的神经网络将接受培训以进行分类，以进行诊断，该目的可以在会议上处理任意数量的说话者。提出的分离系统将使用开放的多通道扬声器分离数据集评估，这些数据集既包含房间混响和背景噪声。在不利的声学环境中，该项目的结果有望大大提高连续扬声器分离的性能以及扬声器诊断的表现，从而有助于弥补识别单聊天者语音和识别多对待者的演讲之间的性能差距。这个项目的是开发一个深度学习系统，该系统可以在会话或会议设置中不断地分开单个说话者，并准确地认识到这些说话者的话语。在最新进步的基础上，该项目以与说话者无关的方式分离和增强了重叠的演讲者，该项目主要集中于演讲者诊断，该项目的目的是将同一扬声器的语音发言分组。为了实现说话者诊断，将进行基于深度学习的顺序分组，并将其整合到空间和频谱扬声器的特征。通过顺序组织，同时流将与较早的分离扬声器流进行分组，以形成顺序流，每个流都与当前时间的同一扬声器的所有话语相对应。将研究说话者的本地化和分类，以使连续分组能够在会议场景中创建新的顺序流并处理任意数量的说话者。随着空间维度的增加，提议的诊断方法为谁讲话的问题提供了解决方案，即在何时何地讲话，显着扩大了传统的人何时讲话的范围。提出的分离系统将使用多通道扬声器分离数据集进行评估，这些数据集包含在记录的对话中包含高度重叠的语音，以及在真实环境中存在的房间混响和背景噪声。主要评估指标将是自动语音识别中的单词错误率。说话者诊断的性能将使用诊断错误率来衡量。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的审查标准通过评估来获得支持的。