CRI: CI-P: Creating the Largest Speech Emotional Database by Leveraging Existing Naturalistic Recordings

CRI：CI-P：利用现有的自然录音创建最大的语音情感数据库

• 批准号: 1823166
• 负责人: Carlos Busso
• 金额: $ 9.94万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-09-01 至 2021-02-28
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1823166&HistoricalAwards=false
• 关键词: CRI; CI; Creating; Largest; Speech

This community infrastructure planning project aims to consider the needs from other researchers in the design of the largest publicly available naturalistic speech emotional database, broadening the impact of the corpus across speech processing areas. The project includes a workshop with researchers with relevant but diverse expertise to introduce the current protocol for data collection, and requests their recommendations for improvements. The proposed activity will improve the protocol to address the needs from the community. Affective computing is an important research area aiming to understand, analyze, recognize, and synthesize human emotions. Providing emotion capabilities to current speech-based interfaces can facilitate transformative applications in areas related to Human Computer Interaction (HCI), healthcare, security and defense, education and entertainment. The research infrastructure envisioned in this project will open new opportunities that we cannot address with current speech emotional databases. In the area of affective computing, the proposed corpus will provide suitable training sets to explore learning algorithms that are powerful, but require large amount of labeled data. It is expected that the size, naturalness, and speaker and recording variety in the proposed corpus will allow the community to create robust models that generalize across applications. Improvements on speech emotion recognition systems will facilitate the transition of these algorithms into practical applications, providing unique societal benefits. The proposed infrastructure will also play a key role on other speech processing tasks. For the first time, the community will have the infrastructure to address speaker verification and automatic speech recognition solutions against variations due to emotion.The proposed infrastructure relies on a novel approach based on emotion retrieval along with crowdsource-based annotations to effectively build a large, naturalistic emotional database with balanced emotional content, reduced cost and reduced manual labor. The database considers podcast recordings that are available in audio-sharing websites. Although the approach of building affective databases using media content has been previously explored, the contribution of this study is the use of machine learning algorithms to retrieve audio clips with balanced emotional content, providing natural stimuli with wider spectrum of emotions. The proposed approach relies on automatic algorithms to post-process podcasts and a cost effective annotation process, which make it possible to build large scale speech emotional databases. This approach provides natural emotional renditions that are difficult to obtain with alternative data collection protocols. This project involves the research community from the design of the corpus, which is the key goal in this community infrastructure planning project. The community also play a key role in the selection of target sentences to be emotionally annotated, with novel grand challenges where the goal is to recognize and retrieve target emotional behaviors in unconstrained, unlabeled recordings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该社区基础设施规划项目旨在考虑其他研究人员在设计最大的公开自然语音情感数据库时的需求，扩大语料库在语音处理领域的影响。该项目包括一个由具有相关但不同专业知识的研究人员参加的研讨会，以介绍当前的数据收集协议，并征求他们的改进建议。拟议的活动将改进协议以满足社区的需求。情感计算是旨在理解、分析、识别和综合人类情感的重要研究领域。为当前基于语音的界面提供情感功能可以促进人机交互（HCI）、医疗保健、安全和国防、教育和娱乐等领域的变革性应用。该项目设想的研究基础设施将带来新的机遇，这是我们当前的语音情感数据库无法解决的。在情感计算领域，所提出的语料库将提供合适的训练集来探索功能强大但需要大量标记数据的学习算法。预计所提议的语料库中的大小、自然度以及说话者和录音的多样性将使社区能够创建跨应用程序通用的强大模型。语音情感识别系统的改进将有助于这些算法转化为实际应用，从而提供独特的社会效益。拟议的基础设施还将在其他语音处理任务中发挥关键作用。社区将首次拥有解决说话者验证和自动语音识别解决方案的基础设施，以应对因情绪而变化的问题。所提出的基础设施依赖于一种基于情感检索和基于众包的注释的新颖方法，以有效地构建一个大型的、自然的情感数据库，情感内容均衡，降低成本，减少体力劳动。该数据库考虑音频共享网站中提供的播客录音。尽管之前已经探索过使用媒体内容构建情感数据库的方法，但本研究的贡献在于使用机器学习算法来检索具有平衡情感内容的音频片段，从而提供具有更广泛情感的自然刺激。所提出的方法依赖于自动算法来后处理播客和具有成本效益的注释过程，这使得建立大规模语音情感数据库成为可能。这种方法提供了自然的情感再现，这是使用其他数据收集协议难以获得的。该项目从语料库的设计就涉及到研究社区，这是该社区基础设施规划项目的关键目标。该社区还在选择要进行情感注释的目标句子方面发挥着关键作用，提出了新颖的重大挑战，其目标是在不受约束、未标记的录音中识别和检索目标情感行为。该奖项反映了 NSF 的法定使命，并被认为是值得的通过使用基金会的智力优势和更广泛的影响审查标准进行评估来获得支持。

项目成果

An Efficient Temporal Modeling Approach for Speech Emotion Recognition by Mapping Varied Duration Sentences into Fixed Number of Chunks

通过将不同持续时间的句子映射到固定数量的块来进行语音情感识别的有效时间建模方法

• DOI: 10.21437/interspeech.2020-2636
• 发表时间: 2020-10
• 期刊: Interspeech 2020
• 作者: Lin, Wei;Busso, Carlos
• 通讯作者: Busso, Carlos

The MSP-Conversation Corpus

MSP-对话语料库

• DOI: 10.21437/interspeech.2020-2444
• 发表时间: 2020-10
• 期刊: Interspeech 2020
• 作者: Martinez;Abdelwahab, Mohammed;Busso, Carlos
• 通讯作者: Busso, Carlos

Modeling Uncertainty in Predicting Emotional Attributes from Spontaneous Speech

通过自发言语预测情感属性的不确定性建模

• DOI: 10.1109/icassp40776.2020.9054237
• 发表时间: 2020-05
• 期刊: speech and signal processing (ICASSP 2020
• 作者: Sridhar, Kusha;Busso, Carlos
• 通讯作者: Busso, Carlos

Semi-Supervised Speech Emotion Recognition With Ladder Networks

使用阶梯网络的半监督语音情感识别

• DOI: 10.1109/taslp.2020.3023632
• 发表时间: 2020-01
• 期刊: and Language Processing
• 作者: Parthasarathy, Srinivas;Busso, Carlos
• 通讯作者: Busso, Carlos