RI: Small: Integrative, Semantic-Aware, Speech-Driven Models for Believable Conversational Agents with Meaningful Behaviors

RI：小型：集成的、语义感知的、语音驱动的模型，用于具有有意义行为的可信会话代理

• 批准号: 1718944
• 负责人: Carlos Busso
• 金额: $ 49.41万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-09-01 至 2022-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1718944&HistoricalAwards=false
• 关键词: RI; Small; Integrative; Semantic; Aware

This project will analyze, model and synthesize human behaviors to create a believable Conversational Agent (CA). A CA is a virtual agent that interacts with a user, displaying human-like behaviors not only through speech but also through facial expressions and head movements. Replicating or representing human behavior includes generating gestures that are synchronized with speech, convey appropriate meaning in the message, and respond to the behaviors displayed by the user. An appealing approach to synthesize human-like behaviors is the use of data-driven methods, which have the potential of capturing naturalistic variations of the behaviors. Modeling the dependencies between speech and gestures brings insights about verbal and nonverbal communication, underlying the production and coordination mechanisms used during natural human interactions. CAs can be used in a variety of health care applications, such as helping hearing impaired individuals and teaching social skills to autistic children. Tutoring systems that display human-like behaviors to communicate and acknowledge active listening will engage better with the students, helping them in their learning. The project promises a fertile ground for interdisciplinary training of graduate and undergraduate students. The models will be evaluated with an assistive agent (CA or embodied robot) interacting with UT Dallas students, serving as a platform to reach out students from all majors, especially woman and underrepresented minorities.The project will take an integrative, cross-disciplinary approach to generate believable and meaningful behaviors by exploring the intrinsic relation between speech, head motion, and facial expressions, constrained by important aspects of spoken language. The planned research leverages some of the latest developments in the field of deep learning in an integrative fashion, pulling together acoustic features and semantic language structure, to build models that are able to account for the correlation between various facial and head movements. The speech-driven approach will capture the variability of human behavior in a manner that is not easily possible with rule-based approaches. Dialog acts and emotions will be inferred and used to constrain the speech driven models, capturing the relation between high-level conversational functions and facial gestures. The project will offer novel, principled methods to generate behaviors driven by synthesized speech, opening new application domain when only text is available. The approach will capture the acoustic variability in synthesized speech, while maintaining the temporal dependency between gestures and speech. The project will also explore schemes to modify the behaviors of the user by displaying carefully designed gestures generated with our data-driven framework. By tracking the behaviors of the user, the system will provide appropriate responses, closing the loop in the interaction.

该项目将分析、建模和综合人类行为，以创建可信的对话代理（CA）。 CA 是一种与用户交互的虚拟代理，不仅通过语音，还通过面部表情和头部动作来显示类人行为。复制或表示人类行为包括生成与语音同步的手势、在消息中传达适当的含义以及对用户显示的行为做出响应。合成类人行为的一种有吸引力的方法是使用数据驱动的方法，该方法有可能捕获行为的自然变化。对语音和手势之间的依赖关系进行建模可以深入了解言语和非言语交流，这是人类自然交互过程中使用的生产和协调机制的基础。 CA 可用于各种医疗保健应用，例如帮助听力障碍人士和向自闭症儿童教授社交技能。显示出类人行为来进行交流和承认积极倾听的辅导系统将更好地与学生互动，帮助他们学习。该项目为研究生和本科生的跨学科培训提供了肥沃的土壤。这些模型将通过与 UT 达拉斯分校学生互动的辅助代理（CA 或实体机器人）进行评估，作为接触所有专业学生的平台，特别是女性和代表性不足的少数族裔。该项目将采取综合、跨学科的方法通过探索言语、头部运动和面部表情之间的内在关系，并受到口语重要方面的约束，产生可信且有意义的行为。计划中的研究以综合的方式利用深度学习领域的一些最新进展，将声学特征和语义语言结构结合在一起，建立能够解释各种面部和头部运动之间相关性的模型。语音驱动的方法将以基于规则的方法无法轻易实现的方式捕获人类行为的可变性。对话行为和情感将被推断并用于约束语音驱动模型，捕获高级对话功能和面部手势之间的关系。该项目将提供新颖、原则性的方法来生成由合成语音驱动的行为，从而在只有文本可用时开辟新的应用领域。该方法将捕获合成语音中的声学变化，同时保持手势和语音之间的时间依赖性。该项目还将探索通过显示由我们的数据驱动框架生成的精心设计的手势来修改用户行为的方案。通过跟踪用户的行为，系统将提供适当的响应，从而形成交互的闭环。

项目成果

Meaningful head movements driven by emotional synthetic speech

由情感合成语音驱动的有意义的头部运动

• DOI: 10.1016/j.specom.2017.07.004
• 发表时间: 2017-12
• 期刊: Speech Communication
• 影响因子: 3.2
• 作者: Sadoughi, Najmeh;Liu, Yang;Busso, Carlos
• 通讯作者: Busso, Carlos

Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks

具有条件序列生成对抗网络的语音驱动的表达性说话嘴唇

• DOI: 10.1109/taffc.2019.2916031
• 发表时间: 2018-06-01
• 期刊: IEEE Transactions on Affective Computing
• 影响因子: 11.2
• 作者: Najmeh Sadoughi;C. Busso
• 通讯作者: C. Busso

Expressive Speech-Driven Lip Movements with Multitask Learning

具有多任务学习功能的富有表现力的言语驱动的嘴唇运动

• DOI: 10.1109/fg.2018.00066
• 发表时间: 2018-05-15
• 期刊: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)
• 作者: Najmeh Sadoughi;C. Busso
• 通讯作者: C. Busso

End-to-End Audiovisual Speech Recognition System With Multitask Learning

具有多任务学习的端到端视听语音识别系统

• DOI: 10.1109/tmm.2020.2975922
• 发表时间: 2024-09-13
• 期刊: IEEE Transactions on Multimedia
• 影响因子: 7.3
• 作者: Fei Tao;C. Busso
• 通讯作者: C. Busso

Joint Learning of Speech-Driven Facial Motion with Bidirectional Long-Short Term Memory

语音驱动的面部运动与双向长短期记忆的联合学习

• DOI: 10.1007/978-3-319-67401-8_49
• 发表时间: 2017-08-27
• 作者: Najmeh Sadoughi;C. Busso
• 通讯作者: C. Busso