CRII: HCC: Modeling computer-mediated task-oriented dialogues with multi-modality information theoretic approaches

CRII：HCC：用多模态信息理论方法对计算机介导的面向任务的对话进行建模

• 批准号: 2105192
• 负责人: Yang Xu
• 金额: $ 16.69万
• 依托单位: San Diego State University Foundation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-06-01 至 2024-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2105192&HistoricalAwards=false
• 关键词: CRII; HCC; Modeling; computer; mediated

Effective real-time remote communication is challenging because unlike face-to-face conversations, information is exchanged indirectly through computer screens and speakers. In these situations, rich non-verbal information such as body poses, hand gestures, and facial expressions is often blocked or neglected. In order to improve the quality of existing remote communication systems, this project will develop an in-depth quantitative understanding of the kinds of information non-verbal aspects of communication might add to the conversation. The research team will address this issue by developing novel measures of non-verbal communication content based on state-of-the-art machine learning and computer vision techniques. The team will also develop software toolkits that can support both behavioral scientists who study conversation and designers of future computer-mediated communication systems. The research team proposes a series of studies that use information theoretic models to distill information from the non-verbal communication channels, including facial expressions, upper-body gestures, and whole-body poses, and synthesize them into the ultimate goal of predicting the success of computer-mediated collaborative tasks. As part of the work the team will develop several novel analysis methods. These include discretizing and symbolizing non-verbal information through joint representation learning and aggregation, using neural sequential models to estimate the amount of information in parallel discretized series, and using the temporal-spectral patterns in all communication channels to predict the task success. A multimodal information incorporated corpus of task-oriented dialogues will also be collected as part of the work.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

有效的实时远程通信具有挑战性，因为与面对面的对话不同，信息是通过计算机屏幕和扬声器间接交换的。 在这些情况下，经常被阻止或忽略丰富的非语言信息，例如身体姿势，手势和面部表情。为了提高现有远程通信系统的质量，该项目将对对话的非语言方面进行深入的定量了解。研究团队将通过基于最先进的机器学习和计算机视觉技术的非语言通信内容来解决这个问题。团队还将开发软件工具包，这些工具包可以支持研究对话的行为科学家和未来计算机介导的通信系统的设计师。研究小组提出了一系列研究，使用信息理论模型从非语言交流渠道中提炼信息，包括面部表情，上体的手势和全身姿势，并将它们综合为预测计算机介导的协作任务成功的最终目标。 作为工作的一部分，团队将开发几种新颖的分析方法。 这些包括使用神经顺序模型通过联合表示学习和聚合来离散和象征非语言信息，以平行离散的序列估算信息量，并使用所有通信渠道中的时间光谱模式来预测任务成功。作为工作的一部分，还将收集一个多式联运信息，并将以任务为导向的对话集成。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛的影响评估审查标准通过评估来获得支持的。

项目成果

Spontaneous gestures encoded by hand positions improve language models: An Information-Theoretic motivated study

由手部位置编码的自发手势改善了语言模型：一项信息理论驱动的研究

• 发表时间: 2023
• 期刊: Proceedings of the conference Association for Computational Linguistics Meeting
• 作者: Xu, Yang