RI: Small: Understanding Subtle Non-Social Facial Expressivity to Boost Learning and Computer Interaction

RI：小：理解微妙的非社交面部表情以促进学习和计算机交互

• 批准号: 1911197
• 负责人: Bir Bhanu
• 金额: $ 50万
• 依托单位: University of California-Riverside
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1911197&HistoricalAwards=false
• 关键词: RI; Small; Understanding; Subtle; Non

Facial expressions play a significant role in everyday communication among humans. Computer understanding of these complex and subtle expressions will lead to highly capable interactive cyber-human systems with proactive computers that make more appropriate responses to human interactions. This project brings together an interdisciplinary team of investigators to address key challenges associated with spontaneous microexpression recognition in non-social scenarios. The project concentrates on generating bio-feedback from humans while learning skills, such as online learning, and being recorded and analyzed in continuous color and depth video streams. It will develop computer algorithms for human-machine synergy and test how this information can provide for superior learning when training applications are augmented with expression-informed bio-feedback in near real-time. This represents a significant step forward in training machines to recognize and classify facial microexpressions and maximizing the synergy of cyber-human systems that will improve the quality of life experiences. It will provide a computing environment within the reach of common people in which the interests or even the health of people can be detected and predicted, with significant impacts on skill learning, education and information retrieval.The project develops an approach to the understanding of complex and subtle facial microexpressions and bio-feedback where the synergy between cyber and human systems can be fully exploited. It addresses key challenges associated with computational understanding and modeling of intelligence in challenging, realistic contexts. It uses assessment and intervention based on facial microexpressions to maximize synergy of cyber and human systems for skill learning. First, it considers deep learning and closed-loop video analysis for optimized skill learning in a reinforcement learning framework. Second, it develops novel representation of facial microexpressions from color and depth video streams and use them for person independent emotion recognition as well as person-specific emotions recognition when a learning task is adapted. Third, it exploits not only the color camera but also the integrated depth camera for precise measurements, which has not been used for microexpressions. The focus is to determine the extent to which real-time classification of microexpressions can provide for more appropriate interactivity that will facilitate human learning in real applications. The results will be broadly disseminated through a website that will have regular releases of databases and software tools by offering tutorials, workshops and demos at major professional meetings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

面部表情在人类之间的日常交流中起着重要作用。对这些复杂而微妙的表达式的计算机理解将导致具有积极主动的计算机的高功能交互式网络人类系统，从而对人类互动做出更适当的反应。该项目汇集了一个跨学科的研究人员团队，以应对与自发的微表达识别相关的关键挑战。该项目集中于在学习技能（例如在线学习）以及以连续的色彩和深度视频流进行录制和分析的同时，从人类那里产生生物反馈。它将开发用于人机协同作用的计算机算法，并测试当培训应用程序在近乎实时的表达信息中增强培训应用时如何提供出色的学习。这代表了训练机迈出的重要一步，以识别和对面部微表达进行分类，并最大程度地提高网络人类系统的协同作用，以改善生活质量的体验。它将在普通民族的范围内提供一个计算环境，在这种环境中，可以检测和预测人们的利益甚至人的健康，对技能学习，教育和信息检索产生重大影响。该项目开发了一种方法来理解对复杂和微妙的面部微表达以及对网络和人类系统之间的协同作用的复杂和生物反馈的方法。它解决了与挑战，现实的环境中智能的计算理解和建模有关的关键挑战。它使用基于面部微表达的评估和干预措施来最大程度地提高网络和人类系统的协同作用，以进行技能学习。首先，它考虑了深度学习和闭环视频分析，以在强化学习框架中优化技能学习。其次，它从颜色和深度视频流中开发了面部微表达的新颖表示，并将其用于人独立的情感识别，以及在适应学习任务时特定于人的情绪识别。第三，它不仅利用了彩色摄像头，还利用了集成的深度摄像头进行精确的测量，这尚未用于微表达。重点是确定微表达的实时分类可以为更合适的互动性提供多大分类，从而有助于人类在实际应用中学习。结果将通过一个网站大致传播，该网站将通过在主要专业会议上提供教程，讲习班和演示来定期发布数据库和软件工具。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛影响的审查标准来通过评估来支持的。

项目成果

Micro-Expression Classification based on Landmark Relations with Graph Attention Convolutional Network

基于图注意力卷积网络的地标关系微表情分类

• 发表时间: 2021
• 期刊: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR
• 作者: Kumar, Ankith Jain;Bhanu, Bir
• 通讯作者: Bhanu, Bir

Towards Visually Explaining Variational Autoencoders

• DOI: 10.1109/cvpr42600.2020.00867
• 发表时间: 2019-11
• 期刊: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Wenqian Liu;Runze Li;Meng Zheng;S. Karanam;Ziyan Wu;B. Bhanu;R. Radke;O. Camps

Depth Videos for the Classification of Micro-Expressions

• DOI: 10.1109/icpr48806.2021.9412976
• 发表时间: 2021-01
• 期刊: 2020 25th International Conference on Pattern Recognition (ICPR)
• 作者: A. Kumar;B. Bhanu;Christopher Casey;S. Cheung;A. Seitz

Three Stream Graph Attention Network using Dynamic Patch Selection for the classification of micro-expressions

• DOI: 10.1109/cvprw56347.2022.00277
• 发表时间: 2022-06
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
• 作者: Ankith Jain Rakesh Kumar;B. Bhanu