CAREER:Towards Causal Multi-Modal Understanding with Event Partonomy and Active Perception

职业：通过事件部分和主动感知实现因果多模态理解

基本信息

批准号：
2348690
负责人：
Sathyanarayanan Aakur
金额：
$ 51.42万
依托单位：
Auburn University
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2023
资助国家：
美国
起止时间：
2023-10-01 至 2027-05-31
项目状态：
未结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2348690&HistoricalAwards=false
关键词：
CAREER Towards Causal Multi Modal

项目摘要

Events are central to causal, visual understanding in complex, dynamic environments. From collaborative robots that assist humans with complex tasks to surveillance systems that detect anomalous behavior, there is a need to understand events, their composition, and their interaction for effective machine perception. This project will explore how events are structured in multimodal data and how they can be leveraged to help design better, embodied agents that can construct and leverage compositional event representations to help function in complex, real-world environments. The developed algorithms could have a broad impact in numerous fields including Artificial Intelligence (AI) and education, such as the future of workforce training. In addition to scientific impact, the project performs complementary educational and outreach activities. Specifically, it engages the broader scientific community in the use of AI and computer vision (CV) research to augment the future of workforce training through workshops and seminars, introduces and enhances the AI and CV education at Oklahoma State University, and develops and fosters an entrepreneurial mindset in computer science education and research through integrated educational activities.The research focuses on the ideas of energy-based neuro-symbolic learning, using Grenander’s Pattern Theory formalism, abductive reasoning, and active embodied vision for learning and using temporal causality for richer, multimodal event understanding. The specific research aims of the project are three-fold. First, it seeks to learn the partonomy of common, everyday events by expressing the hierarchical structure in the form of Bayesian Rose Trees, whose semantics are populated by an energy-based pattern theory inference engine. Second, it will research ways to leverage this event partonomy into understanding actions in videos beyond recognition and perceive the current action in the context of the overall task being performed. This inference mechanism will enable an embodied, intelligent agent to recognize the current action and infer higher-level concepts such as human intent and goals in a unified energy-based framework. Third, it will realize the partonomy-based understanding framework in an embodied agent while augmenting it with active multimodal feedback. It will allow the embodied agent to perform active reasoning through feedback from the environment by controlling its geometric parameters (such as position, orientation, and pose) to navigate clutter and resolve any ambiguity in the perceived event structure. This project is jointly funded by Robust Intelligence (RI) Program and the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

事件是复杂动态环境中因果视觉理解的核心，从协助执行复杂任务的协作机器人到检测异常行为的监视系统，都需要了解事件、事件的组成及其交互，以实现有效的机器感知。该项目将探索如何在多模态数据中构建事件，以及如何利用它们来帮助设计更好的具体代理，这些代理可以构建和利用组合事件表示来帮助在复杂的现实环境中发挥作用。所开发的算法可能会产生广泛的影响。在许多领域，包括人工智能 (AI) 和教育，例如劳动力培训的未来除了科学影响外，该项目还开展补充性教育和推广活动，让更广泛的科学界参与人工智能和计算机视觉 (CV) 的使用。研究通过讲习班和研讨会来增强劳动力培训的未来，介绍和加强俄克拉荷马州立大学的人工智能和计算机视觉教育，并通过综合教育活动发展和培养计算机科学教育和研究的创业思维。该研究重点是基于能量的神经符号的思想该项目的具体研究目标有三个：首先，它旨在学习共同的部分性。，通过以贝叶斯玫瑰树的形式表达层次结构，其语义由基于能量的模式理论推理引擎填充，其次，它将研究如何利用这种事件部分来理解视频中无法识别的动作。并在正在执行的整体任务的背景下感知当前的行动，这种推理机制将使具体的智能代理能够识别当前的行动并在统一的基于能量的框架中推断出更高层次的概念，例如人类意图和目标。第三，它将在实体主体中实现基于部分的理解框架，同时通过主动多模态反馈对其进行增强。它将允许实体主体通过控制其几何参数（例如位置、方向、和姿势）来导航混乱并解决任何问题该项目由鲁棒情报 (RI) 计划和刺激竞争性研究既定计划 (EPSCoR) 联合资助。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力评估进行评估，认为值得支持。优点和更广泛的影响审查标准。