RI: Small: Understanding Hand Interaction In The Jumble of Internet Videos

RI：小：在混乱的互联网视频中理解手部交互

• 批准号: 2426592
• 负责人: David Fouhey
• 金额: $ 43.7万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-01-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2426592&HistoricalAwards=false
• 关键词: RI; Small; Understanding; Hand; Interaction

Hands are the primary way that humans interact with and manipulate the world. Intelligent machines will need to be able to understand how humans use their hands if they are to understand human actions and to work in the world humans have built with their hands. Unfortunately, videos that show people using their hands are surprisingly difficult to understand for current artificial intelligence (AI) systems. Hands may be temporarily hidden as people interact with objects, and even if they are visible, hands can interact with a myriad of different objects ranging from refrigerator handles to coffee mugs to garage door openers. This project develops systems that can enable learning about how humans use their hands from large scale Internet video data. As hands are central to many other areas of study, this project has the potential to empower research in many other disciplines. For instance, robotics researchers may use the systems to teach robots how to interact with objects by observation. Similarly, kinesiologists and mechanical engineers who study how the human hand is used could use the systems to better quantify hand motions and thus improve the lives of people. This project aims to achieve its goal via three technical directions that together advance the science of understanding human activities and affordances (human/object interaction). The first direction of the project will build systems for automatically parsing hand interaction data from large-scale video. The goal of this direction is to understand what the hand is doing in terms of interaction with the world in physical terms as opposed to via naming the interaction with nouns and verbs. To help understand the context of an interaction, the second direction aims to build learning-based systems that can understand human poses from partial observations that occur naturally in video data. Finally, the third direction puts these systems together by building a graph of interaction where hand interaction examples are nodes, and edges are induced by observations of human pose. This web of interactions will enable systems to learn about how humans can manipulate objects from large-scale data across viewpoints and examples and enable new applications of computer vision.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

手是人类与世界互动和操纵世界的主要方式。如果智能机器想要理解人类的行为并在人类用手构建的世界中工作，那么它们就需要能够理解人类如何使用双手。不幸的是，对于当前的人工智能（AI）系统来说，展示人们用手的视频非常难以理解。当人们与物体互动时，手可能会暂时隐藏起来，即使它们是可见的，手也可以与无数不同的物体互动，从冰箱把手到咖啡杯再到车库门开启器。该项目开发的系统可以从大规模互联网视频数据中了解人类如何使用双手。由于手是许多其他研究领域的核心，因此该项目有潜力促进许多其他学科的研究。例如，机器人研究人员可以使用这些系统来教机器人如何通过观察与物体交互。同样，研究人手如何使用的运动学家和机械工程师可以使用这些系统更好地量化手部运动，从而改善人们的生活。该项目旨在通过三个技术方向实现其目标，这三个方向共同推进理解人类活动和可供性（人/物交互）的科学。该项目的第一个方向将构建用于自动解析大规模视频中的手部交互数据的系统。这个方向的目标是通过物理术语来理解手在与世界的交互方面正在做什么，而不是通过名词和动词来命名交互。为了帮助理解交互的背景，第二个方向旨在构建基于学习的系统，该系统可以从视频数据中自然发生的部分观察中理解人体姿势。最后，第三个方向通过构建交互图将这些系统组合在一起，其中手部交互示例是节点，边缘是通过对人体姿势的观察而产生的。这种交互网络将使系统能够了解人类如何通过观点和示例从大规模数据中操纵对象，并实现计算机视觉的新应用。该奖项反映了 NSF 的法定使命，并通过使用基金会的评估进行评估，被认为值得支持。智力价值和更广泛的影响审查标准。