Collaborative Research: NRI: FND: Graph Neural Networks for Multi-Object Manipulation

合作研究：NRI：FND：用于多对象操作的图神经网络

• 批准号: 2024057
• 负责人: Dieter Fox
• 金额: $ 40.5万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2024057&HistoricalAwards=false
• 关键词: Collaborative; Research; NRI; FND; Graph

For robots to act as ubiquitous assistants in daily life, they must regularly contend with environments involving many objects and objects built of many constituent parts. Current robotics research focuses on providing solutions to isolated manipulation tasks, developing specialized representations that do not readily work across tasks. This project seeks to enable robots to learn to represent and understand the world from multiple sensors, across many manipulation tasks. Specifically, the project will examine tasks in heavily cluttered environments that require multiple distinct picking and placing actions. This project will develop autonomous manipulation methods suitable for use in robotic assistants. Assistive robots stand to make a substantial impact in increasing the quality of life of older adults and persons with certain degenerative diseases. These methods also apply to manipulation in natural or man-made disasters areas, where explicit object models are not available. The tools developed in this project can also improve robot perception, grasping, and multi-step manipulation skills for manufacturing. With their ability to learn powerful representations from raw perceptual data, deep neural networks provide the most promising framework to approach key perceptual and reasoning challenges underlying autonomous robot manipulation. Despite​ ​their success, existing approaches scale poorly to the diverse set of scenarios autonomous robots will handle in natural environments. These current limitations of neural networks arise from being trained on isolated tasks, use of different architectures for different problems, and inability to scale to complex scenes containing a varying or large number of objects. This project hypothesizes that graph neural networks provide a powerful framework that can encode multiple sensor streams over time to provide robots with rich and scalable representations for multi-object and multi-task perception and manipulation. This project examines a number of extensions to graph neural networks in order to address current limitations for their use in autonomous manipulation. Furthermore this project examines novel ways of leveraging learned graph neural networks for manipulation planning and control in clutter and for multi-step, multi-object manipulation tasks. In order to train these large-scale graph net representations this project will use extremely large scale, physically accurate, photo-realistic simulation. All perceptual and behavior generation techniques developed in this project will be experimentally validated on a set of challenging real-world manipulation tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

为了使机器人在日常生活中充当无处不在的助手，他们必须定期包含涉及许多组成部分的许多物体和物体的环境。当前的机器人研究重点是为孤立的操纵任务提供解决方案，开发了不容易跨任务工作的专门表示。该项目旨在使机器人能够在许多操纵任务中学会从多个传感器中代表和理解世界。具体而言，该项目将检查在杂乱无章的环境中的任务，这些任务需要多个不同的选择和放置操作。该项目将开发适用于机器人助手的自主操纵方法。辅助机器人将对增加老年人和某些退化性疾病的人的生活质量产生次要影响。这些方法还适用于在没有明确对象模型的自然或人为灾难区域的操纵。该项目中开发的工具还可以改善机器人的感知，抓握和多步制造技巧。深层神经网络具有从原始感知数据中学习强大的表示的能力，提供了最有希望的框架，以应对自动驾驶机器人操纵的关键感知和推理挑战。尽管它们成功了，但现有方法的扩展很差，而不是对神经元网络的当前局限性的自主机器人组合，这是由于接受隔离任务的培训，使用不同的架构来解决不同的问题，并且无法扩展到包含各种物体或大量对象的复杂风景。该项目假设图形神经元网络提供了一个强大的框架，可以随着时间的推移编码多个传感器流，以为机器人提供丰富而可扩展的表示，以用于多任务和多任务感知和操作。该项目检查了图形中性网络的许多扩展，以解决其在自动操作中使用的当前限制。此外，该项目还研究了利用学习的图形中性网络的新颖方式，以在混乱中进行操纵计划和控制，并进行多步，多对象的操纵任务。为了训练这些大规模的图表净表示，该项目将使用非常大规模的，物理上精确的照片现实模拟。该项目中开发的所有知觉和行为产生技术将在一组挑战者现实世界中的操纵任务上进行实验验证。该奖项反映了NSF的法定任务，并通过使用该基金会的知识分子优点和更广泛的影响来评估标准。

项目成果

Unseen Object Instance Segmentation for Robotic Environments

• DOI: 10.1109/tro.2021.3060341
• 发表时间: 2020-07
• 期刊: IEEE Transactions on Robotics
• 影响因子: 7.8
• 作者: Christopher Xie;Yu Xiang;Arsalan Mousavian;D. Fox

M2T2: Multi-Task Masked Transformer for Object-centric Pick and Place

• DOI: 10.48550/arxiv.2311.00926
• 发表时间: 2023-11
• 作者: Wentao Yuan;Adithyavairavan Murali;Arsalan Mousavian;Dieter Fox

Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation

学习用于看不见的对象实例分割的 RGB-D 特征嵌入

• 发表时间: 2021
• 期刊: Conference on Robot Learning CoRL
• 作者: Xiang, Yu;Xie, Christopher;Mousavian, Arsalan;Fox, Dieter
• 通讯作者: Fox, Dieter

RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks

RICE：使用图神经网络在杂乱的环境中细化实例掩码

• 发表时间: 2021
• 期刊: Conference on Robot Learning
• 作者: Xie, Christopher;Mousavian, Arsalan;Xiang, Yu;Fox, Dieter
• 通讯作者: Fox, Dieter

SORNet: Spatial object-centric representations for sequential manipulation

SORNet：用于顺序操作的以空间对象为中心的表示

• 发表时间: 2022
• 期刊: Conference on Robot Learning
• 作者: Yuan, Wentao;Paxton, Chris;Desingh, Karthik;Fox, Dieter
• 通讯作者: Fox, Dieter