NRI: Integrating Perception and Manipulation of Deformable Objects by Learning Implicit Representations

NRI：通过学习隐式表示来集成可变形物体的感知和操纵

• 批准号: 2220876
• 负责人: Nima Fazeli
• 金额: $ 74.97万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2220876&HistoricalAwards=false
• 关键词: NRI; Integrating; Perception; Manipulation; Deformable

The goal of this National Robotics Initiative 3.0 project is to develop models and algorithms that enable autonomous robotic manipulation of deformable objects by integrating sight and touch for contact-rich interactions. Manipulation of deformable objects is essential in many day-to-day tasks ranging from cooking (e.g., using a compliant spatula to scrape a wok) to manufacturing (e.g., assembly tasks involving flexible parts and materials such as composites and cables). These tasks require the robot to perform complex maneuvers to bring deformable objects into contact and apply forces to bend and shape them to achieve tasks. Recent advances in collaborative robots (e.g., the Franka Emika Panda and Kuka LBR iiwa robots) have made robotic manipulation of deformable objects physically possible but robots still lack the models and algorithms necessary to perform practical tasks such as cooking, cleaning, and flexible manufacturing. This project will have broad societal impact through its applications in in-home assistive and manufacturing robotics. The ability to robustly manipulate deformable structures is an important precursor technology towards realizing intelligent robotic assistants. Robotics and their assistive applications have the potential to inspire children to pursue careers in STEM fields and meet the needs of America's growing assistive and manufacturing robotics industry. Integration of the research activities with education will emphasize actively involving undergraduates in research activities and introducing new lecture material and projects into undergraduate and graduate courses. Also, special emphasis will be given to recruit qualified students from under-represented groups.This project will develop integrated methods for perception and manipulation of elastically deformable objects via novel implicit function representations. Elastic objects are ubiquitous in people's lives, from spatulas and sponges in the kitchens to elastic rods on manufacturing floors to surgical tools and tissue. Their manipulation is an essential skill for practical robotic systems. The proposed integrative approach brings together methods from computer vision and robotics to address the need for seamless visio-tactile reasoning and acting that is essential for robotics applications. Specifically, this approach integrates and extends recent theoretical and computational advances in implicit function learning to model, perceive, and intelligently manipulate elastic objects. Implicit representations of 3D geometries have recently gained traction in computer vision owing to their compactness and computational efficiency; however, they have yet to be deployed successfully for robotics. Towards their successful integration, the outcomes of this project are: 1) A generalization of implicit function theory and algorithms for 3D visio-tactile deformable object representations. This addresses the need for dynamic and multi-modal deformable object representations particularly suited for robotics; specially to streamline the sense-reason-act pipeline. 2) Algorithmic tools for implicit state-estimation. These algorithms will bridge inherently uncertain robotic sensing modalities with implicit representations. 3) Theoretic and algorithmic foundations for implicit control and planning. The resulting tools from this project will enable real-time closed-loop control of deformable objects for practical robotic systems. For evaluation, this project will explore the integration of these algorithms for assistive robotics (food preparation) and warehouse logistics (dense packing). The research outcomes contribute to several related fields including continuum mechanics, computer vision, and learning theory where object deformations subject to boundary conditions and/or implicit function theory are studied. In particular, this approach will offer a flexible, generalizable, and computationally efficient alternative to the current state-of-the-art methods using finite element analysis or particle models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该国家机器人计划3.0项目的目标是开发模型和算法，通过将视力和触摸集成到接触良好的互动中，可以自主的机器人操纵可变形物体。在许多日常任务中，对可变形物体的操纵至关重要，从烹饪（例如，使用合规的刮刀刮擦炒锅）到制造业（例如，涉及灵活的零件和材料（例如复合材料和电缆））的制造业（例如，组装任务）。这些任务要求机器人执行复杂的操作，以使可变形的物体接触并施加力以弯曲和塑造其完成任务。最近的协作机器人（例如，Franka Emika Panda和Kuka LBR IIWA机器人）的最新进展使机器人操纵对可变形物体的操纵在物理上可能成为可能，但机器人仍然缺乏执行诸如烹饪，清洁和灵活制造等实用任务所需的模型和算法。该项目将通过其在家庭辅助和制造机器人技术中的应用产生广泛的社会影响。坚强操纵可变形结构的能力是实现智能机器人助手的重要先驱技术。机器人技术及其辅助应用有可能激发儿童从事STEM领域的职业，并满足美国日益增长的辅助和制造机器人行业的需求。研究活动与教育的整合将强调积极参与研究活动的大学生，并将新的讲座材料和项目引入本科和研究生课程。此外，将特别强调来自代表性不足的群体的合格学生。该项目将通过新颖的隐式功能表示，开发综合方法来感知和操纵弹性变形对象。弹性物体在人们的生活中无处不在，从厨房中的spatulas和海绵，到制造地板上的弹性杆，再到手术工具和组织。他们的操纵是实用机器人系统的重要技能。所提出的集成方法将计算机视觉和机器人技术的方法汇集在一起​​，以满足对机器人应用必不可少的无缝粘性静态推理和行动的需求。具体而言，这种方法集成并扩展了隐性功能学习的最新理论和计算进步，以建模，感知和智能地操纵弹性对象。由于其紧凑性和计算效率，3D几何形状的隐式表示最近在计算机视觉方面受到了关注。但是，它们尚未成功部署用于机器人技术。为了成功地集成，该项目的结果是：1）3D粘膜可变形对象表示的隐式函数理论和算法的概括。这解决了对特别适合机器人技术的动态和多模式变形对象表示的需求。特别是为了简化感官 - 季节动作管道。 2）用于隐式状态估计的算法工具。这些算法将以隐式表示固有不确定的机器人传感方式桥接。 3）理论和算法基础，用于隐式控制和计划。该项目的最终工具将实现对实用机器人系统的可变形对象的实时闭环控制。为了进行评估，该项目将探讨这些算法的辅助机器人技术（食品准备）和仓库物流（密集包装）的集成。研究结果有助于几个相关领域，包括连续力学，计算机视觉和学习理论，其中研究了受边界条件和/或隐式函数理论的对象变形。特别地，这种方法将提供使用有限元分析或粒子模型的当前最新方法的灵活，可推广和计算有效的替代方法。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的智力优点和更广泛的影响审查标准通过评估来进行评估的。

项目成果

Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specification

• DOI: 10.1109/icra48891.2023.10161125
• 发表时间: 2023-03
• 期刊: 2023 IEEE International Conference on Robotics and Automation (ICRA)
• 作者: Jiayi Pan;Glen Chou;D. Berenson

Focused Adaptation of Dynamics Models for Deformable Object Manipulation

• DOI: 10.1109/icra48891.2023.10161366
• 发表时间: 2022-09
• 期刊: 2023 IEEE International Conference on Robotics and Automation (ICRA)
• 作者: P. Mitrano;A. LaGrassa;Oliver Kroemer;D. Berenson

Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control

通过膜进行操纵：高分辨率和高变形触觉传感和控制

• 发表时间: 2022
• 期刊: Conference on Robot Learning (CoRL
• 作者: Oller, Miquel;Berenson, Dmitry;Fazeli Nima
• 通讯作者: Fazeli Nima