NRI:FND: Unifying standard physics-based control with learning-based perception and action to enable safe and agile object manipulation using unmanned aerial vehicles

NRI：FND：将基于物理的标准控制与基于学习的感知和行动相结合，以使用无人机实现安全、敏捷的物体操纵

• 批准号: 1925189
• 负责人: Marin Kobilarov
• 金额: $ 74.96万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-15 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1925189&HistoricalAwards=false
• 关键词: NRI; FND; Unifying; standard; physics

Flying robots capable of object manipulation will enable new applications such as load pickup and delivery, infrastructure inspection and repair, agricultural crop management and harvesting. Currently though, aerial vehicles are limited in their agility and robustness when in close contact with their surroundings. More specifically, controlling aerial robots to interact with the natural environment requires complex models for inferring object dynamics in real-time through shape and appearance, dealing with contact and compliance, relying on complex perceptual cues such as occlusions or shadows, while at the same time ensuring safety and reliability. Currently, standard algorithms for robotic perception and control are not sufficient for such tasks. While machine learning techniques have proven powerful for vision-based perception and more recently for control in simple environments, current learning techniques are not directly suitable for agile autonomous vehicles where safety is critical and failed actions can be fatal for the robot and humans around it. To overcome these challenges, this project proposes a framework that combines standard control methods with learning-based perception and action in an integrated framework equipped with formal high-confidence guarantees on performance. The proposed methodology aims to enable autonomous vehicles to accomplish tasks that are currently impossible or infeasible to achieve with standard methods. The project will develop computational theory and algorithms that combine standard, i.e. physics and logic-based, control methods with learning-based control, implement a software framework and apply it to aerial manipulation tasks. More specifically, a fully differentiable framework will be developed that integrates components with known dynamics based on classical physical state representation and components that adapt to a given task through a learned implicit state representation that captures rich inertial and visual sensing. Then, a methodology for robust policy optimization with safety certificates will be developed based on high-fidelity stochastic models learned from robot data and then used to compute action policies in simulation using learned synthetic sensor models. The policies can be equipped with high-confidence formal bounds on performance and safety, which are validated and adapted in the real world. As a result, the robotic system can operate efficiently with guarantees on performance and safety. Finally, a fault-tolerant autonomy software framework will be implemented and the algorithms validated using three applications of aerial manipulation: object pick-up and transport in cluttered environments; remote sensor placement and infrastructure inspection; agricultural crop sampling and management. The proposed theory and methods are generally applicable to any robotic system operating in challenging environments, beyond aerial vehicles.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

能够操纵物体的飞行机器人将带来新的应用，例如货物拾取和运送、基础设施检查和维修、农作物管理和收割。但目前，飞行器在与周围环境密切接触时，其灵活性和稳健性受到限制。更具体地说，控制空中机器人与自然环境交互需要复杂的模型，通过形状和外观实时推断物体动态，处理接触和顺从性，依赖复杂的感知线索，如遮挡或阴影，同时确保安全可靠。目前，机器人感知和控制的标准算法不足以完成此类任务。虽然机器学习技术已被证明对于基于视觉的感知以及最近在简单环境中的控制来说非常强大，但当前的学习技术并不直接适用于敏捷的自动驾驶车辆，因为安全至关重要，失败的操作可能对机器人和周围的人类造成致命的影响。为了克服这些挑战，该项目提出了一个框架，将标准控制方法与基于学习的感知和行动结合在一个集成框架中，并配备正式的高可信度性能保证。所提出的方法旨在使自动驾驶车辆能够完成目前使用标准方法不可能或不可行的任务。该项目将开发计算理论和算法，将标准的（即基于物理和逻辑的）控制方法与基于学习的控制相结合，实现软件框架并将其应用于空中操纵任务。更具体地说，将开发一个完全可微的框架，该框架将基于经典物理状态表示的已知动力学组件与通过学习的隐式状态表示（捕获丰富的惯性和视觉感知）适应给定任务的组件相集成。然后，将基于从机器人数据中学习的高保真随机模型开发一种具有安全证书的稳健策略优化方法，然后用于使用学习的合成传感器模型在模拟中计算动作策略。这些政策可以配备高可信度的性能和安全正式界限，并在现实世界中进行验证和调整。因此，机器人系统可以在性能和安全性得到保证的情况下高效运行。最后，将实施容错自主软件框架，并使用空中操纵的三种应用来验证算法：杂乱环境中的物体拾取和运输；远程传感器放置和基础设施检查；农作物采样和管理。所提出的理论和方法通常适用于在飞行器之外的挑战性环境中运行的任何机器人系统。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Perception-Based UAV Fruit Grasping Using Sub-Task Imitation Learning

• DOI: 10.1109/airpharo52252.2021.9571066
• 发表时间: 2021-10
• 期刊: 2021 Aerial Robotic Systems Physically Interacting with the Environment (AIRPHARO)
• 作者: Gabriel Baraban;Siddharth Kothiyal;Marin Kobilarov

Learn Proportional Derivative Controllable Latent Space from Pixels

• DOI: 10.1109/case49997.2022.9926535
• 发表时间: 2021-10
• 期刊: 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE)
• 作者: Weiyao Wang;Marin Kobilarov;Gregory Hager

Improving the Reliability of Pick-and-Place With Aerial Vehicles Through Fault-Tolerant Software and a Custom Magnetic End-Effector

通过容错软件和定制磁性末端执行器提高飞行器取放的可靠性

• DOI: 10.1109/lra.2021.3093864
• 发表时间: 2021
• 期刊: IEEE Robotics and Automation Letters
• 影响因子: 5.2
• 作者: Garimella, Gowtham;Sheckells, Matthew;Kim, Soowon;Baraban, Gabriel;Kobilarov, Marin
• 通讯作者: Kobilarov, Marin

Robust Policy Search for an Agile Ground Vehicle Under Perception Uncertainty

• DOI: 10.1109/iros51168.2021.9636552
• 发表时间: 2021-09
• 期刊: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
• 作者: S. Sefati;Subhransu Mishra;Matthew Sheckells;Kapil D. Katyal;Jin Bai;Gregory Hager;Marin Kobilarov