NRI:FND: Unifying standard physics-based control with learning-based perception and action to enable safe and agile object manipulation using unmanned aerial vehicles

NRI：FND：将基于物理的标准控制与基于学习的感知和行动相结合，以使用无人机实现安全、敏捷的物体操纵

• 批准号: 1925189
• 负责人: Marin Kobilarov
• 金额: $ 74.96万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-15 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1925189&HistoricalAwards=false
• 关键词: NRI; FND; Unifying; standard; physics

Flying robots capable of object manipulation will enable new applications such as load pickup and delivery, infrastructure inspection and repair, agricultural crop management and harvesting. Currently though, aerial vehicles are limited in their agility and robustness when in close contact with their surroundings. More specifically, controlling aerial robots to interact with the natural environment requires complex models for inferring object dynamics in real-time through shape and appearance, dealing with contact and compliance, relying on complex perceptual cues such as occlusions or shadows, while at the same time ensuring safety and reliability. Currently, standard algorithms for robotic perception and control are not sufficient for such tasks. While machine learning techniques have proven powerful for vision-based perception and more recently for control in simple environments, current learning techniques are not directly suitable for agile autonomous vehicles where safety is critical and failed actions can be fatal for the robot and humans around it. To overcome these challenges, this project proposes a framework that combines standard control methods with learning-based perception and action in an integrated framework equipped with formal high-confidence guarantees on performance. The proposed methodology aims to enable autonomous vehicles to accomplish tasks that are currently impossible or infeasible to achieve with standard methods. The project will develop computational theory and algorithms that combine standard, i.e. physics and logic-based, control methods with learning-based control, implement a software framework and apply it to aerial manipulation tasks. More specifically, a fully differentiable framework will be developed that integrates components with known dynamics based on classical physical state representation and components that adapt to a given task through a learned implicit state representation that captures rich inertial and visual sensing. Then, a methodology for robust policy optimization with safety certificates will be developed based on high-fidelity stochastic models learned from robot data and then used to compute action policies in simulation using learned synthetic sensor models. The policies can be equipped with high-confidence formal bounds on performance and safety, which are validated and adapted in the real world. As a result, the robotic system can operate efficiently with guarantees on performance and safety. Finally, a fault-tolerant autonomy software framework will be implemented and the algorithms validated using three applications of aerial manipulation: object pick-up and transport in cluttered environments; remote sensor placement and infrastructure inspection; agricultural crop sampling and management. The proposed theory and methods are generally applicable to any robotic system operating in challenging environments, beyond aerial vehicles.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

能够进行物体操纵的飞行机器人将实现新的应用，例如拾取和交付，基础设施检查和维修，农作物管理和收获。但是，目前，与周围环境密切接触时，航空车的敏捷性和稳健性受到限制。更具体地说，控制空中机器人与自然环境相互作用需要复杂的模型，以通过形状和外观实时推断对象动态，处理接触和合规性，并依靠复杂的感知提示，例如遮挡或阴影，同时确保安全性和可靠性。当前，用于机器人感知和控制的标准算法不足以完成此类任务。尽管机器学习技术已被证明对基于视觉的感知有效，而最近在简单环境中控制了控制技术，但当前的学习技术并不直接适合安全性至关重要的敏捷自动驾驶汽车，而失败的动作对于机器人和周围的人类可能是致命的。为了克服这些挑战，该项目提出了一个将标准控制方法与基于学习的感知和行动相结合的框架，并在配备正式的高信任保证绩效的综合框架中。拟议的方法旨在使自动驾驶汽车能够完成目前不可能或无法实现标准方法的任务。该项目将开发结合标准的计算理论和算法，即物理和基于逻辑的控制方法与基于学习的控制，实施软件框架并将其应用于航空操纵任务。更具体地说，将开发一个完全可区分的框架，该框架将组件与已知的动力学集成基于经典的物理状态表示和通过捕获丰富的惯性和视觉传感的知识隐式状态表示来适应给定任务的组件。然后，将根据从机器人数据中学到的高保真随机模型开发出一种具有安全证书的强大策略优化方法，然后使用学识渊博的合成传感器模型在模拟中计算动作策略。这些政策可以在绩效和安全性方面具有高信心的正式界限，这些界限在现实世界中得到了验证和改编。结果，机器人系统可以有效运行，并保证绩效和安全性。最后，将实现容忍故障的自治软件框架，并使用空中操纵的三个应用程序验证了算法：在混乱的环境中进行对象拾取和运输；遥感和基础设施检查；农作物采样和管理。拟议的理论和方法通常适用于在充满挑战的环境中运行的任何机器人系统，除航空器之外。该奖项反映了NSF的法定任务，并且使用基金会的智力优点和更广泛的影响审查标准，被认为值得通过评估来提供支持。

项目成果

Perception-Based UAV Fruit Grasping Using Sub-Task Imitation Learning

• DOI: 10.1109/airpharo52252.2021.9571066
• 发表时间: 2021-10
• 期刊: 2021 Aerial Robotic Systems Physically Interacting with the Environment (AIRPHARO)
• 作者: Gabriel Baraban;Siddharth Kothiyal;Marin Kobilarov

Learn Proportional Derivative Controllable Latent Space from Pixels

• DOI: 10.1109/case49997.2022.9926535
• 发表时间: 2021-10
• 期刊: 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE)
• 作者: Weiyao Wang;Marin Kobilarov;Gregory Hager

Improving the Reliability of Pick-and-Place With Aerial Vehicles Through Fault-Tolerant Software and a Custom Magnetic End-Effector

通过容错软件和定制磁性末端执行器提高飞行器取放的可靠性

• DOI: 10.1109/lra.2021.3093864
• 发表时间: 2021
• 期刊: IEEE Robotics and Automation Letters
• 影响因子: 5.2
• 作者: Garimella, Gowtham;Sheckells, Matthew;Kim, Soowon;Baraban, Gabriel;Kobilarov, Marin
• 通讯作者: Kobilarov, Marin

Robust Policy Search for an Agile Ground Vehicle Under Perception Uncertainty

• DOI: 10.1109/iros51168.2021.9636552
• 发表时间: 2021-09
• 期刊: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
• 作者: S. Sefati;Subhransu Mishra;Matthew Sheckells;Kapil D. Katyal;Jin Bai;Gregory Hager;Marin Kobilarov