NRI: FND: Using Multi-Modal Data to Make Robotic Grasp Algorithms Aware of Human Preferences for Safe Collaborative Robot-Human Handover Interactions with Novel Objects

NRI：FND：使用多模态数据使机器人抓取算法了解人类偏好，以实现与新物体的安全协作机器人-人类切换交互

• 批准号: 2023998
• 负责人: Natasha Banerjee
• 金额: $ 30.49万
• 依托单位: Clarkson University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-15 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2023998&HistoricalAwards=false
• 关键词: NRI; FND; Using; Multi; Modal

This project contributes advancements in promoting safe collaborative robot-to-human handovers by making robots with manipulator arms aware of human preferences for interactions with objects. In environments such as healthcare facilities, warehousing, retail, engine repair, and aircraft assembly, where robots may be expected to collaborate with humans for successful accomplishment of tasks, it is essential that robotic manipulators hand over objects such that people can optimally hold them, without fear of the object falling or the person being injured by the gripper or arm, and without the inconvenience of the object being unreachable. To enable safe handovers, the project will provide algorithms that use data on human interactions with objects captured from multiple viewpoints to automatically predict preferred locations of human grasp on objects, optimal orientation and distance of the object from the person, and safe point of release of the object by manipulator grippers. The research team will reach out to two-year and four-year colleges with limited technological opportunities in the North Country to provide research opportunities to women and students from underrepresented communities.The project advances research in ubiquitous co-robots by providing holistic fine-grained insight through multi-modal sensing on natural behaviors of people as they interact with each other and with objects in their environments. The project accomplishes three objectives to address the gap on propagating understanding of human handover preferences to large collections of novel in-the-wild objects for customizability of co-robots to new environments. First, the research team will collect a large multi-viewpoint multi-modal dataset on two-person handovers and perform empirical analysis of the collected data to understand preferences on hold locations, end pose, and release point using subject ratings of object presentations. Modalities used will consist of depth cameras to acquire understanding on object geometry and spatial relationships, and thermal cameras to analyze locations of human contact based on heat transferred to object surfaces. This work will provide a quantitative decomposition of human preferences for handover parameters in terms of geometric form and functionality of objects. Second, the team will create perception algorithms based on probabilistic models to perform prediction of handover parameters ranked in order of preference using depth images of objects as input. This work enables equipping co-robots with human-like awareness of diversity in preferences, and the priorities that people assign to interactions. Third, the team will provide robotic manipulators that use the trained perception algorithms to perform handover manipulations on novel objects while being aware of human behavior. Successful accomplishment of the project activities will enable rapid propagation of robotic manipulators aware of human handover behavior to new objects and environments for enhanced social acceptability of co-robots.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目通过使带有机械臂的机器人了解人类与物体交互的偏好，为促进安全的协作机器人与人类的交接做出了贡献。在医疗保健设施、仓储、零售、发动机维修和飞机组装等环境中，机器人可能需要与人类协作以成功完成任务，因此机器人操纵器必须移交物体，以便人们可以最佳地握住它们，无需担心物体掉落或人员被抓手或手臂伤害，也无需担心物体无法触及的不便。为了实现安全交接，该项目将提供算法，使用从多个视角捕获的人类与物体交互的数据，自动预测人类抓握物体的首选位置、物体与人的最佳方向和距离，以及释放物体的安全点。通过机械手抓取物体。研究团队将接触北部国家技术机会有限的两年制和四年制大学，为来自代表性不足社区的妇女和学生提供研究机会。该项目通过提供整体细粒度的研究来推进无处不在的协作机器人的研究通过多模态感知来洞察人与人之间以及与环境中的物体互动时的自然行为。该项目实现了三个目标，以解决将人类移交偏好传播到大量新颖的野外物体的理解方面的差距，以实现协作机器人在新环境中的可定制性。首先，研究团队将收集有关两人交接的大型多视点多模态数据集，并对收集的数据进行实证分析，以利用对象呈现的主题评分来了解对保持位置、结束姿势和释放点的偏好。使用的模式将包括深度相机，用于了解物体的几何形状和空间关系，以及热感相机，用于根据传递到物体表面的热量来分析人体接触的位置。这项工作将根据对象的几何形式和功能对人类对切换参数的偏好进行定量分解。其次，该团队将创建基于概率模型的感知算法，以使用对象的深度图像作为输入，对按偏好顺序排列的切换参数进行预测。这项工作使协作机器人能够像人类一样意识到偏好的多样性以及人们分配给交互的优先级。第三，该团队将提供机器人操纵器，使用经过训练的感知算法对新物体进行切换操作，同时了解人类行为。该项目活动的成功完成将使能够意识到人类移交行为的机器人操纵器快速传播到新的物体和环境，从而提高协作机器人的社会接受度。该奖项反映了 NSF 的法定使命，并通过使用基金会的评估进行评估，认为值得支持。智力价值和更广泛的影响审查标准。

项目成果

Fantastic Breaks: A Dataset of Paired 3D Scans of Real-World Broken Objects and Their Complete Counterparts

• DOI: 10.1109/cvpr52729.2023.00454
• 发表时间: 2023-03
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: N. Lamb;C. Palmer;Benjamin Molloy;Sean Banerjee;N. Banerjee

Reinforcement-Learning Based Robotic Assembly of Fractured Objects Using Visual and Tactile Information

• DOI: 10.1109/icara56516.2023.10125938
• 发表时间: 2023-02
• 期刊: 2023 9th International Conference on Automation, Robotics and Applications (ICARA)
• 作者: Xinchao Song;N. Lamb;Sean Banerjee;N. Banerjee

Studying How Object Handoff Orientations Relate to Subject Preferences on Handover

• DOI: 10.1109/arso56563.2023.10187566
• 发表时间: 2023-06
• 期刊: 2023 IEEE International Conference on Advanced Robotics and Its Social Impacts (ARSO)
• 作者: N. Wiederhold;Mingjun Li;N. Lamb;DiMaggio Paris;Alaina Tulskie;Sean Banerjee;N. Banerjee