FMRG: Adaptable and Scalable Robot Teleoperation for Human-in-the-Loop Assembly

FMRG：用于人在环装配的适应性和可扩展的机器人远程操作

• 批准号: 2037101
• 负责人: Steven Feiner
• 金额: $ 374.92万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-01-01 至 2025-12-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2037101&HistoricalAwards=false
• 关键词: FMRG; Adaptable; Scalable; Robot; Teleoperation

The COVID-19 pandemic has accelerated the adoption of remote working in many industries. The ability for employees to work remotely, often from home, has become crucial to an organization's long-term resilience and growth potential. However, while advances in software and networking have made it possible for information workers to work remotely, most manufacturing workers cannot, because the infrastructure that is needed doesn't exist. This Future Manufacturing (FM) project will research an adaptable and scalable robot teleoperation system that allows factory workers to work remotely. The research will benefit both the manufacturing industry and the workforce by increasing access to manufacturing employment and improving working conditions and safety. By combining human-in-the-loop design with machine learning, this research can broaden the adoption of automation in manufacturing to new tasks. Beyond manufacturing, the research will also lower the entry barrier to using robotic systems for a wide range of real-world applications, such as assistive and service robots. The research team is collaborating with NYDesigns and LaGuardia Community College to translate research results to industrial partners and develop training programs to educate and prepare the future manufacturing workforce.This research suggests three key ideas to enable human-in-the-loop assembly: First, the system uses a physical scene understanding algorithm that converts the real-world robot workspace into a virtual manipulable three-dimensional scene representation. Next, a three-dimensional Virtual Reality user interface will be used to allow users to specify high-level task goals using this scene representation. Finally, the system uses a goal-driven reinforcement learning algorithm to infer an effective planning policy, given the task goals and the robot configuration. This system can overcome several limitations of existing teleoperation systems. By separating high-level task planning from low-level robot control using a physical scene representation, the system allows the operator to specify task goals without having expert knowledge of the robot hardware and configuration. By using reinforcement learning for low-level control, the system is more generalizable to new tasks and hardware.This award is co-funded by the Divisions of Civil Mechanical and Manufacturing Innovation, Electrical, Communications and Cyber Systems, Computer and Network Systems, Undergraduate Education, and Behavioral and Cognitive Sciences and the Cyber Physical Systems, NSF Scholarships in Science, Technology, Engineering, and Mathematics, and Advanced Technological Education Programs.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

COVID-19 大流行加速了许多行业远程工作的采用。员工通常在家远程工作的能力对于组织的长期弹性和增长潜力至关重要。然而，虽然软件和网络的进步使信息工作者可以远程工作，但大多数制造业工人却不能，因为所需的基础设施不存在。这个未来制造（FM）项目将研究一种适应性强且可扩展的机器人远程操作系统，该系统允许工厂工人远程工作。该研究将增加制造业就业机会并改善工作条件和安全，使制造业和劳动力受益。通过将人机交互设计与机器学习相结合，这项研究可以将制造领域自动化的应用范围扩大到新的任务。除了制造业之外，该研究还将降低将机器人系统用于各种现实世界应用（例如辅助机器人和服务机器人）的准入门槛。研究团队正在与 NYDesigns 和拉瓜迪亚社区学院合作，将研究成果转化为工业合作伙伴，并制定培训计划来教育和准备未来的制造业劳动力。这项研究提出了实现人机循环装配的三个关键想法：首先，该系统使用物理场景理解算法，将现实世界的机器人工作空间转换为虚拟可操作的三维场景表示。接下来，将使用三维虚拟现实用户界面来允许用户使用此场景表示来指定高级任务目标。最后，系统使用目标驱动的强化学习算法，在给定任务目标和机器人配置的情况下推断出有效的规划策略。该系统可以克服现有远程操作系统的一些限制。通过使用物理场景表示将高级任务规划与低级机器人控制分开，该系统允许操作员指定任务目标，而无需具备机器人硬件和配置的专业知识。通过使用强化学习进行低级控制，该系统更适用于新任务和硬件。该奖项由土木机械和制造创新、电气、通信和网络系统、计算机和网络系统、本科生等部门共同资助教育、行为和认知科学以及网络物理系统、NSF 科学、技术、工程和数学奖学金以及先进技术教育项目。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准。

项目成果

AdaGrasp: Learning an Adaptive Gripper-Aware Grasping Policy

AdaGrasp：学习自适应抓取器感知抓取策略

• DOI: 10.1109/icra48506.2021.9560833
• 发表时间: 2021-05
• 期刊: 2021 IEEE International Conference on Robotics and Automation (ICRA
• 作者: Xu, Zhenjia;Qi, Beichun;Agrawal, Shubham;Song, Shuran
• 通讯作者: Song, Shuran

Precueing Object Placement and Orientation for Manual Tasks in Augmented Reality

在增强现实中预先确定手动任务的对象放置和方向

• DOI: 10.1109/tvcg.2022.3203111
• 发表时间: 2022-11
• 期刊: IEEE Transactions on Visualization and Computer Graphics
• 影响因子: 5.2
• 作者: Liu, Jen;Tversky, Barbara;Feiner, Steven
• 通讯作者: Feiner, Steven

Rearrangement Planning for General Part Assembly

总装重排规划

• 发表时间: 2023-07-01
• 作者: Yulong Li;Andy Zeng;Shuran Song
• 通讯作者: Shuran Song

Multi-Level Precues for Guiding Tasks Within and Between Workspaces in Spatial Augmented Reality

用于在空间增强现实中工作空间内和工作空间之间指导任务的多级提示

• DOI: 10.1109/tvcg.2023.3320246
• 发表时间: 2023-11
• 期刊: IEEE Transactions on Visualization and Computer Graphics
• 影响因子: 5.2
• 作者: Volmer, Benjamin;Liu, Jen;Matthews, Brandon;Bornkessel;Feiner, Steven;Thomas, Bruce H.
• 通讯作者: Thomas, Bruce H.

Adaptive Visual Cues for Guiding a Bimanual Unordered Task in Virtual Reality

用于指导虚拟现实中双手无序任务的自适应视觉提示

• DOI: 10.1109/ismar55827.2022.00059
• 发表时间: 2022-10
• 期刊: 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR
• 作者: Liu, Jen;Wang, Portia;Tversky, Barbara;Feiner, Steven
• 通讯作者: Feiner, Steven