Collaborative Research: HCC: Medium: Aligning Robot Representations with Humans

合作研究：HCC：媒介：使机器人表示与人类保持一致

• 批准号: 2310759
• 负责人: Daniel Brown
• 金额: $ 30.55万
• 依托单位: University of Utah
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-08-15 至 2026-07-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2310759&HistoricalAwards=false
• 关键词: Collaborative; Research; HCC; Medium; Aligning

This project seeks to make robots more robust and aligned with human preferences and values. Traditionally, robot behaviors and objectives were trained to include a set of hand-crafted features (i.e., variables represented in the data) that reflect task-relevant aspects of the environment. Using well-chosen features is very data-efficient, but it is unrealistic for human engineers to identify and write code ahead of time for all the features that could matter. Training modern high-capacity models from a lot of data is a great alternative, as long as we do not probe the learned models on novel (out-of-distribution) inputs. The reason these models fail to generalize to out-of-distribution inputs is that they will generally fail to learn the correct representation, comprising the features that matter, and instead pick up on spurious patterns in the data. The central goal of this project is to enable robots to arrive at the underlying correct representation for objectives (and, hence, behaviors). And since learning the objective function---what the human user wants---is fundamentally about humans, this work proposes that only the human can determine what actually matters vs. what is spurious. The research will introduce the problem of aligning robot representations to humans. The key observation behind the project is that traditional input used in learning, such as demonstrations or comparisons, which is designed to teach the robot the full task, is not ideal for aligning the robot’s representation. With representation alignment defined as a problem, there is the opportunity to design new types of human feedback that help the robot explicitly isolate the right representation. The project will develop new types of human feedback and algorithms for efficiently learning from them to arrive at an aligned representation. Preliminary work leveraged this observation to introduce feature traces---a novel type of human input through which users can teach the robot about specific features they care about. The project will pursue four objectives that together tackle the aspects of aligning robot representations with humans: (1) Teaching one feature at a time, beyond feature traces: It will investigate new input types for aligning robot representations with users, contribute active learning algorithms that help the human teacher provide the most informative input, and build transparency tools that enable robots to teach back to the user their current understanding of the representation. (2) Extracting features all at once from new, representation-specific human input: It will investigate new human input types that teach the full representation all at once by combining self-supervised representation learning methods with human-centric representation learning. (3) Using a correct representation in the right way: Given a new task, the robot needs to learn which features matter and in which contexts. (4) Extending earlier work to policy learning: It will extend new tools to the policy learning setting and use the lens of human-aligned representations to enable better policy generalization to new users and to improve goal mis-generalization in reinforcement learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目旨在使机器人更加强大并符合人类的偏好和价值观。传统上，机器人的行为和目标经过训练，包括一组反映任务相关方面的手工特征（即数据中表示的变量）。使用精心选择的特征非常具有数据效率，但对于人类工程师来说，提前识别所有可能重要的特征并编写代码是不现实的，从大量数据中训练现代高容量模型是一件好事。替代方案，只要因为我们没有在新的（分布外）输入上探索学习模型，这些模型无法推广到分布外输入的原因是它们通常无法学习正确的表示，包括重要的特征。 ，而不是发现数据中的虚假模式，该项目的中心目标是使机器人能够获得目标（以及今后的行为）的基本正确表示。人类用户想要的——从根本上讲是关于人类的，这项工作提出只有人类才能确定什么是真正重要的，什么是虚假的。该研究将引入将机器人表示与人类对齐的问题。该项目背后的关键观察是学习中使用的传统输入，例如演示或比较。旨在教导机器人完成完整的任务，对于对齐机器人的表示来说并不理想，由于表示对齐被定义为一个问题，因此有机会设计新类型的人类反馈来帮助机器人明确地隔离正确的表示。将开发新型人类反馈和算法为了有效地向他们学习以获得一致的表示，初步工作利用了这种观察来引入特征痕迹——一种新型的人类输入，用户可以通过这种方式向机器人传授他们关心的特定特征。该项目将追求四个目标。共同解决将机器人表示与人类对齐的各个方面：（1）一次教授一个特征，超越特征轨迹：它将研究用于将机器人表示与用户对齐的新输入类型，贡献主动学习算法，帮助人类教师提供最丰富的信息输入，并构建透明度工具，使(2)从新的、特定于表示的人类输入中一次性提取特征：它将研究新的人类输入类型，通过结合自学习一次性教授完整的表示。 （3）以正确的方式使用正确的表示学习方法：给定一个新任务，机器人需要学习哪些特征重要以及在哪些上下文中。（4）将早期的工作扩展到策略学习。 ：它将把新工具扩展到政策学习环境和使用人性化表示的视角，为新用户提供更好的政策概括，并改善强化学习中的目标错误概括。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响进行评估，被认为值得支持审查标准。