Collaborative Research: RI: Medium: Bootstrapping natural feedback for reinforcement learning

合作研究：RI：中：引导强化学习的自然反馈

• 批准号: 2212310
• 负责人: Jacob Andreas
• 金额: $ 120万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2025-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2212310&HistoricalAwards=false
• 关键词: Collaborative; Research; RI; Medium; Bootstrapping

Many modern applications of artificial intelligence---from industrial automation to content recommendation---depend on machine learning algorithms that train automated agents to interact with their environments. But the two main approaches to interactive learning, reinforcement learning and imitation, require so much supervision or training time that it is prohibitively expensive to apply them to most real-world problems. Human learning does not suffer from this shortcoming, in large part because humans learn not from rewards or demonstrations, but instead from extended interaction with skilled teachers who use signals like gesture and language. This project will lay a foundation for research on interactive learning with rich feedback, from the perspective of individual agents, human--agent teams, and multi-agent populations. It will yield new capabilities for interactive training of automated agents, expanding both the effectiveness and accessibility of such techniques. Support for natural, interactive feedback will also improve the customizability of such systems, making on-the-fly adaptation or retraining accessible to users without significant computing power, data annotation resources or even programming ability.The project is organized into three broad research objectives. First, it will develop a formal framework for grounding feedback, using simple supervisory signals (provided during or after execution) to bootstrap learned interpretation of more complex feedback types. Second, it will develop algorithms for learning to solicit feedback. These algorithms will turn the one-way process of reinforcement learning into a two-way interaction, enabling agents to proactively query supervisors for information about the compositional and causal structure of the environment. Third, it will develop new mechanisms and techniques for providing feedback, via software tools that assist human supervisors in selecting or generating maximally informative feedback signals. Research under each of these objectives will be carried out in simulated environments, benchmarked using complex tasks spanning navigation, robot manipulation, and furniture assembly, and evaluated in terms of its benefits to sample efficiency, end-to-end development time, and usability.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人工智能的许多现代应用 - 从工业自动化到内容建议---取决于训练自动化代理以与环境互动的机器学习算法。但是，互动学习，强化学习和模仿的两种主要方法需要大量的监督或培训时间，以至于将它们应用于大多数现实世界中的问题非常昂贵。人类的学习不会遭受这种缺点的困扰，这在很大程度上是因为人类不是从奖励或示范中学习，而是与使用手势和语言之类的信号的熟练老师进行扩展互动。该项目将从各个代理人，人类 - 代理团队和多机构人口的角度来看，为互动学习的研究奠定基础。 它将产生新的功能，用于对自动化代理进行互动培训，从而扩大此类技术的有效性和可访问性。支持自然的交互式反馈还将改善此类系统的可定制性，在没有强大的计算能力，数据注释资源甚至编程能力的情况下，可以在线适应或重新训练。该项目被组织为三个广泛的研究目标。首先，它将使用简单的监督信号（在执行期间或之后提供）来开发一个正式的框架来接地反馈，以学习对更复杂的反馈类型的解释。其次，它将开发学习征集反馈的算法。这些算法将把增强学习的单向过程转变为双向相互作用，使代理能够主动查询主管，以获取有关环境组成和因果关系结构的信息。第三，它将通过软件工具开发新的机制和技术来提供反馈，以帮助人类主管选择或生成最大信息的反馈信号。这些目标下的研究将在模拟环境中进行，使用跨越导航，机器人操纵和家具组装的复杂任务进行基准测试，并根据其对样品效率的好处，端到端的开发时间和可用性进行评估。该奖项反映了NSF的法定任务，并通过评估了基金会的范围，并通过基金会的范围进行了评估和宽广的效果。

项目成果

Guiding Pretraining in Reinforcement Learning with Large Language Models

• DOI: 10.48550/arxiv.2302.06692
• 发表时间: 2023-02
• 作者: Yuqing Du;Olivia Watkins;Zihan Wang;Cédric Colas;Trevor Darrell;P. Abbeel;Abhishek Gupta;Jacob Andreas