RI: Medium: Learning Task-Specific Representations for Broadly Capable Reinforcement Learning Agents

RI：中：学习具有广泛能力的强化学习代理的特定任务表示

• 批准号: 1955361
• 负责人: George Konidaris
• 金额: $ 119.97万
• 依托单位: Brown University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1955361&HistoricalAwards=false
• 关键词: RI; Medium; Learning; Task; Specific

While artificially intelligent agents have achieved expert-level performance on some specialized tasks, progress on designing agents that are broadly capable---able to reach adequate performance on a wide range of tasks---remains elusive. One major obstacle is that the sensors and actuators required by a general-purpose agent must be very complex, to support all the different tasks it may be required to solve. The resulting complexity makes decision-making much harder and drastically hinders the effectiveness of such agents. By contrast, agents that do only one thing can be given much simpler inputs and outputs that are carefully designed to be low-dimensional, highly informative, and task-relevant; such agents often demonstrate satisfactory performance. This project posits that a key requirement for generally intelligent agents is the ability to autonomously formulate such representations for themselves---as abstactions over their complex sensor and actuator spaces---and plans to design new algorithms to do so. AI systems with this ability could be re-tasked to solve many different problems without modification, rather than requiring substantial (and often prohibitive) engineering effort for each new application.This project aims to develop new algorithms that enable agents to learn compact, task-specific abstractions of new problems, by combining and extending techniques for discovering high-level actions, discovering perceptual abstractions that support planning with high-level actions, and formally characterizing the complexity and value loss of using those abstractions. The project will: 1) design new algorithms for reward-driven (and therefore task-specific) perceptual- and action-abstraction discovery; 2) enable inter-task abstraction transfer (which avoids having to re-learn abstractions from scratch each time) through new algorithms for learning generalized skills and constructing modular action-perception-abstraction packages, and new theory characterizing the value loss of using such generalized abstractions; and 3) create principled methods for incrementally constructing a library of modular action-perception abstractions and for adaptively recruiting existing action-state abstractions to solve new tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

虽然人工智能代理已经在某些专门任务上实现了专家级的性能，但在设计具有广泛能力的代理（能够在广泛的任务上达到足够的性能）方面仍然难以取得进展。一个主要障碍是通用代理所需的传感器和执行器必须非常复杂，以支持它可能需要解决的所有不同任务。由此产生的复杂性使决策变得更加困难，并极大地阻碍了此类代理的有效性。相比之下，只做一件事的智能体可以得到更简单的输入和输出，这些输入和输出经过精心设计，具有低维度、高信息量和任务相关性；此类代理通常表现出令人满意的性能。该项目认为，一般智能代理的一个关键要求是能够为自己自主地制定此类表示（作为对其复杂传感器和执行器空间的抽象），并计划设计新的算法来实现这一点。具有这种能力的人工智能系统可以重新分配任务，无需修改即可解决许多不同的问题，而不是为每个新应用程序都需要大量（而且通常是令人望而却步的）工程工作。该项目旨在开发新的算法，使代理能够学习紧凑的任务新问题的具体抽象，通过组合和扩展用于发现高级操作的技术，发现支持高级操作规划的感知抽象，并正式描述使用这些抽象的复杂性和价值损失。该项目将：1）设计新的算法，用于奖励驱动（因此特定于任务）的感知和动作抽象发现； 2）通过学习通用技能和构建模块化动作-感知-抽象包的新算法，以及描述使用这种通用技能的价值损失的新理论，实现任务间抽象迁移（避免每次从头开始重新学习抽象）抽象； 3）创建原则性方法，逐步构建模块化动作感知抽象库，并自适应地招募现有的动作状态抽象来解决新任务。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力评估进行评估，认为值得支持。优点和更广泛的影响审查标准。

项目成果

Robustly Learning Composable Options in Deep Reinforcement Learning

• DOI: 10.24963/ijcai.2021/298
• 发表时间: 2021-08
• 作者: Akhil Bagaria;J. Senthil;Matthew Slivinski;G. Konidaris

Model-based Lifelong Reinforcement Learning with Bayesian Exploration

• DOI: 10.48550/arxiv.2210.11579
• 发表时间: 2022-10
• 期刊: ArXiv
• 作者: Haotian Fu;Shangqun Yu;Michael S. Littman;G. Konidaris

Skill Discovery for Exploration and Planning using Deep Skill Graphs

使用深度技能图进行探索和规划的技能发现

• 发表时间: 2021
• 期刊: Proceedings of the Thirty-Eighth International Conference on Machine Learning
• 作者: Bagaria, A;Senthil, J.;Konidaris, G.D.
• 通讯作者: Konidaris, G.D.

Coarse-Grained Smoothness for Reinforcement Learning in Metric Spaces

度量空间中强化学习的粗粒度平滑度

• 发表时间: 2023
• 期刊: Proceedings of the 26th International Conference on Artificial Intelligence and Statistics
• 作者: Gottesman, O;Asadi, K;Allen, C;Lobel, S;Konidaris, GD;Littman, ML
• 通讯作者: Littman, ML

Autonomous Learning of Object-Centric Abstractions for High-Level Planning

用于高层规划的以对象为中心的抽象的自主学习

• 发表时间: 2022
• 期刊: Proceedings of the The Tenth International Conference on Learning Representations
• 作者: James, S;Rosman, B;Konidaris, G.D.
• 通讯作者: Konidaris, G.D.