Collaborative Research: AF: Small: Parallel Reinforcement Learning with Communication and Adaptivity Constraints

协作研究：AF：小型：具有通信和适应性约束的并行强化学习

基本信息

批准号：
2006591
负责人：
Qin Zhang
金额：
$ 24.22万
依托单位：
Indiana University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2020
资助国家：
美国
起止时间：
2020-10-01 至 2023-09-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2006591&HistoricalAwards=false
关键词：
Collaborative Research AF Small Parallel

项目摘要

Reinforcement learning has witnessed great research advancement in recent years and achieved successes in many practical applications. However, reinforcement-learning algorithms also have the reputation for being data- and computation-hungry for large-scale applications. This project will address this issue by studying the important question of how to make reinforcement-learning algorithms scalable via introducing multiple learning agents and allowing them to collect data and learn optimal strategies collaboratively. The outcomes of this project will have impacts on numerous areas where reinforcement learning is used at a scale, e.g., multi-phase clinical trials, training autonomous-driving algorithms, crowdsourcing tasks, pricing, and assortment optimization for stores at different locations. The research products will be disseminated via talks at academic conferences and workshops, universities, industrial labs, and online media, and will also be integrated in two courses on the forefront of reinforcement learning and big-data algorithms.More technically, this project will study how to address the fundamental constraints on communication and adaptivity for the learning agents. In particular, this project will investigate a handful of collaborative learning models, including full communication, synchronized communication, synchronized communication with limited adaptivity, and asynchronized communication, and study the following general questions: (1) what is the fundamental advantage of allowing adaptivity in the parallel learning model; (2) are there inherent differences on the degree of parallelism between model-based and model-free reinforcement learning; (3) what is the impact of asynchronized communication; and (4) is it possible to communication-efficiently parallelize general algorithmic techniques in reinforcement learning? The team of researchers will address these questions by studying a set of core problems, including best arm(s) identification and regret minimization in multi-armed bandits, contextual bandits, finite-state Markov decision process (MDP) learning, reinforcement learning with function approximates, and coordinated exploration in MDPs. Through studying these questions, this project will bring new techniques, perspectives, and insight to communication-efficient parallel reinforcement learning. This project will also have a significant impact on a number of related research areas such as control theory, operations research, information theory and communication complexity, and multi-agent systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

强化学习近年来取得了巨大的研究进展，并在许多实际应用中取得了成功。然而，强化学习算法也因大规模应用程序需要大量数据和计算而闻名。该项目将通过研究如何通过引入多个学习代理并允许它们收集数据并协作学习最佳策略来使强化学习算法可扩展的重要问题来解决这个问题。该项目的成果将对大规模使用强化学习的众多领域产生影响，例如多阶段临床试验、训练自动驾驶算法、众包任务、定价和不同地点商店的分类优化。研究成果将通过学术会议和研讨会、大学、工业实验室和在线媒体的演讲进行传播，并将整合到强化学习和大数据算法前沿的两门课程中。从技术上讲，该项目将研究如何解决学习代理的沟通和适应性的基本限制。特别是，本项目将研究一些协作学习模型，包括完全通信、同步通信、有限自适应同步通信和异步通信，并研究以下一般性问题：（1）允许自适应的根本优势是什么？并行学习模型；（2）基于模型和无模型的强化学习在并行度上是否存在固有的差异；（3）异步通信有什么影响； (4) 是否可以在强化学习中有效地并行化通用算法技术？研究人员团队将通过研究一系列核心问题来解决这些问题，包括多臂老虎机中的最佳手臂识别和遗憾最小化、上下文老虎机、有限状态马尔可夫决策过程（MDP）学习、函数强化学习MDP 中的近似和协调探索。通过研究这些问题，该项目将为高效沟通的并行强化学习带来新的技术、观点和见解。该项目还将对控制论、运筹学、信息论与通信复杂性、多智能体系统等多个相关研究领域产生重大影响。该奖项体现了NSF的法定使命，经评审认为值得支持利用基金会的智力优势和更广泛的影响审查标准。