Collaborative Research: AF: Small: Parallel Reinforcement Learning with Communication and Adaptivity Constraints

协作研究：AF：小型：具有通信和适应性约束的并行强化学习

基本信息

批准号：
2006591
负责人：
Qin Zhang
金额：
$ 24.22万
依托单位：
Indiana University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2020
资助国家：
美国
起止时间：
2020-10-01 至 2023-09-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2006591&HistoricalAwards=false
关键词：
Collaborative Research AF Small Parallel

项目摘要

Reinforcement learning has witnessed great research advancement in recent years and achieved successes in many practical applications. However, reinforcement-learning algorithms also have the reputation for being data- and computation-hungry for large-scale applications. This project will address this issue by studying the important question of how to make reinforcement-learning algorithms scalable via introducing multiple learning agents and allowing them to collect data and learn optimal strategies collaboratively. The outcomes of this project will have impacts on numerous areas where reinforcement learning is used at a scale, e.g., multi-phase clinical trials, training autonomous-driving algorithms, crowdsourcing tasks, pricing, and assortment optimization for stores at different locations. The research products will be disseminated via talks at academic conferences and workshops, universities, industrial labs, and online media, and will also be integrated in two courses on the forefront of reinforcement learning and big-data algorithms.More technically, this project will study how to address the fundamental constraints on communication and adaptivity for the learning agents. In particular, this project will investigate a handful of collaborative learning models, including full communication, synchronized communication, synchronized communication with limited adaptivity, and asynchronized communication, and study the following general questions: (1) what is the fundamental advantage of allowing adaptivity in the parallel learning model; (2) are there inherent differences on the degree of parallelism between model-based and model-free reinforcement learning; (3) what is the impact of asynchronized communication; and (4) is it possible to communication-efficiently parallelize general algorithmic techniques in reinforcement learning? The team of researchers will address these questions by studying a set of core problems, including best arm(s) identification and regret minimization in multi-armed bandits, contextual bandits, finite-state Markov decision process (MDP) learning, reinforcement learning with function approximates, and coordinated exploration in MDPs. Through studying these questions, this project will bring new techniques, perspectives, and insight to communication-efficient parallel reinforcement learning. This project will also have a significant impact on a number of related research areas such as control theory, operations research, information theory and communication complexity, and multi-agent systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

近年来，强化学习在许多实际应用中取得了成功，并取得了成功。但是，加强学习算法也因大规模应用程序的数据和计算而享有声誉。该项目将通过研究如何通过介绍多个学习代理并允许它们收集数据并协作学习最佳策略来扩展的强化学习算法的重要问题来解决这个问题。该项目的结果将对许多领域有影响，例如，多相临床试验，培训自动驾驶算法，众包任务，定价和分类优化的商店。研究产品将通过在学术会议和讲习班，大学，工业实验室和在线媒体上进行会谈来分散，并且还将集成在强化学习和大数据算法的最前沿的两门课程中。在技术上，该项目将研究如何针对学习和适应学习型的基本约束。特别是，该项目将研究一些协作学习模型，包括完整的沟通，同步的沟通，有限的适应性和异步交流，并研究以下一般问题：（1）在平行学习模型中允许适应性的基本优势是什么；（2）在基于模型和无模型的增强学习之间的并行程度上存在固有的差异；（3）异步通信的影响是什么？（4）是否有可能在增强学习中有效地使一般算法技术平行？研究人员团队将通过研究一组核心问题来解决这些问题，包括最佳武器识别和遗憾的多臂匪徒，上下文匪徒，有限状态马尔可夫决策过程（MDP）学习，具有功能近似值的增强学习以及MDP中的协调探索。通过研究这些问题，该项目将为沟通和平行的增强学习带来新的技术，观点和洞察力。该项目还将对许多相关研究领域（例如控制理论，运营研究，信息理论和沟通复杂性以及多代理系统）产生重大影响。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的智力优点和更广泛影响的评估来通过评估来支持的。