Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Efficient Reinforcement Learning

合作研究：CIF：媒介：高效强化学习的统计和算法基础

基本信息

批准号：
2106778
负责人：
Yuejie Chi
金额：
$ 80万
依托单位：
Carnegie-Mellon University
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2021
资助国家：
美国
起止时间：
2021-10-01 至 2025-09-30
项目状态：
未结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2106778&HistoricalAwards=false
关键词：
Collaborative Research CIF Medium Statistical

项目摘要

As a data-driven paradigm for sequential decision making in unknown environments, Reinforcement Learning (RL) has received significant interest in recent years owing to its potential ability to solve difficult problems associated with future societal and scientific developments. However, the explosion of both model dimensionality and complexity in current and emerging applications exacerbates the challenge of achieving efficient RL in sample-starved situations, where data collection is expensive, time-consuming, or even high-stake, e.g., in clinical trials, online advertising, and autonomous systems. As a result, understanding and improving the sample and computational efficiencies of RL algorithms, sometimes under additional resource and system-level constraints, are rightly understood as critical to the successful deployment of RL in the future. In this project the PIs are involving students at all levels with diverse backgrounds in Electrical and Computer Engineering, and in Statistics, are developing education modules on RL to enrich the curriculum, and are co-organizing workshops and outreach activities to enable the broader dissemination of the project outcomes.Despite decades-long research efforts, the statistical and computational underpinnings of RL are still far from being well understood, especially when it comes to finite-sample and finite-time issues which are of crucial operational value. This research project is bridging the theory-practice gap of modern algorithmic approaches to RL. It is doing so by (i) characterizing fundamental limits for the sample and computations complexities in various RL settings, (ii) by developing performance guarantees and uncertainty quantification schemes, and (iii) by designing new computationally efficient algorithms that are provably near-optimal in terms of sample complexity in both single-agent and multi-agent settings. The expected outcomes will enable the trustworthy adoption of RL algorithms in sample-starved environments. The complementary expertise of the research team is being leveraged to enrich the statistical and algorithmic foundations of RL through model-, policy-, and value-based approaches. New efficient algorithms that rely on function approximation schemes are being developed in order to address the curse of dimensionality; the resulting techniques are intended to lead to non-asymptotic analysis tools that deal with the complicated statistical dependencies present in RL. This rich research agenda is expected to foster multidisciplinary efforts at the intersection of high-dimensional statistics, non-convex optimization, control theory, information theory, and machine learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

作为在未知环境中进行连续决策的数据驱动范式，近年来，强化学习（RL）由于解决了与未来的社会和科学发展相关的困难问题，因此受到了浓厚的兴趣。但是，在当前和新兴应用中，模型维度和复杂性的爆炸式爆炸加剧了在样本饥饿的情况下达到有效RL的挑战，在样品中，数据收集昂贵，耗时，甚至高率，例如在临床试验中，临床试验，在线广告和自主系统。结果，理解和改善RL算法的样本和计算效率，有时在其他资源和系统级别的约束下，正确地理解为对未来RL成功部署至关重要。在该项目中，PI在电气和计算机工程领域的各个级别的学生都参与，并且在统计学中，正在开发RL的教育模块以丰富课程，并共同组织研讨会和外展活动，以使更广泛地传播有关课程的活动。项目成果。尽管长达数十年的研究工作，RL的统计和计算基础仍然远远没有得到充分理解，尤其是在有限样本和有限的时间问题上，这些问题具有至关重要的运营价值。该研究项目正在弥合RL的现代算法方法的理论实践差距。它是通过（i）表征样品和计算各种RL设置中的复杂性的基本限制，（ii）通过开发性能保证和不确定性量化方案，以及（iii）设计新的计算有效算法，这些算法几乎是优秀的在单位和多代理设置中的样本复杂性方面。预期的结果将使在样本饥饿的环境中值得信赖的RL算法采用。研究团队的互补专业知识正在利用通过模型，政策和基于价值的方法来丰富RL的统计和算法基础。为了解决维度的诅咒，正在开发依赖功能近似方案的新的有效算法。最终的技术旨在导致非吸收分析工具，以处理RL中存在的复杂统计依赖性。预计这一丰富的研究议程将在高维统计，非凸优化，控制理论，信息理论和机器学习的交集中促进多学科的努力。该奖项反映了NSF的法定任务，并被认为是值得通过评估的支持。基金会的智力优点和更广泛的影响审查标准。