RI: Small: Towards Optimal and Adaptive Reinforcement Learning with Offline Data and Limited Adaptivity

RI：小型：利用离线数据和有限的适应性实现最优和自适应强化学习

• 批准号: 2007117
• 负责人: Yu-Xiang Wang
• 金额: $ 45万
• 依托单位: University of California-Santa Barbara
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2007117&HistoricalAwards=false
• 关键词: RI; Small; Towards; Optimal; Adaptive; RI; Small; Towards; Optimal; Adaptive

Reinforcement learning (RL) is one of the fastest-growing research areas in machine learning. RL-based techniques have led to several recent breakthroughs in artificial intelligence, such as beating human champions in the game of Go. The application of RL to real life problems, however, remains limited, even in areas where a large amount of data has already been collected. The crux of the problem is that most existing RL methods require an environment for the agent to interact with, but in real-life applications, it is rarely possible to have access to such an environment — deploying an algorithm that learns by trial-and-errors may have serious legal, ethical and safety issues. This project aims to address this conundrum by developing algorithms that learn from offline data. The outcome of the research could significantly reduce the overhead of using RL techniques in real-life sequential decision-making problems such as those in power transmission, personalized medicine, scientific discoveries, computer networking and public policy.The project focuses on two settings that aim at addressing the aforementioned challenge of limited access to an environment. In the first setting, the agent is given only the historical data from logged interactions with the environment. In the second setting, the agent is able to change how it interacts with the environment only a few times. The investigators will develop mathematical theory that describes the difficulty of the problem and ensures that the developed algorithms are robust and optimal in the sense that they use the least possible resources (data, energy, computation). Using techniques such as marginalized importance sampling, uniform convergence and batched exploration, the project will generalize the recent line of work in ``breaking the curse of horizon'' to allow function approximations and establish the much-needed statistical learning theory for offline and low-adaptive reinforcement learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

增强学习（RL）是机器学习中增长最快的研究领域之一。基于RL的技术导致了最近在人工智能方面取得的一些突破，例如在GO游戏中击败人类冠军。但是，即使在已经收集了大量数据的领域，RL在现实生活中的应用也仍然有限。问题的症结在于，大多数现有的RL方法都需要一个环境供代理商进行互动，但是在现实生活中，很少有可能访问这种环境 - 部署通过反复试验来学习的算法可能会有严重的法律，道德和安全问题。该项目旨在通过开发从离线数据学习的算法来解决这个难题。该研究的结果可能会大大减少在现实生活中使用RL技术的开销，例如在权力传播，个性化医学，科学发现，计算机网络和公共政策中使用RL技术。该项目侧重于解决有限访问环境的PRIORE的两个环境。在第一个设置中，仅赋予代理与与环境相互作用的历史数据。在第二个设置中，代理只能更改其与环境相互作用的方式几次。研究人员将使用诸如边缘化重要性取样，统一的融合和批处理探索等技术来发展数学理论，该项目将概括“打破地平线的曲线”中的最新工作，以允许函数近似值，以允许运行近似值，并建立了非常需要的统计学学习理论，以表现出良好的统计学和稳定的启发。使用基金会的智力优点和更广泛的影响评估标准进行评估。