CNS Core: Small: Online Safe Reinforcement Learning for Wireless Resource Allocation

CNS 核心：小型：用于无线资源分配的在线安全强化学习

基本信息

批准号：
1910112
负责人：
Gustavo de Veciana
金额：
$ 49.96万
依托单位：
University of Texas at Austin
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2019
资助国家：
美国
起止时间：
2019-10-01 至 2023-09-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1910112&HistoricalAwards=false
关键词：
CNS Core Small Online Safe

项目摘要

Next generation wireless networks are being engineered to meet a complex mix of application requirements, from traditional mobile broadband (e.g., web browsing, video streaming) to new emerging applications (e.g., augmented reality, self-driving cars, industrial automation, robotics) with heterogeneous much more stringent reliability-latency requirements. The ability to support these requirements drives the potential to deliver the new business models and new revenue streams that would enable deployments of the new technology. Thus, wireless scheduling and resource allocation will take center stage in terms of enabling technologies for such networks. Meanwhile reinforcement learning (RL) using deep networks has emerged as a powerful framework to devise polices that optimize complex systems' performance (including wireless systems); however, these usually do not come with any formal guarantees. The central thesis of this research is that RL-based resource allocation policies without operational guarantees, e.g., throughput-optimality/stability, are unlikely to be accepted and/or deployed, thus a key requirement to make these techniques usable, is to develop approaches which ensure safety guarantees. The research advances the state-of-the-art in safe reinforcement learning, with specific applications to wireless systems, but also is expected to benefit other application domains as well as society more broadly, through planned efforts in education, innovation, achieving diversity, engaging the community and industry, and disseminating results to a wider public.This research centers on the development and analysis of a safe reinforcement learning (Safe-RL) framework, which optimizes rewards over short-time scales, and also provides theoretically strong long-term throughput-optimality guarantees for state-of-art wireless scheduling algorithms. The key underlying observation is that many of today's scheduling algorithms derive their performance guarantees from Lyapunov analysis. The project leverages the innovative concept of guardrails -- constraints on the state-dependent actions of Safe-RL -- that guarantee that the wireless system's Lyapunov evolution stay within a bounded perturbation of classical algorithms. This guarantee, in turn, ensures that Safe-RL has safety/stability properties of state-of-the-art schedulers, while leveraging RL to realize complex performance tradeoffs. The research consists of three inter-related thrusts. Thrust 1 develops the foundations and representations for safe-RL at the core of this project, along with a theoretical basis for safety guarantees and new classes of efficient learning for wireless system network abstractions. Thrust 2 focuses on the application of safe-RL theory to wireless resource allocation, including addressing challenges associated with joint scheduling of real-time and broadband traffic, learning and exploiting traffic patterns, and an exploration of the degree to which a policy hits guardrails as an indication of system anomalies or need for re-optimization. Thrust 3 centers on the challenging but necessary task of validating the safe-RL framework leveraging an industrial strength multi-cell simulator.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

正在设计下一代无线网络，以满足应用要求的复杂组合，从传统的移动宽带（例如，网络浏览，视频流）到新的新兴应用程序（例如，增强现实，自动驾驶汽车，工业自动化，机器人技术，机器人技术）具有异质性的更严格的可靠性延迟要求。支持这些需求的能力促进了提供新的业务模型和新收入流的潜力，从而可以部署新技术。因此，无线调度和资源分配将在为此类网络启用技术方面处于中心位置。同时，使用深网的加强学习（RL）已成为一个强大的框架，以设计优化复杂系统性能（包括无线系统）的政策；但是，这些通常没有任何正式的保证。这项研究的核心论点是，基于RL的资源分配政策没有运营保证，例如，吞吐量 - 最佳/稳定性不太可能被接受和/或部署，因此使这些技术可用的关键要求是开发确保安全保证的方法。 The research advances the state-of-the-art in safe reinforcement learning, with specific applications to wireless systems, but also is expected to benefit other application domains as well as society more broadly, through planned efforts in education, innovation, achieving diversity, engaging the community and industry, and disseminating results to a wider public.This research centers on the development and analysis of a safe reinforcement learning (Safe-RL) framework, which optimizes rewards over short-time scales, and also为最先进的无线调度算法提供理论上强大的长期吞吐量保证。关键的基本观察结果是，当今许多日程安排算法从Lyapunov分析中得出了其性能保证。该项目利用了护栏的创新概念 - 对安全RL的州依赖性行为的约束 - 保证无线系统的Lyapunov Evolution保持在经典算法的有限扰动中。反过来，这种保证可确保安全RL具有最先进的调度程序的安全/稳定性，同时利用RL实现复杂的性能权衡。该研究由三个相互关联的推力组成。推力1开发了该项目核心的Safe-RL的基础和表示，以及安全保证的理论基础和无线系统网络摘要的新的有效学习类别。推力2的重点是将安全RL理论应用于无线资源分配，包括解决与实时和宽带流量的联合计划相关的挑战，学习和利用交通模式，以及探索政策将护栏作为系统异常或需要重新启动的系统异常的程度。在验证验证安全RL框架的挑战但必要任务上的3个中心利用了工业实力多单元模拟器。该奖项反映了NSF的法定任务，并被认为是通过基金会的智力优点和更广泛影响的评估标准来评估的值得支持的。

项目成果

期刊论文数量（10）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Constrained Network Slicing Games: Achieving Service Guarantees and Network Efficiency

DOI：
10.1109/tnet.2023.3262810
发表时间：
2023-12
期刊：
IEEE/ACM Transactions on Networking
影响因子：
0
作者：
Jiaxiao Zheng;Albert Banchs;G. Veciana
通讯作者：
Jiaxiao Zheng;Albert Banchs;G. Veciana

MmWave Codebook Selection in Rapidly-Varying Channels via Multinomial Thompson Sampling

DOI：
10.1145/3466772.3467044
发表时间：
2021-07
期刊：
Proceedings of the Twenty-second International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing
影响因子：
0
作者：
Yi Zhang;S. Basu;S. Shakkottai;R. Heath
通讯作者：
Yi Zhang;S. Basu;S. Shakkottai;R. Heath

Distributed Reinforcement Learning Based Delay Sensitive Decentralized Resource Scheduling

DOI：
10.23919/wiopt58741.2023.10349880
发表时间：
2023-08
期刊：
2023 21st International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt)
影响因子：
0
作者：
Geetha Chandrasekaran;G. Veciana
通讯作者：
Geetha Chandrasekaran;G. Veciana

Meta-Scheduling for the Wireless Downlink Through Learning With Bandit Feedback

通过学习强盗反馈进行无线下行链路元调度