RI: Small: Coordinating Multi-Agent Learning through Emergent Distributed Supervisory Control

RI：小型：通过紧急分布式监督控制协调多智能体学习

• 批准号: 1116078
• 负责人: Victor Lesser
• 金额: $ 45万
• 依托单位: University of Massachusetts Amherst
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2011
• 资助国家: 美国
• 起止时间: 2011-09-01 至 2014-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1116078&HistoricalAwards=false
• 关键词: RI; Small; Coordinating; Multi; Agent

The project is focused on developing coordination policies for large-scale multi-agent systems operating in uncertain environments through the use of multi-agent reinforcement learning (MARL). Existing MARL techniques do not scale well. This research addresses the scaling issue by using coordination technology to "coordinate" the individual agent learning so as to speed up convergence and lead to learned policies that better reflect overall system objectives. This novel idea is being implemented using an emergent supervisory organization with low overhead that exploits non-local information to dynamically coordinate and shape the learning processes of individual agents while still allowing agents to react autonomously to local feedback. A key question is how to automate the development of the supervisory control process (including supervisory information generation and organization formation). One approach to automation is using a formal model of interactions among agents that also includes a model of global system objectives and policy space of agents to derive the information necessary for appropriate supervisory control. Another approach is the formulation of the supervision problem as a distributed constraint optimization problem. The results of this work provide a necessary component for the development of a wide variety of next-generation adaptive applications, such as smart power grids, cloud computing, and large-scale sensor networks. The broader impact stems from the wide applicability of the resulting learning technology for distributed control, undergraduate and graduate educational activities at UMass, dissemination efforts that make the experimental domain and algorithms publically available, and the development of international collaborations.

该项目的重点是通过使用多智能体强化学习（MARL）为在不确定环境中运行的大规模多智能体系统制定协调策略。现有的 MARL 技术不能很好地扩展。这项研究通过使用协调技术来“协调”个体代理的学习来解决扩展问题，从而加速收敛并产生更好地反映整体系统目标的学习策略。这一新颖的想法正在使用一个低开销的新兴监督组织来实现，该组织利用非本地信息来动态协调和塑造各个代理的学习过程，同时仍然允许代理对本地反馈做出自主反应。一个关键问题是如何自动化开发监管控制流程（包括监管信息生成和组织形成）。自动化的一种方法是使用代理之间交互的正式模型，该模型还包括代理的全局系统目标和策略空间模型，以获取适当监督控制所需的信息。另一种方法是将监督问题表述为分布式约束优化问题。这项工作的结果为开发各种下一代自适应应用程序（例如智能电网、云计算和大规模传感器网络）提供了必要的组件。更广泛的影响源于由此产生的学习技术对分布式控制的广泛适用性、麻省大学本科生和研究生的教育活动、公开实验领域和算法的传播工作以及国际合作的发展。