RI: Small: Coordination in tightly coupled domains: Stepping stone rewards to induce the correct joint actions

RI：小：紧密耦合领域中的协调：垫脚石奖励以诱导正确的联合行动

基本信息

批准号：
1815886
负责人：
Kagan Tumer
金额：
$ 40万
依托单位：
Oregon State University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2018
资助国家：
美国
起止时间：
2018-09-01 至 2023-08-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1815886&HistoricalAwards=false
关键词：
RI Small Coordination tightly coupled

项目摘要

This project introduces a new multiagent learning approach that leads to coordinated behavior in tightly coupled domains, that is, in domains where all agents must do the right thing at the right time for the team to achieve its goals. For example, getting a team of agents to lift and move an object heavier than the payload capacity of any single agent requires a sufficient number of agents to perform the correct action at the correct time. Unfortunately, most current learning methods fail in such situations because they rely on reinforcing the correct agent behavior only after the agents stumble upon the right actions. But what if the agents never jointly find the right actions? This project addresses this issue by introducing "stepping-stone rewards" that incentivize agents to perform the right actions even if their teammates have not yet found the correct complementary actions. The impact of this project will be to create larger and more capable multiagent teams that can be deployed in industry (such as factory robots that are not limited to a single task), in the field (such as autonomous search and rescue systems), in education (such as interactive learning via online gameplay) and in the home (such as networks of smart appliances).The main technical contribution of this project is to shift the learning problem faced by an agent from "did I take the correct action?" to "would my action have been correct had other agents taken the complementary action?" In tightly coupled multiagent domains, the first question results in very little positive feedback, creating a difficult to impossible learning problem. The new stepping stone rewards leverage hypothetical partners (partners that are surmised by an agent to explore the joint-action space) to overcome this difficulty by assessing the potential benefits of a particular action. Intuitively, stepping-stone rewards create a gradient for the agents to follow to enable fast and efficient learning in tightly coupled domains.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目介绍了一种新的多种学习方法，该方法导致紧密耦合的域中的协调行为，也就是说，在所有代理商必须在正确的时间做正确的事情才能实现其目标的域中。例如，让一组代理团队提起和移动比任何单个代理的有效载荷能力更重的对象需要足够数量的代理在正确的时间执行正确的操作。不幸的是，当前大多数学习方法在这种情况下失败了，因为它们仅在代理人偶然发现正确的动作之后才能加强正确的代理行为。但是，如果代理商从未共同找到正确的行动怎么办？该项目通过引入“垫脚石奖励”来解决这个问题，即使他们的队友尚未找到正确的互补行动，也激励代理人执行正确的行动。该项目的影响将是创建可以在行业中部署的更大，更有能力的多种团队（例如不限于单个任务的工厂机器人），在教育中（例如自主搜索和救援系统），在教育中（例如通过在线游戏玩法进行交互式学习）以及在家中（例如，智能设备的网络）对“对智能设备的贡献”进行了“对我的贡献”， “如果其他特工采取了互补行动，我的行动是否正确？”在紧密耦合的多基因域中，第一个问题导致很少的积极反馈，从而造成了难以实现的学习问题。新的垫脚石奖励通过评估特定行动的潜在好处来克服这一困难，利用了假设伙伴（宣传代理商来探索联合行动空间的伙伴）。从直觉上讲，垫脚石的奖励为代理商效仿创造了一个梯度，以便在紧密耦合的领域中快速有效地学习。该奖项反映了NSF的法定任务，并认为值得通过基金会的智力优点和更广泛的影响审查标准通过评估来获得支持。