Exploring Causality in Reinforcement Learning for Robust Decision Making

探索强化学习中的因果关系以实现稳健决策

• 批准号: EP/Y003187/1
• 负责人: Yali Du
• 金额: $ 20.97万
• 依托单位: King's College London
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2023
• 资助国家: 英国
• 起止时间: 2023 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FY003187%2F1
• 关键词: Exploring; Causality; Reinforcement; Learning; Robust; Exploring; Causality; Reinforcement; Learning; Robust

Reinforcement learning (RL) has seen significant development in recent years and has demonstrated impressive capabilities in decision-making tasks, such as games (AlphaStar, OpenAI Five), chatbots (ChatGPT, and recommendation systems (Microsoft). The techniques of RL can also be applied to many fields, such as transportation, network communications, autonomous driving, sequential treatment in healthcare, robotics, and control. Unlike traditional supervised learning, RL focuses on making a sequence of decisions to achieve a long-term goal. This makes it particularly well-suited for solving complex problems. However, while RL has the potential to be highly effective, there are challenges that need to be addressed in order to make it more practical for real-world applications, where changing factors cannot be fully considered in training the agent, such as traffic regulations, weather, and clouds. To empower RL algorithms to be deployed in a range of real applications, we need to evaluate and improve the robustness of RL when facing complex changes in the real world and task shifts.In this project, we aim to develop robust and generalisable reinforcement learning techniques from a causal modelling perspective. The first thrust focuses on utilising causal model learning to create compact and robust representations of tasks. This compact and robust task representation can greatly benefit the overall performance of the RL agent by reducing the complexity of the problem and making the agent's decision-making process more efficient. As a result, the agent can learn faster and generalise better to unseen tasks, which is especially important in real-world scenarios where data is scarce and the complexity of tasks can vary greatly.The second research thrust focuses on the development of efficient and generalisable algorithms for task assignment transfer. This can enable the RL agent to adapt to new tasks more quickly and effectively and to generalise the learned knowledge to different but related tasks. This is crucial for real-world scenarios where the agent needs to operate in different environments or the task requirements change over time.One example of an application that would benefit from these contributions is autonomous driving in an industrial setting. While RL agents are usually trained in simulators, they may not perform well in real-world road scenarios and can be easily distracted by task-irrelevant information. For example, visual images that autonomous cars observe contain predominantly task-irrelevant information, like cloud shapes and architectural details, which should not influence the decision on driving.In this project, we aim to enable the agent to learn a compact and robust representation of the task, enabling it to only retain state information that is relevant to the task, adapt to changing driving scenarios safely, and generalise its knowledge to related tasks such as adapting to the different driving rules in the United States (right-hand drive).A causal understanding can help identify the minimal sufficient representations that are essential for policy learning and transferring and achieve safe and controllable explorations by leveraging causal structures and counterfactual reasoning.It can mitigate the issues that are suffered by most existing RL approaches, such as being data-hungry and lacking interpretability and generalisability.The outcome of this project can greatly improve the scalability and adaptability of RL agents, making them more suitable for real-world applications.

Reinforcement learning (RL) has seen significant development in recent years and has demonstrated impressive capabilities in decision-making tasks, such as games (AlphaStar, OpenAI Five), chatbots (ChatGPT, and recommendation systems (Microsoft). The techniques of RL can also be applied to many fields, such as transportation, network communications, autonomous driving, sequential treatment in healthcare, robotics, and control. Unlike traditional supervised learning, RL专注于做出长期目标的序列。在面对现实世界和任务变化的复杂变化时，需要评估和改善RL的鲁棒性。在这个项目中，我们旨在从因果建模的角度来开发可靠和可推广的强化学习技术。第一个推力重点是利用因果模型学习来创建任务的紧凑而健壮的表示。这种紧凑而健壮的任务表示可以通过降低问题的复杂性并使代理商的决策过程更加有效，从而极大地使RL代理的整体绩效受益。结果，代理可以更快地学习和推广到看不见的任务，这在数据稀缺的现实情况下尤为重要，并且任务的复杂性可能会有很大变化。第二项研究推力侧重于开发有效且可普遍的任务分配转移算法。这可以使RL代理更快，有效地适应新任务，并将学习的知识推广到不同但相关的任务。这对于代理商需要在不同环境或任务要求随时间变化的现实情况下至关重要。从这些贡献中受益的应用程序的一个示例是在工业环境中自动驾驶。尽管RL代理通常是在模拟器中训练的，但在现实世界中的情况下可能无法表现良好，并且很容易被任务信息的信息分散注意力。例如，自动驾驶观察的视觉图像主要包含任务 - 无关紧要的信息，例如云形状和架构细节，不应影响驾驶的决定。在该项目中，我们的目标是使代理人能够紧凑而稳健地代表任务的代表，以学习与该任务相关的信息，并使其与任务相关，并适用于驾驶的信息，并适用于该任务，并将其与任务相关，并确定该任务的信息，并确定该任务的信息，并确定该任务的信息，并确定该任务的信息，并确定该任务的信息，并将其与任务相关，并将其与任务相关，并且可以使该任务相关，并将其与任务相关，并且可以使该任务相关，并且可以使该任务相关，并且可以使该任务相关，并将其与该任务相关，并将其与该任务相关，并将其与该任务相关，并将其与该任务相关。美国的驾驶规则（右手）。一种因果理解可以帮助确定最低的足够代表性，这些表示对政策学习和转移至关重要，并通过利用因果结构和反事实推理来实现安全可控制的探索，它可以减轻大多数现有的RL方法，例如延伸性和不足的能力，例如这一范围的能力，例如这一范围内的能力，例如这一范围的能力。 RL代理，使其更适合实际应用。