Exploring Causality in Reinforcement Learning for Robust Decision Making

探索强化学习中的因果关系以实现稳健决策

• 批准号: EP/Y003187/1
• 负责人: Yali Du
• 金额: $ 20.97万
• 依托单位: King's College London
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2023
• 资助国家: 英国
• 起止时间: 2023 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FY003187%2F1
• 关键词: Exploring; Causality; Reinforcement; Learning; Robust

Reinforcement learning (RL) has seen significant development in recent years and has demonstrated impressive capabilities in decision-making tasks, such as games (AlphaStar, OpenAI Five), chatbots (ChatGPT, and recommendation systems (Microsoft). The techniques of RL can also be applied to many fields, such as transportation, network communications, autonomous driving, sequential treatment in healthcare, robotics, and control. Unlike traditional supervised learning, RL focuses on making a sequence of decisions to achieve a long-term goal. This makes it particularly well-suited for solving complex problems. However, while RL has the potential to be highly effective, there are challenges that need to be addressed in order to make it more practical for real-world applications, where changing factors cannot be fully considered in training the agent, such as traffic regulations, weather, and clouds. To empower RL algorithms to be deployed in a range of real applications, we need to evaluate and improve the robustness of RL when facing complex changes in the real world and task shifts.In this project, we aim to develop robust and generalisable reinforcement learning techniques from a causal modelling perspective. The first thrust focuses on utilising causal model learning to create compact and robust representations of tasks. This compact and robust task representation can greatly benefit the overall performance of the RL agent by reducing the complexity of the problem and making the agent's decision-making process more efficient. As a result, the agent can learn faster and generalise better to unseen tasks, which is especially important in real-world scenarios where data is scarce and the complexity of tasks can vary greatly.The second research thrust focuses on the development of efficient and generalisable algorithms for task assignment transfer. This can enable the RL agent to adapt to new tasks more quickly and effectively and to generalise the learned knowledge to different but related tasks. This is crucial for real-world scenarios where the agent needs to operate in different environments or the task requirements change over time.One example of an application that would benefit from these contributions is autonomous driving in an industrial setting. While RL agents are usually trained in simulators, they may not perform well in real-world road scenarios and can be easily distracted by task-irrelevant information. For example, visual images that autonomous cars observe contain predominantly task-irrelevant information, like cloud shapes and architectural details, which should not influence the decision on driving.In this project, we aim to enable the agent to learn a compact and robust representation of the task, enabling it to only retain state information that is relevant to the task, adapt to changing driving scenarios safely, and generalise its knowledge to related tasks such as adapting to the different driving rules in the United States (right-hand drive).A causal understanding can help identify the minimal sufficient representations that are essential for policy learning and transferring and achieve safe and controllable explorations by leveraging causal structures and counterfactual reasoning.It can mitigate the issues that are suffered by most existing RL approaches, such as being data-hungry and lacking interpretability and generalisability.The outcome of this project can greatly improve the scalability and adaptability of RL agents, making them more suitable for real-world applications.

强化学习（RL）近年来取得了长足的发展，并在决策任务中表现出了令人印象深刻的能力，例如游戏（AlphaStar、OpenAI Five）、聊天机器人（ChatGPT 和推荐系统（Microsoft）。RL 技术还可以与传统的监督学习不同，强化学习专注于做出一系列决策以实现长期目标，例如交通、网络通信、自动驾驶、医疗保健、机器人和控制等。特别然而，虽然强化学习具有高效的潜力，但为了使其更实用于现实世界的应用，在训练中无法充分考虑不断变化的因素，还需要解决一些挑战。为了使 RL 算法能够部署在一系列实际应用中，我们需要评估和提高 RL 在面对现实世界的复杂变化和任务转移时的鲁棒性。这个项目，我们的目标是开发强大且通用的从因果建模的角度来看强化学习技术。第一个重点是利用因果模型学习来创建紧凑且稳健的任务表示。这种紧凑而鲁棒的任务表示可以降低问题的复杂性并使代理的决策过程更加高效，从而极大地提高 RL 代理的整体性能。因此，智能体可以更快地学习并更好地泛化到未见过的任务，这在数据稀缺且任务复杂性可能差异很大的现实场景中尤其重要。第二个研究重点是开发高效且可泛化的智能体任务分配转移算法。这可以使强化学习代理更快、更有效地适应新任务，并将学到的知识推广到不同但相关的任务。这对于代理需要在不同环境中运行或任务要求随时间变化的现实场景至关重要。从这些贡献中受益的应用程序的一个例子是工业环境中的自动驾驶。虽然强化学习智能体通常在模拟器中接受训练，但它们在现实世界的道路场景中可能表现不佳，并且很容易被与任务无关的信息分散注意力。例如，自动驾驶汽车观察到的视觉图像主要包含与任务无关的信息，例如云的形状和建筑细节，这些信息不应影响驾驶决策。在这个项目中，我们的目标是使代理能够学习紧凑而鲁棒的表示使其能够仅保留与任务相关的状态信息，安全地适应不断变化的驾驶场景，并将其知识推广到相关任务，例如适应美国不同的驾驶规则（右侧驾驶）。因果理解可以帮助确定最小的充分条件表示对于政策学习和迁移至关重要，并通过利用因果结构和反事实推理实现安全可控的探索。它可以缓解大多数现有强化学习方法所面临的问题，例如数据匮乏、缺乏可解释性和普遍性。该项目的成果可以极大地提高强化学习代理的可扩展性和适应性，使它们更适合实际应用。