Sparse representations for reinforcement learning

强化学习的稀疏表示

• 批准号: RGPIN-2018-05721
• 负责人: White, Martha
• 金额: $ 2.84万
• 依托单位: University of Alberta
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2019
• 资助国家: 加拿大
• 起止时间: 2019-01-01 至 2020-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=687437
• 关键词: Sparse; representations; reinforcement; learning

A key component of an artificial intelligence system is the ability to process and learn from a high-dimensional, high-volume sensory stream of information. For example, an agent controlling the pumps in an industrial plant continually receives sensory information about temperatures and energy consumption, to continually adjust the motor speed in real-time to optimize performance. To make such decision, the agents needs to be able to predict the long-term outcomes of their behaviour. For example, if the industrial agent can predict the long-term temperature of the motor, given the current state of the system, they can use these predictions to improve their decisions and ensure motors are not damaged. ******Such predictions, however, can be difficult to learn accurately from raw sensory information. Predictions are typically learned as functions of inputted sensory information. For example, the prediction of motor temperature in five minutes could be approximated as a polynomial function of the last ten recorded temperature and motor speeds. Polynomials, however, are only one possible functional form, and not necessarily the best one. Further, to obtain general learning agents, the functional forms should be effective across multiple settings or tasks. This is the goal of representation learning in reinforcement learning: identifying a general mapping from a sequence of raw sensory information to a set of features, that facilitates accurate predictions. ******The goal in my research is to understandboth theoretically and empiricallythe properties of effective representations for a reinforcement learning agent learning on a continual stream of sensory information. A part of this challenge is to identify simpler representations for which we can provide optimization guarantees, but that are nonetheless sufficiently powerful to facilitate learning. Continuing preliminary research, I will explore prototype-based (kernel) representations and a sparse supervised auto-encoder representation. We have already found that, within this class of simpler representations, we can find computational models that provide highly accurate predictions, but are more amenable to theoretical analysis. A core component of this research direction will be to investigate sparsity as a generally useful property of representations, and how we can encode that property into our representation learning algorithms. ******If successful, this research will have important scientific and societal benefits. This research will contribute to a core endeavour in artificial intelligence: understanding how to develop intelligent agents that can learn in complex environments. This understanding, in turn, will contribute to improving the robustness of automated decision-making systems, which are becoming ubiquitous in our world, including in industrial systems and factories, in self-driving vehicles and even in our homes.

人工智能系统的一个关键组成部分是处理高维、大容量感官信息流并从中学习的能力。例如，控制工厂中泵的代理不断接收有关温度和能耗的传感信息，以不断实时调整电机速度以优化性能。为了做出这样的决定，代理人需要能够预测其行为的长期结果。例如，如果工业代理可以在给定系统当前状态的情况下预测电机的长期温度，他们就可以使用这些预测来改进决策并确保电机不会损坏。 ******然而，这样的预测可能很难从原始的感官信息中准确地学习。预测通常是作为输入的感官信息的函数来学习的。例如，五分钟内电机温度的预测可以近似为最后十次记录的温度和电机速度的多项式函数。然而，多项式只是一种可能的函数形式，而且不一定是最好的形式。此外，为了获得通用学习代理，功能形式应该在多种设置或任务中有效。这是强化学习中表示学习的目标：识别从一系列原始感官信息到一组特征的一般映射，以促进准确的预测。 ******我研究的目标是从理论上和经验上理解强化学习代理在连续的感官信息流上学习的有效表示的属性。这一挑战的一部分是确定更简单的表示，我们可以为其提供优化保证，但仍然足够强大以促进学习。继续进行初步研究，我将探索基于原型（内核）的表示和稀疏监督自动编码器表示。我们已经发现，在这类更简单的表示中，我们可以找到提供高度准确预测的计算模型，但更适合理论分析。该研究方向的核心组成部分是研究稀疏性作为表示的普遍有用的属性，以及我们如何将该属性编码到我们的表示学习算法中。 ******如果成功，这项研究将产生重要的科学和社会效益。这项研究将有助于人工智能的核心努力：了解如何开发可以在复杂环境中学习的智能代理。这种理解反过来将有助于提高自动化决策系统的稳健性，自动化决策系统在我们的世界中变得无处不在，包括工业系统和工厂、自动驾驶汽车甚至我们的家庭。