Distributional Reinforcement Learning in the Brain

大脑中的分布式强化学习

• 批准号: 10709775
• 负责人: Jan Drugowitsch
• 金额: $ 58.12万
• 依托单位: HARVARD UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-04-15 至 2025-05-31
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10709775
• 关键词: Distributional; Reinforcement; Learning; Brain

Project Summary The field of artificial intelligence (AI) has recently made remarkable advances that resulted in new and improved algorithms and network architectures that proved efficient empirically in silico. These advances raise new questions in neurobiology: are these new algorithms used in the brain? The present study focuses on a new algorithm developed in the field of reinforcement learning (RL), called distributional RL, which outperforms other state-of-the-art RL algorithms and is regarded as a major advancement in RL. In environments in which rewards are probabilistic with respect to its occurrence and size, traditional RL algorithms have focused on learning to predict a single quantity, the average over all potential rewards. Distributional RL, by contrast, learns to predict the entire distribution over rewards (or values) by employing multiple value predictors that together encode all possible levels of future reward concurrently. Remarkably, theoretical work has shown that a class of distributional RL, called ‘quantile distributional RL’, can arise out of a simple modification of traditional RL that introduces structured variability in dopamine reward prediction error (RPE) signals. This project set out to test the hypothesis that the brain utilizes distributional RL to predict future rewards. Aim 1 will explore the characteristics of distributional RL theoretically and make predictions that allow for testing distributional RL in the brain. Theoretical investigations and simulations will be used to determine how value representations in distributional RL differ from pre-existing population coding schemes for representing probability distributions (probabilistic population codes, distributed distributional codes, etc.). Aim 2 will examine the activity of neurons that are thought to signal RPEs and reward expectation and test various predictions of distributional RL. Specifically, the activity of dopamine neurons in the ventral tegmental area and neurons in the ventral striatum and orbitofrontal cortex will be compared to key predictions of distributional RL. Aim 3 will use optogenetic manipulation to causally demonstrate the relationship between RPE signals and distributional codes.

项目概要 人工智能（AI）领域最近取得了显着的进步，产生了新的和 改进的算法和网络架构在计算机模拟中被证明是有效的。 进步引发了神经生物学的新问题：这些新算法是否在大脑中使用？ 目前的研究重点是强化学习（RL）领域开发的一种新算法，称为 分布式强化学习，其性能优于其他最先进的强化学习算法，被认为是一种主要的强化学习算法 在奖励的发生具有概率性的环境中，强化学习的进步。 和大小，传统的强化学习算法专注于学习预测单个数量，即平均值 相比之下，分布式强化学习学习预测整个分布。 通过使用多个价值预测器来编码所有可能的级别的奖励（或价值） 值得注意的是，理论工作表明一类分布式强化学习， 称为“分位数分布强化学习”，可以通过对传统强化学习的简单修改而产生： 引入了多巴胺奖励预测误差（RPE）信号的结构变异性。 该项目旨在测试大脑利用分布式强化学习来预测未来的假设 目标 1 将从理论上探讨分布式强化学习的特征并做出预测。 允许测试大脑中的分布式强化学习。 用于确定分布强化学习中的值表示与预先存在的群体有何不同 用于表示概率分布的编码方案（概率总体代码、分布式 分布代码等）。目标 2 将检查被认为向 RPE 发出信号的神经元的活动。 奖励期望并测试分布式 RL 的各种预测，具体来说，是 的活动。 腹侧被盖区的多巴胺神经元以及腹侧纹状体和眶额的神经元 皮层将与分布式 RL 的关键预测进行比较，Aim 3 将使用光遗传学。 操纵因果关系来证明 RPE 信号和分布代码之间的关系。