CRCNS: Reward and motivation in neural networks

CRCNS：神经网络中的奖励和动机

• 批准号: 9916069
• 负责人: ALEXEI KOULAKOV
• 金额: $ 43.2万
• 依托单位: COLD SPRING HARBOR LABORATORY
• 依托单位国家: 美国
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-30 至 2024-07-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9916069
• 关键词: Adaptive Behaviors; Animals; Behavior; Behavioral; Clinical; Complex; Computer Analysis; Computer Simulation; Computing Methodologies; Data; Drug Addiction; Electrophysiology (science); Functional disorder; Globus Pallidus; Glutamates; Goals; Human; Hunger; Image; Impairment; Individual; Instruction; Knowledge; Learning; Lesion; Link; Maintenance; Mathematics; Mental Depression; Methods; Molecular Genetics; Motivation; Mus; Neural Network Simulation; Neurons; Nutrient; Outcome; Pharmacology; Physiological; Play; Population; Principal Investigator; Psychological reinforcement; Punishment; Recurrence; Research; Rewards; Role; Signal Transduction; Structure; Testing; Thirst; Training; addiction; base; behavior test; computational neuroscience; expectation; flexibility; improved; in vivo; incentive salience; learning network; mental state; motivated behavior; motivational processes; neural network; neuromechanism; novel; optogenetics; programs; response; theories; tool

The overall goal of this project is to develop a reinforcement learning (RL) theory of motivation, understood here as motivational salience, and to test the conclusions of this theory using experimental observations obtained in the ventral pallidum (VP). Animals' actions depend on the shifting values of internal demands determined by physiological or behavioral conditions, such as thirst, hunger, addiction, specific nutrient deficiency, etc. These need-based modulations of the perceived values of reinforcements (reward or punishment} are described by a mathematical variable called motivational salience or, simply, motivation. Including motivation adds a new level of complexity to RL theory, and allows it to generate flexible ongoing behaviors. Here, we will investigate how motivation can be learned by neuronal networks to generate complex adaptive behaviors and compare the conclusions of our theory with the VP circuits. Previous studies indicate that the VP plays an important role in a variety of behaviors, potentially, by influencing motivational salience. In vivo recordings suggest that VP neuron firing correlates with motivational states. Lesions, pharmacological and optogenetic manipulations in VP cause profound changes in behaviors motivated by natural rewards or drugs of addiction. Dysfunction of this structure is linked to depression and drug addiction in humans. Our theoretical results suggest that distinct classes of neurons in the VP should play essential roles in representing either positive or negative motivational states. We further hypothesize that the functional interactions locally within the VP are critical for generating such signals that guide motivated behaviors. Consistent with predictions of RL theory, in our preliminary studies, we found that individual VP neurons could be classified as either positive or negative 'motivation neurons', as the activities of these neurons represented both expected values of outcomes and motivational states. When population activity is considered, representations of outcome expectation can be distinguished from representations of motivation fluctuating according to the animals' physiological states. Based on the preliminary data, we devised an integrated approach, combining studies in computational analysis and theory (Koulakov lab) with advanced molecular genetic tools, optogenetics, chemogenetics, electrophysiology, and imaging in behaving mice (Li lab), to test our hypotheses through the following Aims: Aim 1. To develop methods for identifying motivation in the population activity of VP neurons. Here we will use novel behavioral and computational methods to disambiguate representations of motivation and outcome expectation in neuronal responses. Aim 2. To develop reinforcement learning theory of motivation and to test its predictions using responses of VP neurons. Here we will develop the Q-learning theory of motivation and compare networks trained using this theory to responses of VP neurons. Aim 3. To identify the circuit basis of representations of motivation in VP neuronal populations. We will identify the network structure in Q-learning networks with motivation, and test predictions using opto- and chemogenetic manipulations in VP. RELEVANCE (See instructions): The neural mechanisms of motivated behaviors remain unclear. In the proposed research program, we will determine the precise circuit mechanisms and computations by which neurons in the ventral pallidum participate in modulating motivated behaviors. Findings from this project will have important clinical implications, as impairments in motivational processes are core features of depression and drug addiction.

该项目的总体目标是开发强化学习（RL）动机理论，理解 这里作为动机显着性，并使用实验观察来检验该理论的结论 在腹侧苍白球（VP）中获得。动物的行为取决于内部需求的不断变化的价值观 由生理或行为条件决定，例如口渴、饥饿、成瘾、特定营养素 这些基于需求的对强化感知价值的调节（奖励或 惩罚}由称为动机显着性或简单地动机的数学变量来描述。 包含动机将 RL 理论的复杂性提高到了一个新的水平，并使其能够产生灵活的持续性 行为。在这里，我们将研究神经元网络如何学习动机以产生 复杂的自适应行为并将我们的理论结论与 VP 电路进行比较。之前的研究 表明副总裁可能通过影响动机在各种行为中发挥重要作用 显着性。体内记录表明 VP 神经元放电与动机状态相关。病变， VP 的药理学和光遗传学操作会引起行为的深刻变化 自然奖励或成瘾药物。该结构的功能障碍与抑郁症和毒瘾有关 在人类中。我们的理论结果表明，VP 中不同类别的神经元应该发挥重要作用 代表积极或消极动机状态的角色。我们进一步假设函数 副总裁内部的本地互动对于产生指导动机行为的信号至关重要。 与 RL 理论的预测一致，在我们的初步研究中，我们发现单个 VP 神经元 可以分为积极或消极的“动机神经元”，因为这些神经元的活动 代表结果的预期值和动机状态。当人口活动量为 考虑到，结果期望的表征可以与动机的表征区分开来 随动物的生理状态而变化。根据初步数据，我们设计了一个 综合方法，将计算分析和理论（Koulakov 实验室）的研究与先进的相结合 行为小鼠的分子遗传工具、光遗传学、化学遗传学、电生理学和成像（Li 实验室），通过以下目标检验我们的假设： 目标 1. 开发识别动机的方法 VP 神经元的群体活动。在这里，我们将使用新颖的行为和计算方法来 消除神经元反应中动机和结果期望的歧义。目标 2. 至 发展强化学习动机理论并使用 VP 神经元的反应来测试其预测。 在这里，我们将发展动机的 Q 学习理论，并将使用该理论训练的网络与 VP 神经元的反应。目标 3. 确定 VP 神经元动机表征的回路基础 人口。我们将通过动机识别 Q 学习网络中的网络结构，并测试预测 在 VP 中使用光遗传学和化学遗传学操作。 相关性（参见说明）： 动机行为的神经机制仍不清楚。在拟议的研究计划中，我们将 确定腹侧苍白球神经元的精确电路机制和计算 参与调节动机行为。该项目的研究结果将具有重要的临床意义 影响，因为动机过程的损害是抑郁症和毒瘾的核心特征。