Distributional value coding and reinforcement learning in the brain

大脑中的分布值编码和强化学习

• 批准号: 10539251
• 负责人: Adam Stanley Lowet
• 金额: $ 3.45万
• 依托单位: HARVARD MEDICAL SCHOOL
• 依托单位国家: 美国
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-08-01 至 2024-07-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10539251
• 关键词: Affect; Algorithms; Anatomy; Animals; Anxiety; Architecture; Behavior; Behavioral; Bernoulli Distribution; Bipolar Disorder; Brain; Brain region; Calcium; Cells; Code; Collaborations; Color; Complex; Corpus striatum structure; Dopamine; Dopamine D2 Receptor; Environment; Event; Future; Gambling; Grain; Heterogeneity; Image; Individual; Laboratories; Lead; Learning; Logic; Machine Learning; Measures; Mental Depression; Mental disorders; Modification; Molecular; Mus; Nature; Neurons; Odors; Outcome; Pattern; Performance; Population; Population Theory; Predictive Value; Probability; Psychological reinforcement; RL13; Resolution; Rewards; Role; Scheme; Shapes; Silicon; Stimulus; Structure; System; Tail; Testing; Ventral Striatum; Ventral Tegmental Area; Water; Work; addiction; burden of illness; cell type; classical conditioning; density; design; dopamine system; dopaminergic neuron; drug of abuse; improved; in vivo; in vivo two-photon imaging; maladaptive behavior; neural circuit; neuromechanism; optimism; receptor; relating to nervous system; theories; two photon microscopy; two-photon

ABSTRACT Making predictions about future rewards in the environment, and taking actions to obtain those rewards, is critical for survival. When these predictions are overly optimistic — for example, in the case of gambling addiction — or overly pessimistic — as in anxiety and depression — maladaptive behavior can result and present a significant disease burden. A fundamental challenge for making reward predictions is that the world is inherently stochastic, and events on the tails of a distribution need not reflect the average. Therefore, it may be useful to predict not only the mean, but also the complete probability distribution of upcoming rewards. Indeed, recent advances in machine learning have demonstrated that making this shift from the average reward to the complete reward distribution can dramatically improve performance in complex task domains. Despite its apparent complexity, such “distributional reinforcement learning” can be achieved computationally with a remarkably simple and biologically plausible learning rule. A recent study found that the structure of dopamine neuron activity may be consistent with distributional reinforcement learning, but it is unknown whether additional neuronal circuity is involved — most notably the ventral striatum (VS) and orbitofrontal cortex (OFC), both of which receive dopamine input and are thought to represent anticipated reward, also called “value”. Here, we propose to investigate whether value coding in these downstream regions is consistent with distributional reinforcement learning. In particular, we will record from these brain regions while mice perform classical conditioning with odors and water rewards. In the first task, we will hold the mean reward constant while changing the reward variance or higher- order moments, and ask whether neurons in the VS and OFC represent information over and above the mean, consistent with distributional reinforcement learning. In principle, this should enable us to decode the complete reward distribution purely from neural activity. In the second task, we will present mice with a panel of odors predicting the same reward amount with differing probabilities. The simplicity of these Bernoulli distributions will allow us to compare longstanding theories of population coding in the brain — that is, how probability distributions can be instantiated in neural activity to guide behavior. In addition to high-density silicon probe recordings, we will perform two-photon calcium imaging in these tasks to assess whether genetically and molecularly distinct subpopulations of neurons in the striatum contribute differentially to distributional reinforcement learning. Finally, we will combine these recordings with simultaneous imaging of dopamine dynamics in the striatum to ask how dopamine affects striatal activity in vivo. Together, these studies will help clarify dopamine’s role in learning distributions of reward, as well as its dysregulation in addiction, anxiety, depression, and bipolar disorder.

抽象的 对环境中未来的奖励进行预测并采取行动以获得这些奖励至关重要 当这些预测过于乐观时——例如，在赌博成瘾的情况下——或者 过度悲观——如焦虑和抑郁——可能会导致适应不良行为，并带来严重的后果。 做出奖励预测的一个基本挑战是世界本质上是随机的， 并且分布尾部的事件不需要反映平均值，因此，预测“不”可能会很有用。 事实上，不仅是平均值，而且是即将到来的奖励的完整概率分布。 机器学习已经证明，从平均奖励到完全奖励的转变 尽管分布看起来很复杂，但它可以显着提高复杂任务领域的性能。 这种“分布式强化学习”可以通过令人惊讶的简单和计算来实现 最近的一项研究发现，多巴胺神经元活动的结构可能是生物学上合理的学习规则。 与分布式强化学习一致，但尚不清楚是否需要额外的神经电路 涉及——最显着的是腹侧纹状体（VS）和眶额皮质（OFC），两者都接收多巴胺 输入 和 被认为代表预期奖励，也称为“价值”。在这里，我们建议进行调查。 这些下游区域的值编码是否与分布式强化学习一致。 特别是，当小鼠用气味和水进行经典调节时，我们将从这些大脑区域进行记录 在第一个任务中，我们将保持平均奖励不变，同时改变奖励方差或更高的值。 对矩进行排序，并询问 VS 和 OFC 中的神经元是否代表超出平均值的信息， 原则上，这与分布式强化学习一致，这应该使我们能够解码完整的内容。 纯粹根据神经活动进行奖励分配在第二个任务中，我们将向小鼠展示一组气味。 以不同的概率预测相同的奖励金额将是伯努利分布的简单性。 让我们能够比较大脑中长期存在的群体编码理论——即概率分布如何 除了高密度硅探针记录之外，我们还可以在神经活动中实例化来指导行为。 将在这些任务中进行双光子钙成像，以评估遗传和分子是否不同 纹状体中的神经元亚群对分布式强化学习的贡献不同。 我们将把这些记录与纹状体中多巴胺动态的同步成像相结合，以询问如何 多巴胺影响体内纹状体活动，这些研究将有助于阐明多巴胺在学习中的作用。 奖励的分配，以及其在成瘾、焦虑、抑郁和双相情感障碍中的失调。