The role of distributional reinforcement learning in human neurons during impulsive choices

分布式强化学习在人类神经元冲动选择过程中的作用

• 批准号: 10335061
• 负责人: Elliot H Smith
• 金额: $ 50.98万
• 依托单位: UNIVERSITY OF UTAH
• 依托单位国家: 美国
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-02-03 至 2026-12-31
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10335061
• 关键词: Address; Affect; Amygdaloid structure; Animals; Anterior; Area; Artificial Intelligence; Basic Science; Behavior; Behavior Disorders; Behavioral; Brain; Brain Stem; Categories; Choice Behavior; Code; Color; Complex; Corpus striatum structure; Data; Decision Making; Development; Dopamine; Dopamine Receptor; Epilepsy; Exhibits; Feedback; Future; Goals; Hippocampus (Brain); Human; Impulse Control Disorders; Impulsive Behavior; Impulsivity; Intractable Epilepsy; Knowledge; Learning; Link; Mathematics; Measures; Medical; Mental Health; Mental disorders; Modeling; Monitor; Mus; Neurons; Neurosciences; Organism; Outcome; Patients; Performance; Population; Probability; Psychiatric therapeutic procedure; Psychological reinforcement; Research; Reversal Learning; Rewards; Risk; Role; Signal Transduction; Substance Use Disorder; Temporal Lobe; Testing; Time; Translating; Update; Work; analog; base; cingulate cortex; dopaminergic neuron; expectation; experience; experimental study; human subject; improved; learning outcome; neuromechanism; neuropsychiatric disorder; neuropsychiatry; novel; optimism; relating to nervous system; response

ABSTRACT Recent developments in artificial intelligence and neuroscience have revealed neural codes for reinforcement that represent predictions of a range of possible future reward outcomes, rather than a singular expected value. This distributional reinforcement learning has enabled improved performance of artificial agents and has straightforward implications for numerous neuropsychiatric disorders, particularly impulse control and substance use disorders. This proposal aims to leverage our experience recording neuronal activity from the brains of human neurosurgical patients in order to translate these recordings in a novel research direction: to understand the mechanisms of human choice behavior. We will determine where distributional codes exist in the human prefrontal and mesial temporal cortices, and how those codes are expressed dynamically in time as humans make impulsive choices during the Balloon Analog Risk Task (BART) and a probabilistic reversal learning task. The results of these experiments will have both important basic scientific implications and will begin to address how distributional reinforcement learning in the human brain contributes to impulsive choices. In order to begin translating this new area of knowledge to understand the underpinnings of human decisions, we will first establish the presence of distributional reinforcement learning in four brain areas that comprise a human decision-making circuit: Orbitofrontal Cortex, Anterior Cingulate Cortex, Amygdala, and Hippocampus. Specific Aim 1 will test the three essential predictions of distributional RL: whether populations of neurons in each of these brain areas exhibit 1) asymmetric scaling of reward prediction errors, 2) diverse reversal points, and 3) that prediction error asymmetries and reversal points correlate across neurons. Specific Aim 2 seeks to decode BART reward prediction distributions from neurons in the aforementioned brain areas and determine how changes in BART reward distributions correlate with the propensity to make impulsive choices. Specific Aim 3 will test how diversity in optimism and pessimism in each neuron recorded from the aforementioned brain areas correlates with valuation or devaluation across trials. The completion of these aims will constitute important basic research findings in discovering distributional RL in the human prefrontal and mesial temporal cortices. By uncovering neural population codes that underlie potentially impulsive choices in human decision-making circuits, these experiments also address fundamental neural mechanisms underlying impulsive choices. This issue is central to addressing important problems for contemporary mental health including substance use disorder and a many other neuropsychiatric disorders. These findings will have readily translatable implications for improving targeted electrical therapies for psychiatric disorders.

抽象的 人工智能和神经科学方面的最新发展揭示了增强神经代码 这代表了一系列可能的未来奖励结果的预测，而不是一个单一的期望值。 这种分配加强学习使人造代理的性能提高了，并具有 对众多神经精神疾病的直接影响，尤其是冲动控制和物质 使用疾病。该建议旨在利用我们从大脑中记录神经元活动的经验 人类神经外科患者以将这些记录转化为新的研究方向：了解 人类选择行为的机制。我们将确定人类中的分布代码在哪里存在 前额叶和介体颞皮层，以及这些代码在人类中如何动态表达 在气球模拟风险任务（BART）和概率逆转学习任务中做出冲动的选择。 这些实验的结果将具有重要的基本科学意义，并将开始解决 人脑中的分布强化学习如何有助于冲动的选择。 为了开始翻译这个新的知识领域以了解人类决定的基础， 我们将首先在四个大脑区域中建立分配加固学习的存在，包括 人类决策回路：眶额皮质，前扣带回皮质，杏仁核和海马。 特定目标1将测试分布RL的三个基本预测：神经元的种群是否在 这些大脑区域中的每一个都表现出1）奖励预测错误的不对称缩放，2）不同的逆转点， 3）预测误差不对称和逆转点在神经元之间相关。特定目标2试图 解码来自上述大脑区域中神经元的BART奖励预测分布，并确定 巴特奖励分布的变化与做出冲动选择的倾向相关。具体的 AIM 3将测试从上述大脑记录的每个神经元中的乐观和悲观情绪的多样性 区域与试验之间的估值或贬值相关。 这些目标的完成将构成在发现分布RL中的重要基础研究结果 人类的前额叶和介体颞皮层。通过发现基于 这些实验在人类决策循环中的潜在冲动选择也涉及基本 脉冲选择的神经机制。这个问题是解决重要问题的核心 当代心理健康，包括药物使用障碍和许多其他神经精神疾病。 这些发现将很容易翻译对改善精神病的靶向电疗法的影响 疾病。