Determining the neural substrates of model-based and model-free reinforcement-learning during Pavlovian conditioning

确定巴甫洛夫条件反射期间基于模型和无模型强化学习的神经基础

基本信息

批准号：
10117323
负责人：
JOHN P O'DOHERTY
金额：
$ 4.31万
依托单位：
CALIFORNIA INSTITUTE OF TECHNOLOGY
依托单位国家：
美国
项目类别：
财政年份：
2016
资助国家：
美国
起止时间：
2016-07-01 至 2022-03-31
项目状态：
已结题

来源：
https://reporter.nih.gov/project-details/10117323
关键词：
Address Amygdaloid structure Animals Area Behavior Brain Brain region Cell Nucleus Choice Behavior Classification Code Complement Computer Analysis Conditioned Reflex Conditioned Stimulus Cues Data Desire for food Dorsal Environment Evaluation Functional Magnetic Resonance Imaging Goals Homologous Gene Human Incentives Individual Knowledge Learning Link Literature Maintenance Methodology Midbrain structure Modeling Multivariate Analysis Nature Neurobiology Neurons Neurosciences Operant Conditioning Outcome Patients Pattern Pharmaceutical Preparations Play Process Psychological reinforcement Psychology Recording of previous events Research Resolution Rewards Role Signal Transduction Stimulus Structure Techniques Testing Ventral Striatum Ventral Tegmental Area addiction base classical conditioning flexibility imaging study insight interest neurophysiology neurosurgery novel operation psychologic public health relevance reinforcer relating to nervous system response

项目摘要

DESCRIPTION (provided by applicant): The study of the psychological, computational and neurobiological basis of Pavlovian conditioning is one of the longest standing research questions in psychology and neuroscience. In spite of the ubiquity and the importance of this form of learning, the computational mechanisms underlying the learning and expression of Pavlovian associations' remains poorly understood. Here, we investigate whether or not there exists two distinct forms of Pavlovian conditioning, a model-based form in which the expression of conditioned responses to a conditioned stimulus is sensitive to the incentive value of the associated unconditioned stimulus (US), and another "model-free" form in which conditioned responses elicited by a conditioned stimulus are insensitive to the current US value. The distinction between model-based and model-free reinforcement-learning mechanisms has received strong empirical support in the domain of instrumental conditioning, but little is known about whether or not a similar dichotomy exists in Pavlovian conditioning. Understanding the nature of the encoding of Pavlovian associations in the brain is important because of the critical role that learned Pavlovian associations might play in the maintenance of addiction, in which cues linked to drug outcomes might promote or invigorate responding for drugs, even if those drugs are no longer deemed valuable/desirable to the individual. In the present application we address this goal by performing both functional magnetic resonance imaging (fMRI) and single-unit recordings in humans while they undergo sequential Pavlovian conditioning with appetitive outcomes. We will use a number of different cutting-edge experimental and analytical techniques, including computational based analyses, multivariate pattern classification and high-resolution fMRI. We will test for the existence of these different representations in a number of distinct structures in the brain including the amygdala, orbitofrontal cortex, ventral striatum and dopaminergic midbrain. Because we will be using high- resolution fMRI, we will have the capacity to resolve the contribution of distinct sub-regions within these brain structures to model-based and model-free Pavlovian learning, including the basolateral versus centromedial amygdala, the human homologue of the core versus shell of the accumbens, different sectors of orbitofrontal cortex, and dorsal versus ventral parts of the substantial nigra and ventral tegmental area. To complement the fMRI studies, we will record from neurons primarily in the amygdala and orbitofrontal cortex in human neurosurgery patients while they while they perform one of the main tasks used in the fMRI studies, thereby enabling us to gain insight into the relationship between the observed fMRI signals and underlying neuronal activity in at least two of our key regions of interest. By combining across these different techniques and methodologies, we will be able to address the question of whether or not model-based and model-free forms of Pavlovian conditioning are implemented in parallel in the brain, and begin to gain insight into the specific contributions of different brain regions towards these two very distinct forms of learning.

描述（由申请人提供）：巴甫洛夫条件反射的心理学、计算和神经生物学基础的研究是心理学和神经科学中历史最悠久的研究问题之一，尽管这种学习形式普遍存在且很重要，但其计算机制。巴甫洛夫关联的学习和表达的背后仍然知之甚少，在这里，我们研究是否存在两种不同形式的巴甫洛夫条件作用，一种基于模型的形式，其中表达对条件刺激的条件反应。对相关无条件刺激（US）的激励值敏感，而另一种“无模型”形式，其中条件刺激引起的条件反应对当前 US 值不敏感。基于模型和无模型之间的区别。强化学习机制在工具性条件反射领域得到了强有力的实证支持，但人们对巴甫洛夫条件反射中是否存在类似的二分法知之甚少，因为巴甫洛夫关联在大脑中的编码性质非常重要。学到的角色巴甫洛夫关联可能在成瘾的维持中发挥作用，其中与药物结果相关的线索可能会促进或激发对药物的反应，即使这些药物不再被认为对个体有价值/可取。在本申请中，我们通过执行来实现这一目标。我们将使用许多不同的尖端实验和分析技术，包括基于计算的分析、多变量模式分类和分析。我们将测试大脑中许多不同结构中这些不同表征的存在，包括杏仁核、眶额皮质、腹侧纹状体和因为我们将使用高分辨率功能磁共振成像，我们将有能力解决这些大脑结构中不同子区域对基于模型和无模型巴甫洛夫学习的贡献，包括基底外侧杏仁核与中心内侧杏仁核。人类伏隔核与壳、眶额皮质不同部分、黑质和腹侧被盖区的背侧与腹侧部分的同源物。为了补充功能磁共振成像研究，我们将记录人类神经外科患者在执行功能磁共振成像研究中使用的主要任务之一时主要位于杏仁核和眶额皮质的神经元，从而使我们能够深入了解观察到的通过结合这些不同的技术和方法，我们将能够解决基于模型和无模型形式的巴甫洛夫条件反射是否有效的问题。在大脑中并行实施，并开始深入了解不同大脑区域对这两种截然不同的学习形式的具体贡献。