Frontocortical Signaling Signatures in Flexible Reinforcement Learning

灵活强化学习中的额皮质信号特征

• 批准号: 10304186
• 负责人: HUGH T BLAIR
• 金额: $ 23.4万
• 依托单位: UNIVERSITY OF CALIFORNIA LOS ANGELES
• 依托单位国家: 美国
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-11-17 至 2023-03-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10304186
• 关键词: Animal Model; Animals; Anterior; Area; Behavior; Behavioral; Calcium; Chronic; Code; Coupling; Cues; Custom; Development; Discrimination; Discrimination Learning; Electrodes; Electrophysiology (science); Environment; Exploratory/Developmental Grant; Failure; Feedback; Image; Impairment; Lead; Learning; Light; Mediating; Modeling; Monitor; Outcome; Palate; Pattern; Performance; Phase; Photons; Predictive Value; Probability; Process; Psychological reinforcement; Psychopathology; Rattus; Reversal Learning; Rewards; Role; Sensory; Signal Transduction; Slice; Stimulus; Testing; Time; Uncertainty; Update; Work; adaptive learning; base; behavior change; cingulate cortex; design; designer receptors exclusively activated by designer drugs; expectation; experience; experimental study; flexibility; improved; in vivo; neuropsychiatric disorder; neuropsychiatry; neuroregulation; new technology; novel; novel strategies; relating to nervous system; time use

Various neuropsychiatric conditions lead to failures in generating accurate models of the reward environment or inabilities in using those models to guide flexible behavior, very often manifesting as impaired reversal learning. The anterior cingulate cortex (ACC) and the orbitofrontal cortex (OFC) are frontocortical regions important for flexible reinforcement learning, and have been theorized to work in a hierarchy of parallel processes for reward- based choice. In OFC, there is priority encoding of lower-level attributes like reward-predictive value of sensory cues, the palatability of specific rewards, and the current stimulus-reward mappings relevant to behavior. In ACC, these variables are thought to be multiplexed for higher-level computations of reward prediction error (RPE) and confidence/uncertainty of predictions, which are used to monitor performance and update behavioral strategies when necessary (particularly overall trial strategy following positive feedback, i.e., WinStay). These computations may depend upon propagation of spikes from OFC to ACC. However, it remains poorly understood how flexible reward learning is mediated by interactions between OFC and ACC. Here we will investigate this question using a robust animal model of adaptive learning under uncertainty: stimulus-based probabilistic reversal learning (PRL). In freely behaving rats, we will use a combination of in vivo 1-photon calcium imaging and electrophysiology, chemogenetics, and closed-loop neural control of reward delivery to examine how OFC and ACC regulate PRL. Using new technology that we have recently developed for online decoding of calcium activity we will use a novel strategy of regulating reward delivery based upon neural activity in ACC and OFC to test whether flexible reward learning depends upon accurate neural representations in these frontocortical areas. To date, we have: demonstrated effective DREADDs manipulation in vivo and in transduced cortical slices; designed and tested custom electrode arrays to perform chronic in vivo electrophysiological recordings in these areas simultaneously; and imaged ensemble activity time-locked to behavior, which has proven stable over multiple sessions, ideal to study learning. Leveraging these technical advances and using this capacity as a platform, we propose to identify the precise cortico-cortical mechanisms of encoding variables in flexible reinforcement learning across two Aims. Collectively, these experiments will: 1) shed new light on the signaling signatures of cortical regions and their respective roles in flexible reinforcement learning, 2) accelerate groundbreaking experiments as they would be performed in closed-loop: control of reversal learning in real-time using decoded neural expectation, and 3) these signals would eventually be compared in animal models of psychopathology because of their known failures in reversal learning. These novel and unconventional approaches make the R21 mechanism ideal for the proposed work.

各种神经精神疾病导致未能产生准确的奖励环境模型或 使用这些模型来指导灵活行为，通常表现为逆转学习受损。 前扣带回皮质（ACC）和眶额皮层（OFC）对额叶区域很重要 灵活的强化学习，并已被理论化在平行过程的层次结构中工作，以奖励 - 基于选择。在OFC中，有优先编码较低级别的属性，例如奖励预测的感觉 提示，特定奖励的可容纳性以及与行为相关的当前刺激奖励映射。在ACC中， 这些变量被认为是多重的，以用于奖励预测误差（RPE）和 预测的信心/不确定性，用于监视性能和更新行为策略 必要时（尤其是遵循积极反馈的总体试验策略，即赢家）。这些计算 可能取决于OFC到ACC的峰值传播。但是，它仍然不太了解灵活多么灵活 奖励学习是由OFC和ACC之间的相互作用介导的。在这里我们将调查这个问题 在不确定性下使用健壮的自适应学习动物模型：基于刺激的概率逆转 学习（PRL）。在自由行为的大鼠中，我们将结合体内1-光子钙成像和 电生理学，化学遗传学和对奖励交付的闭环神经控制，以研究OFC和OFC的方式 ACC调节PRL。使用我们最近开发的新技术用于在线解码钙活动 我们将使用一种新的策略来调节基于ACC和OFC中神经活动的奖励交付进行测试 灵活的奖励学习是否取决于这些额叶区域中准确的神经表示。到 日期，我们有：在体内和转导的皮质切片中表现出有效的恐怖操作；设计 并测试了自定义电极阵列以在这些区域执行慢性体内电生理记录 同时地;并成像合奏活动时间锁定为行为，这已证明在多个上稳定 会议，理想的学习学习。利用这些技术进步并将这种能力作为平台，我们 建议识别柔性增强中编码变量的精确皮质皮质机制 跨两个目标学习。总的来说，这些实验将：1）对 皮质区域及其在柔性增强学习中的角色，2）加速开创性 实验将在闭环中进行：实时使用解码的逆转学习 神经期望和3）最终将在心理病理学动物模型中比较这些信号 由于他们在逆转学习方面已知的失败。这些新颖和非常规的方法使 R21机制非常适合拟议的工作。