CRCNS: Representational foundations of adaptive behavior in natural and artificial

CRCNS：自然和人工适应性行为的代表性基础

• 批准号: 9052441
• 负责人: Nathaniel Douglass Daw
• 金额: $ 42.55万
• 依托单位: PRINCETON UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2018-05-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9052441
• 关键词: Accounting; Achievement; Adaptive Behaviors; Algorithms; Anatomy; Area; Artificial Intelligence; Behavior; Behavioral; Brain; Cognitive; Cognitive Science; Complex; Computer Simulation; Data; Decision Making; Fostering; Foundations; Goals; Hippocampus (Brain); Human; Industry; International; Learning; Life; Light; Machine Learning; Maps; Medical; Methods; Military Personnel; Neurosciences; Parietal Lobe; Play; Process; Psychological reinforcement; Psychology; Research; Research Personnel; Rewards; Robotics; Role; Shapes; Structure; System; Techniques; Training; Training and Education; Underrepresented Minority; Work; base; behavioral economics; cognitive neuroscience; cognitive system; computational neuroscience; coping; developmental psychology; driving behavior; frontal lobe; human subject; interest; neuroimaging; neurophysiology; neuroregulation; programs; receptive field; relating to nervous system; scale up; simulation; success; theories; tool

 DESCRIPTION (provided by applicant): Overview: Among the most celebrated success stories in computational neuroscience is the discovery that many aspects of decision-making can be understood in terms of the formal framework of reinforcement learning (RL). Ideas drawn from RL have shed light on many behavioral phenomena in learning and action selection, on the functional anatomy and neural processes underlying reward-driven behavior, and on fundamental aspects of neuromodulatory function. However, for all these successes, RL-based work is haunted by an inconvenient truth: Standard RL algorithms scale poorly to large, complex problems. If human learning and decision-making are driven by RL-like mechanisms, how is it that we cope with the kinds of rich, large-scale tasks that are typical of everyday life? Existing research in both psychology and neuroscience hints at one answer to this question: Complex problems can be conquered if the decision-maker is equipped with compact, intelligently formatted representations of the task. This principle is seen in studies of expert play in chess, which show that chess masters leverage highly integrative internal representations of board configurations; in studies of frontal and parietal lobe function, which have revealed receptive fields strongly shaped by task contingencies; and studies on the hippocampus, which point to the role of this structure in supporting a hierarchically organized 'cognitive map,' of task space. Not coincidentally, the critical role of representation has come increasingly to the fore in RL-based research in machine learning and robotics, with growing interest in techniques for dimensionality reduction, hierarchy and deep learning. The present project aims toward a systematic, empirically validated account of the role of representation in supporting RL and goal-directed behavior at large. The project brings together three investigators with complementary expertise in cognitive and computational neuroscience (Botvinick, Gershman) and machine learning and robotics (Konidaris). Together, we propose an integrative, interdisciplinary program of research, applying behavioral and neuroimaging work with human subjects, computational modeling of neurophysiological and behavioral data, formal mathematical work and simulations with artificial agents. The proposed studies are diverse in theme and method, but work together toward a theory that is both formally grounded and empirically constrained. At a more concrete level, our research focuses on four specific classes of representation, considering the computational impact of each for RL, as well as the relevance of each to neuroscience and human behavior. As detailed in our Project Description, these include (1) metric embedding, (2) spectral decomposition, (3) hierarchical representation and (4) symbolic representation. In addition to investigating the implications of each of these four forms of representation individually, we hypothesize that they fit together into a tiered system, which works as a whole to support the sometimes competing demands of learning and action control. Intellectual Merit (provided by applicant): Understanding how representational structure impacts learning and decision making is a core challenge in cognitive science, behavioral neuroscience and, artificial intelligence. Success in establishing a computationally explicit, empirically validated theory in this area, with a specific focus on the role of representation in R, would represent an important achievement with wide repercussions. The strategy of leveraging conceptual tools from machine learning to investigate human behavior and brain function can offer considerable scientific leverage, as our own previous research illustrates. The proposed work is motivated by and builds upon established lines of research, bringing these together in order to capitalize on opportunities for synergy. In addition to answering specific empirical and computational questions, the proposed work aims to open up new avenues for future research in an important area of inquiry. Broader Impact (provided by applicant): The proposed work lies at the crossroads of neuroscience, psychology, artificial intelligence and machine learning, and promises to advance the growing exchange among these fields. The project brings together investigators with contrasting disciplinary affiliations, with the explicit goal of bridging between intellectual cultres. The proposed work is likely to find a wide scientific audience, given its relevance to cognitive and developmental psychology, behavioral, cognitive and systems neuroscience, and behavioral economics. However, the work is likely to be of equal interest within artificial intelligence, machine learning, and robotics, where a current challenge is precisely to understand how representation learning can allow RL to scale up to large problems. Representational approaches to RL are already of intense interest within industry, where the present investigators have a record of active engagement. The topic of the proposed work has applicability in other areas as well, including education and training, and military and medical decision support. The plan for the project has a robust training component at both graduate and postdoctoral levels, with a commitment to fostering involvement of underrepresented minorities, as well as international engagement.

 描述（由申请人提供）： 概述：计算神经科学中最著名的成功故事之一是发现决策的许多方面都可以用强化学习 (RL) 的正式框架来理解。阐明了学习和行动选择中的许多行为现象、奖励驱动行为背后的功能解剖和神经过程，以及神经调节功能的基本方面。然而，尽管取得了这些成功，基于强化学习的工作却被一个令人难以忽视的事实所困扰：标准强化学习算法很难扩展到大型、复杂的问题，如果人类的学习和决策是由类似强化学习的机制驱动的，那么我们如何应对日常生活中常见的各种丰富的、大规模的任务呢？心理学和神经科学的研究暗示了这个问题的一个答案：如果决策者配备了紧凑的、智能格式的任务表示，则可以在国际象棋专家对弈的研究中看到这一原则，这些研究表明。那个国际象棋大师利用板结构的高度整合的内部表征；在额叶和强顶叶功能的研究中，这些研究揭示了由任务偶然性形成的接受域；以及对海马体的研究，这些研究指出了这种结构在支持分层组织的“认知”中的作用。任务空间的地图。 并非巧合的是，随着人们对降维、层次结构和深度学习技术的兴趣日益浓厚，表示的关键作用在机器学习和机器人技术中基于强化学习的研究中日益凸显。 本项目旨在对表征在支持强化学习和目标导向行为方面的作用进行系统的、经过实证验证的解释。该项目汇集了三位在认知和计算神经科学（Botvinick、Gershman）和机器学习方面具有互补专业知识的研究人员。我们共同提出了一个综合的、跨学科的研究计划，将行为和神经成像工作应用于人类受试者、神经生理学和行为数据的计算建模、正式的数学工作和人工代理的模拟。所提出的研究在主题和方法上各不相同，但共同努力建立一个既有形式基础又有经验约束的理论。在更具体的层面上，我们的研究重点关注四种特定的表示类别，考虑到每种类别对强化学习的计算影响，如下所示。以及每个与神经科学和人类行为的相关性，除了研究之外，还包括（1）度量嵌入、（2）谱分解、（3）层次表示和（4）符号表示。其中每一项的影响四种单独的表现形式，我们勇敢地把它们组合在一起 一个分层系统，作为一个整体来支持学习和行动控制有时相互竞争的需求。 智力优势（由申请人提供）：理解表征结构如何影响学习和决策是认知科学、神经科学和人工智能领域的核心行为挑战，并在该领域成功建立了计算明确、经经验验证的理论，并有特定的重点。正如我们之前的研究所表明的那样，利用机器学习的概念工具来研究人类行为和大脑功能的策略可以提供相当大的科学杠杆作用。动机为并以现有的研究为基础，将这些研究结合起来，以利用协同作用的机会。除了回答具体的实证和计算问题外，拟议的工作还旨在为重要的研究领域的未来研究开辟新的途径。 。 更广泛的影响（由申请人提供）：拟议的工作位于神经科学、心理学、人工智能和机器学习的十字路口，并有望促进这些领域之间日益增长的交流，该项目将不同学科背景的研究人员聚集在一起，并具有明确的意义。鉴于其与认知和发展心理学、行为、认知和系统神经科学以及行为经济学的相关性，拟议的工作可能会找到广泛的科学受众。人工之内人工智能、机器学习和机器人技术，当前的挑战正是理解表征学习如何让强化学习扩展到大型问题，强化学习的表征方法已经引起了业界的强烈兴趣，目前的研究人员在该领域有积极的记录。拟议工作的主题也适用于其他领域，包括教育和培训以及军事和医疗决策支持。该项目的计划在研究生和博士后层面都有强大的培训内容，并致力于培养。代表性不足的少数群体的参与，如以及国际参与。