Collaborative Research: NCS-FR: Beyond the ventral stream: Reverse engineering the neurocomputational basis of physical scene understanding in the primate brain

合作研究：NCS-FR：超越腹侧流：逆向工程灵长类大脑中物理场景理解的神经计算基础

• 批准号: 2124136
• 负责人: Nancy Kanwisher
• 金额: $ 225万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2124136&HistoricalAwards=false
• 关键词: Collaborative; Research; NCS; FR; Beyond

The last ten years have witnessed an astonishing revolution in AI, with deep neural networks suddenly approaching human-level performance on problems like recognizing objects in an image and words in an audio recording. But impressive as these feats are, they fall far short of human-like intelligence. The critical gap between current AI and human intelligence is that, beyond just classifying patterns of input, humans build mental models of the world. This project begins with the problem of physical scene understanding: how one extracts not just the identities and locations of objects in the visual world, but also the physical properties of those objects, their positions and velocities, their relationships to each other, the forces acting upon them, and the effects of forces that could be exerted on them. It is hypothesized that humans represent this information in a structured mental model of the physical world, and use that model to predict what will happen next, much as the physics engine in a video game generates physically plausible future states of virtual worlds. To test this idea, computational models of physical scene understanding will be built and tested for their ability to predict future states of the physical world in a variety of scenarios. Performance of these models will then be compared to humans and to more traditional deep network models, both in terms of their accuracy on each task, and their patterns of errors. Computational models that incorporate structured representations of the physical world will then be tested against standard convolutional neural networks in their ability to explain neural responses of the human brain (using fMRI) and the monkey brain (using direct neural recording). These computational models will provide the first explicit theories of how physical scene understanding might work in the human brain, at the same time advancing the ability of AI systems to solve the same problems. Because the ability to understand and predict the physical world is essential for planning any action, this work is expected to help advance many technologies that require such planning, from robotics to self-driving cars to brain-machine interfaces. Each of the participating labs will also expand their established track records of recruiting, training, and mentoring women and under-represented minorities at the undergraduate, graduate, and postdoctoral levels. Finally, the collaborating laboratories will continue and increase their involvement in the dissemination of science to the general public, via public talks, web sites, and outreach activities.Deep neural networks have revolutionized object recognition in computers as well as understanding of object recognition in the primate brain, but object recognition is just one aspect of vision, and the ventral stream is just one of many brain systems. Studying physical scene understanding is a step toward scaling this reverse-engineering approach up to the rest of the mind and brain. Predicting what will happen next and planning effective action requires understanding the physical basis and physical relationships in the visual world. Yet it is unknown how humans do this or how machines could. Both challenges are addressed in this project by the building of image computable, neurally mappable computational models of physical scene understanding and prediction (Thread I), and using these models as explicit hypotheses for how the brain might accomplish these tasks, which will then be tested with behavioral and neural data from humans (Thread II) and non-human primates (Thread III). This project aims to make a transformative leap in understanding: from small-scale, special-case models and isolated experimental tests to an integrated large-scale, general-purpose model of a major swathe of the primate brain, that functionally explains much of the immediate content of our perceptual experience in every scene that confronts us. The work will advance theory by developing the first image-computable models capable of human-level physical scene understanding and prediction. Beyond understanding of the mind and brain, this research is directly relevant to AI and robotics (which require physical scene understanding), and brain-machine interfaces (which require understanding of the relevant neural codes). For the broader research community, the project will a) develop public datasets, benchmark tasks, and challenges, b) host adversarial collaborations to address these challenges, and c) host interdisciplinary workshops linking research communities from psychology to AI to neuroscience to address the fundamental questions that span these fields.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

过去十年见证了人工智能领域的惊人革命，深度神经网络在识别图像中的物体和音频录音中的单词等问题上突然接近人类水平的表现。尽管这些壮举令人印象深刻，但它们仍远远低于类人智能。当前人工智能和人类智能之间的关键差距在于，人类除了对输入模式进行分类之外，还建立了世界的心理模型。该项目从物理场景理解问题开始：如何不仅提取视觉世界中物体的身份和位置，还提取这些物体的物理属性、它们的位置和速度、它们彼此之间的关系、作用力对他们的影响，以及可能对他们施加的力量的影响。据推测，人类在物理世界的结构化心理模型中表示这些信息，并使用该模型来预测接下来会发生什么，就像视频游戏中的物理引擎生成虚拟世界的物理上合理的未来状态一样。为了测试这个想法，将建立物理场景理解的计算模型并测试其预测各种场景中物理世界未来状态的能力。然后，这些模型的性能将与人类和更传统的深度网络模型进行比较，无论是在每项任务的准确性还是错误模式方面。然后，将针对标准卷积神经网络测试包含物理世界结构化表示的计算模型，以测试其解释人脑（使用功能磁共振成像）和猴脑（使用直接神经记录）神经反应的能力。这些计算模型将提供第一个关于物理场景理解如何在人脑中发挥作用的明确理论，同时提高人工智能系统解决相同问题的能力。由于理解和预测物理世界的能力对于规划任何行动至关重要，因此这项工作预计将有助于推进许多需要此类规划的技术，从机器人到自动驾驶汽车再到脑机接口。每个参与实验室还将扩大其在本科生、研究生和博士后级别招募、培训和指导女性和代表性不足的少数族裔的既定记录。最后，合作实验室将通过公开演讲、网站和外展活动，继续并更多地参与向公众传播科学。深度神经网络彻底改变了计算机中的对象识别以及对计算机中对象识别的理解。灵长类动物的大脑，但物体识别只是视觉的一方面，腹侧流只是众多大脑系统之一。研究物理场景理解是将这种逆向工程方法扩展到思维和大脑的其余部分的一步。预测接下来会发生什么并计划有效的行动需要了解视觉世界中的物理基础和物理关系。然而，尚不清楚人类如何做到这一点，或者机器如何做到这一点。该项目通过构建物理场景理解和预测的图像可计算、神经可映射计算模型（线程 I）来解决这两个挑战，并使用这些模型作为大脑如何完成这些任务的明确假设，然后对其进行测试包含来自人类（线程 II）和非人类灵长类动物（线程 III）的行为和神经数据。该项目旨在实现理解上的变革性飞跃：从小规模、特殊情况模型和孤立的实验测试，到灵长类大脑主要区域的集成大规模、通用模型，该模型在功能上解释了大部分我们所面对的每个场景中感知体验的直接内容。这项工作将通过开发第一个能够实现人类水平的物理场景理解和预测的图像计算模型来推进理论发展。除了对心智和大脑的理解之外，这项研究还与人工智能和机器人技术（需要理解物理场景）以及脑机接口（需要理解相关的神经代码）直接相关。对于更广泛的研究社区，该项目将 a) 开发公共数据集、基准任务和挑战，b) 举办对抗性合作以应对这些挑战，c) 举办跨学科研讨会，将心理学、人工智能和神经科学等研究社区联系起来，以解决基本问题该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Catalyzing next-generation Artificial Intelligence through NeuroAI

通过 NeuroAI 促进下一代人工智能

• DOI: 10.1038/s41467-023-37180-x
• 发表时间: 2023-12
• 期刊: Nature Communications
• 影响因子: 16.6
• 作者: Zador, Anthony;Escola, Sean;Richards, Blake;Ölveczky, Bence;Bengio, Yoshua;Boahen, Kwabena;Botvinick, Matthew;Chklovskii, Dmitri;Churchland, Anne;Clopath, Claudia;et al
• 通讯作者: et al