Collaborative Research: NCS: FO: Enhancing Episodic Memory through Real-world Integration of Brain Recording and Stimulation with Semantic Alignment of Human and IoT Perception

合作研究：NCS：FO：通过在现实世界中整合大脑记录和刺激以及人类和物联网感知的语义对齐来增强情景记忆

• 批准号: 2124130
• 负责人: Luis Garcia
• 金额: $ 50万
• 依托单位: University of Southern California
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2124130&HistoricalAwards=false
• 关键词: Collaborative; Research; NCS; FO; Enhancing

The goal of this project is to build and evaluate a system for exploring how the human brain processes information in everyday real-world environments. The investigators will directly record from the brain while participants navigate the real world, while synchronously recording information about the participant's first-person experience from a set of sensors including cameras, microphones, eye-tracking, and physiological recordings. Neurosurgical participants with a clinically implanted neural recording and stimulation system volunteer for these experiments, providing rare direct human brain recordings as they move around a real-world environment. The rich sensor data captured from the participant's first-person experience will be analyzed in relation to the neural data to infer how changes in patterns of neural activity over time relate to changes in experience. In addition, stimulation will be applied, at safe levels and timed according to "event boundaries" of the participant's experience, to determine whether memories of specific events can be enhanced. The proposed platform that allows for neural recording, direct brain stimulation, and synchronization with external, wearable devices will open an entirely new area of research at the intersection of computer science, engineering, cognition, and clinical neuroscience. These studies will launch and accelerate an emerging and pivotal area of research that will provide therapeutic interventions, proven in the real-world, for participants afflicted with debilitating cognitive disorders. This project will also make substantial contributions to education and outreach, including the development of K-12 classroom modules, interdisciplinary graduate training, outreach to industry partners in the neuromodulation field, and workshops at local Salt Lake City memory care communities.Development of this neural and first-person experience recording system will entail three collaborative research tasks: i) Synchronizing the Human Experience Relative to Neuronal Events: This module will develop a robust framework to record and synchronize neuronal activity along with internet of things (IoT) sensor data representing a broad subset of human sensory channels. The design will be portable such that the human experience can be reasoned about outside of a simulated lab environment. ii) Real-time Semantic Alignment between Human and IoT Perception: Reasoning about the complex relationships between neural biomarkers and the human experience captured by IoT sensing requires more than sensor synchronization. Neural-symbolic approaches that integrate the perception capabilities of deep learning with human logic will be leveraged to reason about the high-level complex spatiotemporal dependencies across a heterogeneous set of sensors. iii) Enhancing Episodic Memories of Real-world Experiences with DBS: Given a proper characterization of neural oscillations associated with event boundaries, the investigators will work to enhance episodic memories of real-world experiences with wireless deep brain stimulation (DBS) devices by directly stimulating the human brain. Under medical supervision, stimulation will be applied to the human amygdala at and between event boundaries in subjects with implanted stimulation devices as they encounter novel, 3D augmented reality objects while navigating a large-scale, real-world environment. Memory will be subsequently tested in laboratory and real-world settings. These three research areas will develop a situational understanding of neuronal activity in the context of human experience. They will further lay the foundation for future research directions in safe and effective stimulation of the brain in response to human experience in the wild.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目的目的是构建和评估一个系统，用于探索人类大脑如何处理日常现实环境中的信息。调查人员将直接从大脑中记录，而参与者则在现实世界中导航，同时记录有关参与者的第一人称体验的信息，包括相机，麦克风，眼睛跟踪和生理记录。具有临床植入神经记录和刺激系统志愿者的神经外科参与者为这些实验提供了志愿者，在围绕现实世界环境中移动时提供了罕见的直接人脑记录。将分析从参与者的第一人称体验中捕获的丰富传感器数据，以与神经数据有关，以推断神经活动模式随时间的变化与经验变化有关。此外，将在安全水平上应用刺激，并根据参与者经验的“事件边界”进行计时，以确定是否可以增强对特定事件的记忆。提出的平台，允许在计算机科学，工程，认知和临床神经科学的交集上开放神经记录，直接大脑刺激和与外部可穿戴设备同步的直接脑刺激和同步。这些研究将启动并加速研究的研究领域，该研究将为患有衰弱的认知障碍的参与者提供治疗性干预措施，并在现实世界中证明。该项目还将为教育和宣传做出重大贡献，包括开发K-12教室模块，跨学科的研究生培训，与神经调节领域的行业合作伙伴以及在当地盐湖城记忆护理社区的讲习班的宣传。开发这一神经和第一个人体验的录音系统将涉及三个协作的研究，这将涉及三个协作的研究：I）对INUREN的努力，I）相对范围，I）对同步的措施，同步，同步，同步。记录和同步神经元活动以及物联网（IoT）传感器数据的框架，代表了人类感觉渠道的广泛子集。该设计将是便携式的，以便可以在模拟实验室环境外面进行人类体验。 ii）人与物联网感知之间的实时语义一致性：关于神经生物标志物与物联网传感捕获的人类经验之间复杂关系的推理所需要的不仅仅需要传感器同步。将深度学习感知能力与人类逻辑相结合的神经符号方法将被利用，以理解一组异质的传感器集合中的高级复杂时空依赖性。 iii）增强DBS现实世界经历的情节记忆：考虑到与事件边界相关的神经振荡的适当表征，研究人员将致力于通过直接刺激人大脑来增强无线深脑刺激（DBS）设备的真实世界体验的情节记忆。在医疗监督下，刺激将应用于植入刺激设备的受试者中的人类杏仁核，当它们遇到小说时，3D增强现实对象，同时导航大型现实世界环境。内存将随后在实验室和现实世界中进行测试。这三个研究领域将在人类经验的背景下对神经元活动产生情境理解。他们将进一步为未来的研究方向奠定基础，以对野外人类的经验进行安全有效刺激大脑。该奖项反映了NSF的法定任务，并使用基金会的知识分子优点和更广泛的影响审查标准，被认为值得通过评估来提供支持。